From 4edada89447ce25ddaa48a1c7fe84905e78cc5cb Mon Sep 17 00:00:00 2001
From: hiroki-abe-58
Date: Tue, 29 Sep 2026 09:49:15 +0900
Subject: [PATCH 01/39] docs: ADR-029 plan, what the Score order measurements
look at
Co-Authored-By: Claude Opus 5.5 (1M context)
---
docs/DECISIONS.md | 24 ++++++++++++++++++++++++
1 file changed, 24 insertions(+)
diff --git a/docs/DECISIONS.md b/docs/DECISIONS.md
index dff4ea3..37506df 100644
--- a/docs/DECISIONS.md
+++ b/docs/DECISIONS.md
@@ -581,6 +581,30 @@ Serving now skips by default; training does not, so Mode B keeps its data. Mode
---
+## ADR-029 — Score is read in one order; measure the position effect, and offer averaging off by default
+
+**Open.** Proposed in [#1](https://github.com/Abhinavexists/lev/issues/1); the default is the maintainer's call.
+
+**Problem.** Choice and binary Noul are read in two orders and averaged, which cancels the preference for the first letter (ADR-020). Score is not: its levels keep the low-to-high order training uses, because reversing them measured -2.4 points on helpsteer2, inside sampling noise, on a prompt training never produces (FINDINGS §12). What that leaves was not measured. On `interfaze-ai/lev` with Score questions over the 290 states of sokudan's `bench_en`, an identical-option control (every level the same text, so levels differ only by position and code) put slot 0 at -0.332 below the mean in calibrated log-probability and -0.931 in raw scores, and the first slot won 0.270 of the decisions over all six orders of three levels, where an order-free readout is at 1/3 (#1). The check is the one NandhaKishorM/laya#259 added to Laya, whose gate is -0.20 on raw scores.
+
+**Options.**
+
+- (a) Average Score over orders by default: `reversed` (2 rows, as Choice) or `cyclic` (every rotation, K rows). Both show the model level orders training never produces; in the `chat` style each line still names its level (`(level i of K)`).
+- (b) Keep Score in one order by default, ship averaging as `EngineConfig.score_order_average` (off), and put the control in CI: `lev presentation-checks` against a committed report, failing on drift.
+
+**What is measured, fixed before the runs.** One checkpoint (`interfaze-ai/lev`, bf16, one RTX 5090), sokudan's `bench_en` (290) and `bench_ja` (300), `score_order_average` in {off, reversed, cyclic}:
+
+1. The identical-option control and the first-slot rate, for Score and Choice, with the shipped calibration and raw (an empty `CalibrationProfile`).
+2. Score RPS and accuracy on the benches' 3-level urgency question, with Choice and Noul on the same requests expected unchanged (Score rows are the only rows added).
+3. Milliseconds per three-question request, as the cost of each mode.
+4. With averaging off, every answer identical to the current code on the same items.
+
+Every result is reported whichever way it goes, including accuracy that does not improve.
+
+**Evidence.** Pending the runs above.
+
+---
+
## ADR-030 — The server batches across requests and refuses work it cannot do in time
**Accepted.**
From eea147636ea87ca163a576e06d1f168f439827df Mon Sep 17 00:00:00 2001
From: hiroki-abe-58
Date: Tue, 29 Sep 2026 17:47:55 +0900
Subject: [PATCH 02/39] Measure Score's position effect with lev
presentation-checks, and offer Score order averaging (off by default)
Co-Authored-By: Claude Opus 5.5 (1M context)
---
docs/ARCHITECTURE.md | 2 +
packages/lev/src/lev/cli.py | 137 +++++++++++
packages/lev/src/lev/model.py | 28 ++-
packages/lev/src/lev/presentation.py | 281 +++++++++++++++++++++++
packages/lev/src/lev/server.py | 3 +
packages/lev/tests/test_presentation.py | 289 ++++++++++++++++++++++++
6 files changed, 732 insertions(+), 8 deletions(-)
create mode 100644 packages/lev/src/lev/presentation.py
create mode 100644 packages/lev/tests/test_presentation.py
diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md
index 580044c..b14ecc3 100644
--- a/docs/ARCHITECTURE.md
+++ b/docs/ARCHITECTURE.md
@@ -201,6 +201,8 @@ LoRA r32 on `q,k,v,o,gate,up,down` ≈ 25–40 M trainable parameters. **The Fam
- they get **separate fitted temperatures** — one global scalar would be wrong for at least one of them;
- on option sets where **both** are valid we measure agreement. Material disagreement means the router is a correctness hazard, not merely a capacity switch. This is an explicit eval, not an assumption.
+**Order averaging** (added in the build, not in the original design). Under Mode A with the state-first layout, Choice and binary Noul are read in their shown order and reversed, and the two distributions are averaged in canonical order (ADR-020). Score keeps its level order by default; `EngineConfig.score_order_average` opts it into the reversed order or every rotation, at 2x or Kx rows, and `lev presentation-checks` measures what that changes (ADR-029).
+
### 5.4 Output types
| Type | Mode A | Mode B |
diff --git a/packages/lev/src/lev/cli.py b/packages/lev/src/lev/cli.py
index a439f3e..2710bc8 100644
--- a/packages/lev/src/lev/cli.py
+++ b/packages/lev/src/lev/cli.py
@@ -216,12 +216,102 @@ def cmd_serve(args: argparse.Namespace) -> None:
prompt_style=args.prompt_style,
max_pending=args.max_pending,
max_batch_tokens=args.max_batch_tokens,
+ score_order_average=args.score_order_average,
),
host=args.host,
port=args.port,
)
+def _cuda():
+ """`torch.cuda` when a GPU is in use, else None; torch is imported only here."""
+ try:
+ import torch
+ except ImportError:
+ return None
+ return torch.cuda if torch.cuda.is_available() else None
+
+
+def cmd_presentation_checks(args: argparse.Namespace) -> None:
+ """Label-free presentation checks (ADR-029). Exits non-zero when a metric drifts
+ from `--compare` beyond `--tolerance`, or, with `--enforce-gate`, below a gate."""
+ import time
+
+ from . import presentation
+ from .calibrate import CalibrationProfile
+ from .model import load
+
+ states = presentation.read_states(args.states)[: args.limit]
+ engine = load(
+ args.checkpoint,
+ model_id=args.model,
+ cache_dir=args.model_cache,
+ calibration=args.calibration,
+ score_order_average=args.score_order_average,
+ )
+ if args.raw:
+ engine.calibration = CalibrationProfile()
+ kinds = [k.strip() for k in args.kinds.split(",") if k.strip()]
+ grouping = {
+ "split_checks": args.split_checks,
+ "max_score_rows": args.max_score_rows,
+ "score_order_average": args.score_order_average,
+ }
+ cuda = _cuda()
+ if cuda:
+ cuda.empty_cache()
+ cuda.reset_peak_memory_stats()
+ started = time.perf_counter()
+ report = presentation.run(
+ engine.system_one, states, args.lang, kinds, args.slot0_min, **grouping
+ )
+ seconds = time.perf_counter() - started
+ packed = presentation.packed_consistency(
+ engine.system_one, states[: args.packed], args.lang, kinds, **grouping
+ )
+ peak = (
+ {
+ "allocated_gib": round(cuda.max_memory_allocated() / 2**30, 2),
+ "reserved_gib": round(cuda.max_memory_reserved() / 2**30, 2),
+ }
+ if cuda
+ else None
+ )
+ if cuda:
+ cuda.empty_cache()
+ print(presentation.to_markdown(report))
+ print(f"packed vs alone: max |dp| {packed:.2e} over {min(args.packed, len(states))} states")
+ payload = {
+ "checkpoint": args.checkpoint,
+ "states": args.states,
+ "n_states": len(states),
+ "lang": args.lang,
+ "raw": args.raw,
+ "kinds": kinds,
+ **grouping,
+ "seconds": round(seconds, 1),
+ "peak_vram": peak,
+ "packed_max_abs_diff": packed,
+ "checks": report,
+ }
+ if args.out:
+ text = json.dumps(payload, indent=1, ensure_ascii=False) + "\n"
+ Path(args.out).write_text(text, "utf-8")
+ failures = []
+ if args.compare:
+ committed = json.loads(Path(args.compare).read_text("utf-8"))["checks"]
+ failures += presentation.drift(report, committed, args.tolerance)
+ if args.enforce_gate:
+ failures += [
+ f"{kind}.{check}: {r[check]['metric']:+.3f} below {r[check]['gate']:+.2f}"
+ for kind, r in report.items()
+ for check in ("identical", "first_slot")
+ if not r[check]["passed"]
+ ]
+ if failures:
+ sys.exit("presentation checks failed:\n " + "\n ".join(failures))
+
+
def main(argv: list[str] | None = None) -> None:
parser = argparse.ArgumentParser(prog="lev", description=__doc__)
sub = parser.add_subparsers(dest="command", required=True)
@@ -296,10 +386,57 @@ def main(argv: list[str] | None = None) -> None:
default=16384,
help="padded tokens per batched forward across requests (default 16384)",
)
+ serve_parser.add_argument(
+ "--score-order-average",
+ choices=["off", "reversed", "cyclic"],
+ default="off",
+ help="average Score levels over reversed or all cyclic orders (2x / Kx rows; ADR-029)",
+ )
serve_parser.add_argument("--host", default="127.0.0.1")
serve_parser.add_argument("--port", type=int, default=8000)
serve_parser.set_defaults(func=cmd_serve)
+ presentation_parser = sub.add_parser(
+ "presentation-checks",
+ help="label-free position checks for Score and Choice (ADR-029)",
+ )
+ presentation_parser.add_argument("states", help="JSONL with a `state` per line")
+ presentation_parser.add_argument("--checkpoint", default=None)
+ presentation_parser.add_argument("--model", default="Qwen/Qwen3.5-4B-Base")
+ presentation_parser.add_argument("--model-cache", default=None)
+ presentation_parser.add_argument("--calibration", default=None)
+ presentation_parser.add_argument("--lang", choices=["en", "ja"], default="en")
+ presentation_parser.add_argument("--kinds", default="score,choice")
+ presentation_parser.add_argument("--limit", type=int, default=None)
+ presentation_parser.add_argument(
+ "--raw", action="store_true", help="score without calibration (T = 1 everywhere)"
+ )
+ presentation_parser.add_argument(
+ "--score-order-average", choices=["off", "reversed", "cyclic"], default="off"
+ )
+ presentation_parser.add_argument(
+ "--packed", type=int, default=3, help="states for the packed-vs-alone check"
+ )
+ presentation_parser.add_argument(
+ "--split-checks",
+ action="store_true",
+ help="ask the identical and permuted questions in separate calls",
+ )
+ presentation_parser.add_argument(
+ "--max-score-rows",
+ type=int,
+ default=24,
+ help="Score rows per call before the probe is split (cyclic reads a level K times)",
+ )
+ presentation_parser.add_argument("--slot0-min", type=float, default=-0.20)
+ presentation_parser.add_argument("--out", default=None, help="write the report JSON here")
+ presentation_parser.add_argument("--compare", default=None, help="a committed report JSON")
+ presentation_parser.add_argument("--tolerance", type=float, default=0.05)
+ presentation_parser.add_argument(
+ "--enforce-gate", action="store_true", help="also fail when a check is below its gate"
+ )
+ presentation_parser.set_defaults(func=cmd_presentation_checks)
+
data_parser = sub.add_parser("data", help="build the training mixture and the eval set")
data_sub = data_parser.add_subparsers(dest="data_command", required=True)
diff --git a/packages/lev/src/lev/model.py b/packages/lev/src/lev/model.py
index f1714ca..9b6cb04 100644
--- a/packages/lev/src/lev/model.py
+++ b/packages/lev/src/lev/model.py
@@ -46,6 +46,10 @@ class EngineConfig:
noul_readout: Literal["rating", "binary"] = "rating"
# A second option order reduces letter-position bias; costs one extra batch row.
order_average: bool = True
+ # Score levels keep their low-to-high order unless this opts in (ADR-029): "reversed"
+ # adds the reversed order (2 rows per Score, as Choice), "cyclic" every rotation (K rows).
+ # Both show the model level orders training never produces (ADR-020, FINDINGS §12).
+ score_order_average: Literal["off", "reversed", "cyclic"] = "off"
# Serving can skip split codes; training keeps large sets for Mode B (ADR-028).
skip_multi_token_codes: bool = True
# Forking avoids repeated prefix work but adds a forward and cache copies (ADR-023).
@@ -279,16 +283,22 @@ def _respond(self, request: Prepared, logits, hidden) -> SystemOneResponse:
def _orders(self, question: Question, route: Route) -> list[list[int] | None]:
"""Candidate orders to render; None means the original order. Only lettered
Choice and binary Noul in state-first layout are reversed (ordered scales
- keep training's low-to-high order; schema-first shares options)."""
- if (
- not self.config.order_average
- or route.mode is not Mode.LABEL_TOKEN
- or self.config.layout is not Layout.STATE_FIRST
- or isinstance(question, Score)
- or (isinstance(question, Noul) and route.reason != BINARY_NOUL)
- ):
+ keep training's low-to-high order; schema-first shares options), unless
+ `score_order_average` opts a Score in: "reversed" adds one row, "cyclic"
+ K-1, both in orders training never shows (ADR-029)."""
+ if route.mode is not Mode.LABEL_TOKEN or self.config.layout is not Layout.STATE_FIRST:
return [None]
n = len(route.codes or [])
+ if isinstance(question, Score):
+ if self.config.score_order_average == "off" or n < 2:
+ return [None]
+ if self.config.score_order_average == "reversed":
+ return [None, list(reversed(range(n)))]
+ return [None] + [[(j + r) % n for j in range(n)] for r in range(1, n)]
+ if not self.config.order_average or (
+ isinstance(question, Noul) and route.reason != BINARY_NOUL
+ ):
+ return [None]
return [None, list(reversed(range(n)))] if n >= 2 else [None]
def _render(
@@ -520,6 +530,7 @@ def load(
compile: bool = False,
max_label_options: int | None = None,
skip_multi_token_codes: bool = True,
+ score_order_average: Literal["off", "reversed", "cyclic"] = "off",
) -> DecisionEngine:
"""Load a checkpoint -- a release directory, a training output or a Hub id --
into a ready `DecisionEngine`. With no checkpoint, serves `model_id` frozen.
@@ -595,6 +606,7 @@ def load(
prompt_style=prompt_style or "plain",
max_label_options=max_label_options,
skip_multi_token_codes=skip_multi_token_codes,
+ score_order_average=score_order_average,
)
engine = DecisionEngine(model, tokenizer, config, profile, mode_b_head=head)
engine.checkpoint = resolved
diff --git a/packages/lev/src/lev/presentation.py b/packages/lev/src/lev/presentation.py
new file mode 100644
index 0000000..5a06774
--- /dev/null
+++ b/packages/lev/src/lev/presentation.py
@@ -0,0 +1,281 @@
+"""Presentation checks: does an answer move with where an option is shown?
+
+The design of NandhaKishorM/laya#259, over `system_one` (ADR-029). Per question type
+(Score and Choice), on any list of states, with no labels:
+
+- **identical**: every option carries the same text, so options differ only by
+ position and code. Metric: slot 0's log-probability minus the mean over the slots,
+ averaged over states and configurations. 0 is no position effect; below 0, slot 0
+ is disfavoured. Choice keys must be unique, so they are numbered (`1`, `2`, ...)
+ and share one description, as Score levels share one text.
+- **first_slot**: three real options in all 3! = 6 orders per state. Every option
+ sits in every slot exactly twice, so an order-free readout picks slot 0 in exactly
+ 1/3 of the decisions.
+- **packed**: the probe asks its questions in as few `system_one` calls as fit
+ `MAX_SCORE_ROWS` (one, by default); this asks each question alone on a few states and
+ reports the largest probability difference, so what is measured is the presentation
+ and not the packing.
+
+Probabilities are whatever `system_one` returns: calibrated, unless the engine carries
+an empty `CalibrationProfile`. Within one question, calibrated log-probabilities are the
+raw scores divided by the bucket's temperature, so the identical metric shrinks by 1/T
+and the first-slot rate does not change. The proposed gate, -0.20, is Laya's, set on
+raw scores.
+"""
+
+from __future__ import annotations
+
+import itertools
+import json
+import math
+from collections.abc import Callable, Mapping, Sequence
+from typing import Any
+
+KINDS = ("score", "choice")
+IDENTICAL_KS = (3, 4, 5)
+SLOT0_MIN = -0.20
+FIRST_SLOT_MIN = 0.15
+# Score rows per `system_one` call. The default probe is 12 rows off and 24 reversed, one call
+# either way; cyclic reads a K-level question K times (42 rows), which filled a 32 GB GPU.
+MAX_SCORE_ROWS = 24
+
+# en follows Laya's presentation_checks.py; ja the same wording in Japanese.
+TEXT: dict[str, dict[str, Any]] = {
+ "en": {
+ "score": {
+ "instructions": "How urgent is this request?",
+ "identical": ("moderate", "a request"),
+ "options": ("Not urgent", "Soon", "Work is blocked"),
+ },
+ "choice": {
+ "instructions": "Which team should handle this message?",
+ "identical": ("a team", "a request"),
+ "options": {
+ "Billing": "payments, refunds, invoices",
+ "Technical": "bugs, outages, errors",
+ "Sales": "plans, contracts, quotes",
+ },
+ },
+ },
+ "ja": {
+ "score": {
+ "instructions": "この依頼の緊急度は",
+ "identical": ("中程度", "依頼"),
+ "options": ("急がない", "早めに", "業務が止まっている"),
+ },
+ "choice": {
+ "instructions": "この問い合わせはどの部署が担当すべきか",
+ "identical": ("部署", "依頼"),
+ "options": {
+ "請求": "支払い・返金・請求書",
+ "技術": "不具合・障害・エラー",
+ "営業": "料金プラン・契約・見積もり",
+ },
+ },
+ },
+}
+
+SystemOne = Callable[[Any, Mapping[str, dict]], Any]
+
+
+def identical_question(kind: str, text: str, k: int, instructions: str) -> dict:
+ if kind == "score":
+ return {"type": "score", "instructions": instructions, "criteria": [text] * k}
+ return {
+ "type": "choice",
+ "instructions": instructions,
+ "criteria": {str(i + 1): text for i in range(k)},
+ }
+
+
+def probe_questions(lang: str = "en", kinds: Sequence[str] = KINDS) -> dict[str, dict]:
+ """Every probe question for one state, keyed `kind|identical|text|k` or
+ `kind|perm|012`. One `system_one` call answers them all."""
+ questions: dict[str, dict] = {}
+ for kind in kinds:
+ spec = TEXT[lang][kind]
+ for text in spec["identical"]:
+ for k in IDENTICAL_KS:
+ questions[f"{kind}|identical|{text}|{k}"] = identical_question(
+ kind, text, k, spec["instructions"]
+ )
+ options = list(spec["options"])
+ for order in itertools.permutations(range(len(options))):
+ shown = [options[i] for i in order]
+ criteria = shown if kind == "score" else {key: spec["options"][key] for key in shown}
+ questions[f"{kind}|perm|{''.join(map(str, order))}"] = {
+ "type": kind,
+ "instructions": spec["instructions"],
+ "criteria": criteria,
+ }
+ return questions
+
+
+def slot_probs(answer: Any, question: dict) -> list[float]:
+ """An answer's probabilities in the order the options were shown."""
+ probs = answer.probabilities
+ if question["type"] == "score":
+ return [float(probs[i]) for i in range(len(question["criteria"]))]
+ return [float(probs[key]) for key in question["criteria"]]
+
+
+def leave_one_out(per_state: Sequence[float]) -> list[float]:
+ n, total = len(per_state), sum(per_state)
+ loo = [(total - x) / (n - 1) for x in per_state] if n > 1 else list(per_state)
+ return [min(loo), max(loo)]
+
+
+def score_rows(question: dict, score_order_average: str) -> int:
+ """Batch rows a Score question adds: one per order it is read in (`_orders`)."""
+ k = len(question["criteria"])
+ return {"off": 1, "reversed": 2, "cyclic": k}[score_order_average] if k >= 2 else 1
+
+
+def call_groups(
+ questions: dict[str, dict],
+ split_checks: bool = False,
+ max_score_rows: int = MAX_SCORE_ROWS,
+ score_order_average: str = "off",
+) -> list[dict[str, dict]]:
+ """The probe split into `system_one` calls. By default one call, cut only where the
+ Score rows would pass `max_score_rows`, which bounds memory under `cyclic`; with
+ `split_checks` the identical and permuted questions never share a call."""
+ groups: list[dict[str, dict]] = []
+ for check in ("identical", "perm") if split_checks else (None,):
+ current: dict[str, dict] = {}
+ rows = 0
+ for qid, question in questions.items():
+ if check is not None and qid.split("|")[1] != check:
+ continue
+ added = score_rows(question, score_order_average) if question["type"] == "score" else 0
+ if current and rows + added > max_score_rows:
+ groups.append(current)
+ current, rows = {}, 0
+ current[qid] = question
+ rows += added
+ if current:
+ groups.append(current)
+ return groups
+
+
+def run(
+ system_one: SystemOne,
+ states: Sequence[Any],
+ lang: str = "en",
+ kinds: Sequence[str] = KINDS,
+ slot0_min: float = SLOT0_MIN,
+ split_checks: bool = False,
+ max_score_rows: int = MAX_SCORE_ROWS,
+ score_order_average: str = "off",
+) -> dict[str, dict]:
+ """Both checks for each kind, from the `call_groups` calls per state (one by default)."""
+ questions = probe_questions(lang, kinds)
+ groups = call_groups(questions, split_checks, max_score_rows, score_order_average)
+ slot0: dict[str, list[float]] = {kind: [] for kind in kinds}
+ first: dict[str, list[float]] = {kind: [] for kind in kinds}
+ by_config: dict[str, dict[str, list[float]]] = {kind: {} for kind in kinds}
+ by_slot = {kind: [0] * len(TEXT[lang][kind]["options"]) for kind in kinds}
+ for state in states:
+ answers = {}
+ for group in groups:
+ answers.update(system_one(state, group).answers)
+ centred: dict[str, list[float]] = {kind: [] for kind in kinds}
+ firsts: dict[str, list[int]] = {kind: [] for kind in kinds}
+ for qid, question in questions.items():
+ kind, check, *rest = qid.split("|")
+ p = slot_probs(answers[qid], question)
+ if check == "identical":
+ logp = [math.log(max(x, 1e-12)) for x in p]
+ value = logp[0] - sum(logp) / len(logp)
+ centred[kind].append(value)
+ by_config[kind].setdefault("|".join(rest), []).append(value)
+ else:
+ slot = max(range(len(p)), key=p.__getitem__)
+ firsts[kind].append(int(slot == 0))
+ by_slot[kind][slot] += 1
+ for kind in kinds:
+ slot0[kind].append(sum(centred[kind]) / len(centred[kind]))
+ first[kind].append(sum(firsts[kind]) / len(firsts[kind]))
+ report: dict[str, dict] = {}
+ for kind in kinds:
+ identical = sum(slot0[kind]) / len(slot0[kind])
+ rate = sum(first[kind]) / len(first[kind])
+ report[kind] = {
+ "identical": {
+ "metric": identical,
+ "leave_one_out": leave_one_out(slot0[kind]),
+ "gate": slot0_min,
+ "passed": identical >= slot0_min,
+ "by_config": {c: sum(v) / len(v) for c, v in by_config[kind].items()},
+ },
+ "first_slot": {
+ "metric": rate,
+ "leave_one_out": leave_one_out(first[kind]),
+ "gate": FIRST_SLOT_MIN,
+ "passed": rate >= FIRST_SLOT_MIN,
+ "decisions": len(states) * math.factorial(len(TEXT[lang][kind]["options"])),
+ "argmax_by_slot": by_slot[kind],
+ },
+ }
+ return report
+
+
+def packed_consistency(
+ system_one: SystemOne,
+ states: Sequence[Any],
+ lang: str = "en",
+ kinds: Sequence[str] = KINDS,
+ split_checks: bool = False,
+ max_score_rows: int = MAX_SCORE_ROWS,
+ score_order_average: str = "off",
+) -> float:
+ """Largest |p| difference between the probe as `run` asks it and each question alone."""
+ questions = probe_questions(lang, kinds)
+ groups = call_groups(questions, split_checks, max_score_rows, score_order_average)
+ worst = 0.0
+ for state in states:
+ packed = {}
+ for group in groups:
+ packed.update(system_one(state, group).answers)
+ for qid, question in questions.items():
+ alone = system_one(state, {qid: question}).answers[qid]
+ for a, b in zip(
+ slot_probs(packed[qid], question), slot_probs(alone, question), strict=True
+ ):
+ worst = max(worst, abs(a - b))
+ return worst
+
+
+def drift(report: dict, snapshot: dict, tolerance: float) -> list[str]:
+ """Every metric that moved more than `tolerance` from the committed report."""
+ moved = []
+ for kind, checks in snapshot.items():
+ for check, committed in checks.items():
+ now = report.get(kind, {}).get(check, {}).get("metric")
+ if now is None:
+ moved.append(f"{kind}.{check}: missing from this run")
+ elif abs(now - committed["metric"]) > tolerance:
+ moved.append(f"{kind}.{check}: {committed['metric']:+.3f} -> {now:+.3f}")
+ return moved
+
+
+def to_markdown(report: dict) -> str:
+ lines = [
+ "| kind | identical (slot 0 - mean) | leave-one-out | first-slot rate | leave-one-out |",
+ "|---|---|---|---|---|",
+ ]
+ for kind, r in report.items():
+ ident, first = r["identical"], r["first_slot"]
+ lines.append(
+ f"| {kind} | {ident['metric']:+.3f} {'PASS' if ident['passed'] else 'FAIL'} "
+ f"| {ident['leave_one_out'][0]:+.3f} .. {ident['leave_one_out'][1]:+.3f} "
+ f"| {first['metric']:.3f} {'PASS' if first['passed'] else 'FAIL'} "
+ f"| {first['leave_one_out'][0]:.3f} .. {first['leave_one_out'][1]:.3f} |"
+ )
+ return "\n".join(lines)
+
+
+def read_states(path: str) -> list[Any]:
+ """`state` from each line of a JSONL file (bench_en.jsonl, bench_ja.jsonl or your own)."""
+ with open(path, encoding="utf-8") as fh:
+ return [json.loads(line)["state"] for line in fh if line.strip()]
diff --git a/packages/lev/src/lev/server.py b/packages/lev/src/lev/server.py
index e62ccb4..0ce5dad 100644
--- a/packages/lev/src/lev/server.py
+++ b/packages/lev/src/lev/server.py
@@ -30,6 +30,7 @@ def create_app(
max_pending: int = 64,
max_batch_tokens: int = 16384,
timeout: float = 30.0,
+ score_order_average: Literal["off", "reversed", "cyclic"] = "off",
) -> Any:
from fastapi import FastAPI, HTTPException, Response
from starlette.concurrency import run_in_threadpool
@@ -51,6 +52,7 @@ def _load() -> None:
compile=compile,
max_label_options=max_label_options,
skip_multi_token_codes=skip_multi_token_codes,
+ score_order_average=score_order_average,
)
state["batcher"] = Batcher(
engine, max_pending=max_pending, max_batch_tokens=max_batch_tokens, timeout=timeout
@@ -73,6 +75,7 @@ async def health() -> dict:
"noul_readout": config.noul_readout,
"max_label_options": config.max_label_options,
"order_average": config.order_average,
+ "score_order_average": config.score_order_average,
"prefix_mode": config.prefix_mode,
"compiled": config.compile,
"prompt_style": config.prompt_style,
diff --git a/packages/lev/tests/test_presentation.py b/packages/lev/tests/test_presentation.py
new file mode 100644
index 0000000..9b5638f
--- /dev/null
+++ b/packages/lev/tests/test_presentation.py
@@ -0,0 +1,289 @@
+"""Presentation checks (ADR-029) and opt-in Score order averaging.
+
+The checks run against a fake `system_one` that validates every request with
+lev's own `SystemOneRequest` and answers with lev's answer types, so a probe lev
+would reject, or an answer shape the checks misread, fails here without weights.
+"""
+
+from __future__ import annotations
+
+import json
+import math
+
+import pytest
+from lev import (
+ ChoiceAnswer,
+ Score,
+ ScoreAnswer,
+ SystemOneRequest,
+ SystemOneResponse,
+ Usage,
+)
+from lev.model import DecisionEngine, EngineConfig, average_orders
+from lev.presentation import (
+ IDENTICAL_KS,
+ KINDS,
+ SLOT0_MIN,
+ TEXT,
+ drift,
+ packed_consistency,
+ probe_questions,
+ read_states,
+ run,
+ to_markdown,
+)
+from lev.router import route
+from lev.types import Choice
+
+
+def softmax(z: list[float]) -> list[float]:
+ m = max(z)
+ e = [math.exp(v - m) for v in z]
+ return [v / sum(e) for v in e]
+
+
+class FakeEngine:
+ """logit = content[option text] + slot_bias[slot]. `packing` adds a term that
+ depends on how many questions share the call, which a packed readout must not."""
+
+ def __init__(self, slot0: float = 0.0, content: dict | None = None, packing: float = 0.0):
+ self.slot0, self.content, self.packing, self.calls = slot0, content or {}, packing, 0
+
+ def system_one(self, state, questions):
+ request = SystemOneRequest(state=state, questions=questions)
+ self.calls += 1
+ answers = {}
+ for name, q in request.questions.items():
+ texts = list(q.criteria)
+ z = [
+ self.content.get(t, 0.0)
+ + (self.slot0 if i == 0 else 0.0)
+ + self.packing * len(request.questions) * i
+ for i, t in enumerate(texts)
+ ]
+ p = softmax(z)
+ if q.type == "score":
+ answers[name] = ScoreAnswer(
+ score=0.0,
+ probabilities=dict(enumerate(p)),
+ legend=dict(enumerate(q.criteria)),
+ confidence=max(p),
+ )
+ else:
+ keys = list(q.criteria)
+ answers[name] = ChoiceAnswer(
+ choice=keys[p.index(max(p))],
+ probabilities=dict(zip(keys, p, strict=True)),
+ confidence=max(p),
+ )
+ return SystemOneResponse(model="fake", answers=answers, usage=Usage(input_tokens=1))
+
+
+class TestProbeQuestions:
+ @pytest.mark.parametrize("lang", sorted(TEXT))
+ def test_every_probe_is_a_valid_request(self, lang):
+ questions = probe_questions(lang)
+ assert SystemOneRequest(state="s", questions=questions)
+ per_kind = 2 * len(IDENTICAL_KS) + 6
+ assert len(questions) == per_kind * len(KINDS)
+
+ @pytest.mark.parametrize("lang", sorted(TEXT))
+ def test_identical_options_differ_only_by_position(self, lang):
+ for qid, q in probe_questions(lang).items():
+ if "|identical|" not in qid:
+ continue
+ texts = q["criteria"] if q["type"] == "score" else list(q["criteria"].values())
+ assert len(set(texts)) == 1
+ if q["type"] == "choice":
+ assert list(q["criteria"]) == [str(i + 1) for i in range(len(q["criteria"]))]
+
+ @pytest.mark.parametrize("lang", sorted(TEXT))
+ def test_every_option_sits_in_every_slot_twice(self, lang):
+ for kind in KINDS:
+ perms = [
+ list(q["criteria"])
+ for qid, q in probe_questions(lang, [kind]).items()
+ if "|perm|" in qid
+ ]
+ options = list(TEXT[lang][kind]["options"])
+ for slot in range(3):
+ assert sorted(p[slot] for p in perms) == sorted(options * 2)
+
+
+class TestChecks:
+ def test_a_flat_readout_sits_at_zero_and_one_third(self):
+ content = {"Soon": 3.0, "Technical": 3.0}
+ report = run(FakeEngine(content=content).system_one, ["a", "b"])
+ for kind in KINDS:
+ assert report[kind]["identical"]["metric"] == pytest.approx(0.0)
+ assert report[kind]["identical"]["passed"]
+ assert report[kind]["first_slot"]["metric"] == pytest.approx(1 / 3)
+
+ def test_a_planted_slot_zero_penalty_is_measured_exactly(self):
+ report = run(FakeEngine(slot0=-2.0).system_one, ["a", "b", "c"])
+ # centred slot 0 = -2 - (-2/K), averaged over K = 3, 4, 5 and both texts
+ expected = sum(-2.0 * (1 - 1 / k) for k in IDENTICAL_KS) / len(IDENTICAL_KS)
+ for kind in KINDS:
+ assert report[kind]["identical"]["metric"] == pytest.approx(expected)
+ assert not report[kind]["identical"]["passed"]
+ assert report[kind]["first_slot"]["argmax_by_slot"][0] == 0
+ assert report[kind]["first_slot"]["decisions"] == 18
+
+ def test_one_call_per_state(self):
+ engine = FakeEngine()
+ run(engine.system_one, ["a", "b", "c"])
+ assert engine.calls == 3
+
+ def test_the_gate_is_the_proposal_and_can_be_moved(self):
+ engine = FakeEngine(slot0=-0.3)
+ assert not run(engine.system_one, ["a"])["score"]["identical"]["passed"]
+ assert run(engine.system_one, ["a"], slot0_min=-1.0)["score"]["identical"]["passed"]
+ assert SLOT0_MIN == -0.20
+
+
+class TestPackedConsistency:
+ def test_an_isolated_readout_agrees_with_itself_alone(self):
+ assert packed_consistency(FakeEngine(slot0=-1.0).system_one, ["a"]) == pytest.approx(0.0)
+
+ def test_a_readout_that_depends_on_packing_is_caught(self):
+ assert packed_consistency(FakeEngine(packing=0.01).system_one, ["a"]) > 0.01
+
+
+class TestReport:
+ def test_drift_flags_only_what_moved_past_the_tolerance(self):
+ now = run(FakeEngine(slot0=-2.0).system_one, ["a"])
+ committed = json.loads(json.dumps(now))
+ assert drift(now, committed, 0.05) == []
+ committed["score"]["identical"]["metric"] += 0.2
+ assert drift(now, committed, 0.05) == [
+ f"score.identical: {committed['score']['identical']['metric']:+.3f}"
+ f" -> {now['score']['identical']['metric']:+.3f}"
+ ]
+ del now["choice"]
+ assert "choice.identical: missing from this run" in drift(now, committed, 0.05)
+
+ def test_markdown_has_a_row_per_kind(self):
+ text = to_markdown(run(FakeEngine().system_one, ["a"]))
+ assert text.count("\n") == 1 + len(KINDS)
+
+ def test_states_are_read_from_jsonl(self, tmp_path):
+ path = tmp_path / "s.jsonl"
+ path.write_text('{"state": "one"}\n\n{"state": "two", "x": 1}\n', "utf-8")
+ assert read_states(str(path)) == ["one", "two"]
+
+ def test_the_cli_is_registered(self):
+ from lev.cli import main
+
+ with pytest.raises(SystemExit) as exc:
+ main(["presentation-checks", "--help"])
+ assert exc.value.code == 0
+
+
+class TestScoreOrderAverage:
+ """Off by default: a Score keeps its level order unless the config opts in."""
+
+ def orders(self, question, tokenizer, **config):
+ engine = DecisionEngine(model=None, tokenizer=tokenizer, config=EngineConfig(**config))
+ return engine._orders(question, route(question, tokenizer))
+
+ def test_off_by_default(self, rich_tokenizer):
+ assert EngineConfig().score_order_average == "off"
+ assert self.orders(Score(criteria=["low", "mid", "high"]), rich_tokenizer) == [None]
+
+ def test_reversed_is_two_rows(self, rich_tokenizer):
+ q = Score(criteria=["low", "mid", "high"])
+ assert self.orders(q, rich_tokenizer, score_order_average="reversed") == [None, [2, 1, 0]]
+
+ def test_cyclic_is_every_rotation(self, rich_tokenizer):
+ q = Score(criteria=["low", "mid", "high"])
+ assert self.orders(q, rich_tokenizer, score_order_average="cyclic") == [
+ None,
+ [1, 2, 0],
+ [2, 0, 1],
+ ]
+
+ def test_independent_of_the_choice_switch(self, rich_tokenizer):
+ q = Score(criteria=["low", "high"])
+ assert self.orders(
+ q, rich_tokenizer, score_order_average="reversed", order_average=False
+ ) == [None, [1, 0]]
+ c = Choice(criteria={"a": None, "b": None, "c": None})
+ for mode in ("off", "reversed", "cyclic"):
+ assert self.orders(c, rich_tokenizer, score_order_average=mode) == [None, [2, 1, 0]]
+
+ def test_schema_first_never_doubles_the_prefix(self, rich_tokenizer):
+ from lev.prompt import Layout
+
+ q = Score(criteria=["low", "high"])
+ assert self.orders(
+ q, rich_tokenizer, score_order_average="cyclic", layout=Layout.SCHEMA_FIRST
+ ) == [None]
+
+ def test_cyclic_averaging_cancels_a_slot_bias_on_equal_content(self):
+ """Each level sits in each slot once, so a pure position bias averages out."""
+ k = 4
+ orders = [None] + [[(j + r) % k for j in range(k)] for r in range(1, k)]
+ biased = softmax([2.0, 0.0, 0.0, 0.0]) # slot 0 favoured, content equal
+ assert average_orders([biased] * k, orders) == pytest.approx([1 / k] * k)
+
+
+class TestCallGroups:
+ """How the probe is split into calls: one by default, bounded Score rows under cyclic."""
+
+ def rows(self, group, mode):
+ from lev.presentation import score_rows
+
+ return sum(score_rows(q, mode) for q in group.values() if q["type"] == "score")
+
+ def test_off_and_reversed_stay_one_call(self):
+ from lev.presentation import call_groups
+
+ questions = probe_questions("en", ["score"])
+ assert len(call_groups(questions)) == 1
+ assert self.rows(questions, "reversed") == 24
+ assert len(call_groups(questions, score_order_average="reversed")) == 1
+
+ def test_cyclic_is_cut_at_the_row_limit_and_asks_everything_once(self):
+ from lev.presentation import call_groups
+
+ questions = probe_questions("en")
+ assert self.rows(questions, "cyclic") == 42
+ groups = call_groups(questions, score_order_average="cyclic")
+ assert len(groups) > 1
+ assert all(self.rows(g, "cyclic") <= 24 for g in groups)
+ asked = [qid for g in groups for qid in g]
+ assert sorted(asked) == sorted(questions)
+
+ def test_split_checks_never_mixes_the_two_checks(self):
+ from lev.presentation import call_groups
+
+ for group in call_groups(probe_questions("ja"), split_checks=True):
+ assert len({qid.split("|")[1] for qid in group}) == 1
+
+ def test_the_split_does_not_change_an_isolated_readout(self):
+ engine = FakeEngine(slot0=-1.0, content={"Soon": 1.0})
+ one = run(engine.system_one, ["a", "b"])
+ split = run(
+ engine.system_one,
+ ["a", "b"],
+ split_checks=True,
+ max_score_rows=5,
+ score_order_average="cyclic",
+ )
+ assert split == one
+ assert packed_consistency(engine.system_one, ["a"], split_checks=True) == pytest.approx(0.0)
+
+
+class TestCommittedReport:
+ """`data/presentation-checks.json` is what `--compare` reads in CI."""
+
+ def test_it_parses_and_carries_every_metric(self):
+ from pathlib import Path
+
+ path = Path(__file__).resolve().parents[3] / "data" / "presentation-checks.json"
+ committed = json.loads(path.read_text("utf-8"))
+ assert committed["score_order_average"] == "off" and committed["n_states"] == 290
+ for kind in KINDS:
+ for check in ("identical", "first_slot"):
+ assert isinstance(committed["checks"][kind][check]["metric"], float)
+ assert drift(committed["checks"], committed["checks"], 0.0) == []
From 582c93c5b6d726df598dd6592d3966fbbd82a208 Mon Sep 17 00:00:00 2001
From: hiroki-abe-58
Date: Tue, 29 Sep 2026 17:47:56 +0900
Subject: [PATCH 03/39] ADR-029 evidence: before/after on bench_en, bench_ja as
reference; committed presentation report
Co-Authored-By: Claude Opus 5.5 (1M context)
---
data/adr-029-summary.json | 134 ++++++++++++++++++++++++++++++++++
data/presentation-checks.json | 86 ++++++++++++++++++++++
docs/DECISIONS.md | 34 ++++++++-
3 files changed, 253 insertions(+), 1 deletion(-)
create mode 100644 data/adr-029-summary.json
create mode 100644 data/presentation-checks.json
diff --git a/data/adr-029-summary.json b/data/adr-029-summary.json
new file mode 100644
index 0000000..58c6e72
--- /dev/null
+++ b/data/adr-029-summary.json
@@ -0,0 +1,134 @@
+{
+ "about": "Summary of the ADR-029 runs: the numbers behind its tables. interfaze-ai/lev (HF f8ef711), bf16, one RTX 5090, 2026-09-29. Raw per-run reports are not committed and are available on request.",
+ "benches": {
+ "bench_en": "sokudan data/bench_en.jsonl, 290 states (English)",
+ "bench_ja": "sokudan data/bench_ja.jsonl, 300 states (Japanese, outside Lev's target language; reference only)"
+ },
+ "score_presentation": {
+ "bench_en": {
+ "off": {
+ "identical_calibrated": -0.3321,
+ "identical_raw": -0.931,
+ "first_slot": 0.2695,
+ "first_slot_raw": 0.2695
+ },
+ "reversed": {
+ "identical_calibrated": -0.0101,
+ "identical_raw": 0.0878,
+ "first_slot": 0.3207,
+ "first_slot_raw": 0.3207
+ },
+ "cyclic": {
+ "identical_calibrated": -0.0161,
+ "identical_raw": -0.0365,
+ "first_slot": 0.3126,
+ "first_slot_raw": 0.3121
+ }
+ },
+ "bench_ja": {
+ "off": {
+ "identical_calibrated": -0.5184,
+ "identical_raw": -1.4532,
+ "first_slot": 0.24,
+ "first_slot_raw": 0.24
+ },
+ "reversed": {
+ "identical_calibrated": -0.0586,
+ "identical_raw": 0.038,
+ "first_slot": 0.315,
+ "first_slot_raw": 0.3139
+ },
+ "cyclic": {
+ "identical_calibrated": -0.0558,
+ "identical_raw": -0.1313,
+ "first_slot": 0.3017,
+ "first_slot_raw": 0.3006
+ }
+ }
+ },
+ "choice_presentation_off": {
+ "bench_en": {
+ "identical_calibrated": -0.1001,
+ "identical_raw": -0.1597,
+ "first_slot": 0.3333
+ },
+ "bench_ja": {
+ "identical_calibrated": -0.0547,
+ "identical_raw": -0.0362,
+ "first_slot": 0.3306
+ }
+ },
+ "bench_accuracy": {
+ "bench_en": {
+ "off": {
+ "score_acc": 0.5828,
+ "score_rps": 0.1504,
+ "choice_acc": 0.9172,
+ "noul_acc": 0.9379,
+ "noul_auroc": 0.9778,
+ "ms_per_request": 164.6
+ },
+ "reversed": {
+ "score_acc": 0.6517,
+ "score_rps": 0.1275,
+ "choice_acc": 0.9172,
+ "noul_acc": 0.9379,
+ "noul_auroc": 0.978,
+ "ms_per_request": 184.0
+ },
+ "cyclic": {
+ "score_acc": 0.6414,
+ "score_rps": 0.1249,
+ "choice_acc": 0.9172,
+ "noul_acc": 0.9379,
+ "noul_auroc": 0.9784,
+ "ms_per_request": 215.4
+ }
+ },
+ "bench_ja": {
+ "off": {
+ "score_acc": 0.6433,
+ "score_rps": 0.1384,
+ "choice_acc": 0.8867,
+ "noul_acc": 0.8167,
+ "noul_auroc": 0.889,
+ "ms_per_request": 150.9
+ },
+ "reversed": {
+ "score_acc": 0.73,
+ "score_rps": 0.1079,
+ "choice_acc": 0.8867,
+ "noul_acc": 0.8133,
+ "noul_auroc": 0.8877,
+ "ms_per_request": 171.4
+ },
+ "cyclic": {
+ "score_acc": 0.7167,
+ "score_rps": 0.108,
+ "choice_acc": 0.8867,
+ "noul_acc": 0.8133,
+ "noul_auroc": 0.8876,
+ "ms_per_request": 199.9
+ }
+ }
+ },
+ "noul_alone_bench_ja": {
+ "requests": 300,
+ "identical_across_modes": true
+ },
+ "off_vs_main_cf104b6": {
+ "bench_en": {
+ "responses": 40,
+ "identical": true
+ },
+ "bench_ja": {
+ "responses": 40,
+ "identical": true
+ }
+ },
+ "issue_1_reproduced": {
+ "identical_calibrated": -0.3321,
+ "identical_raw": -0.931,
+ "first_slot": 0.2695
+ }
+}
diff --git a/data/presentation-checks.json b/data/presentation-checks.json
new file mode 100644
index 0000000..ebfecbc
--- /dev/null
+++ b/data/presentation-checks.json
@@ -0,0 +1,86 @@
+{
+ "about": "Committed report for `lev presentation-checks --compare` (ADR-029). One run: interfaze-ai/lev (HF f8ef711), bf16, one RTX 5090, the shipped calibration, score_order_average off, kinds score and choice, lang en, 2026-09-29.",
+ "checkpoint": "interfaze-ai/lev",
+ "states": "sokudan bench_en.jsonl (github.com/hiroki-abe-58/sokudan, data/bench_en.jsonl, CC BY 4.0), 290 states",
+ "n_states": 290,
+ "lang": "en",
+ "raw": false,
+ "kinds": [
+ "score",
+ "choice"
+ ],
+ "score_order_average": "off",
+ "split_checks": false,
+ "max_score_rows": 24,
+ "packed_max_abs_diff": 0.0207,
+ "checks": {
+ "score": {
+ "identical": {
+ "metric": -0.3323,
+ "leave_one_out": [
+ -0.3362,
+ -0.3298
+ ],
+ "gate": -0.2,
+ "passed": false,
+ "by_config": {
+ "moderate|3": -0.1182,
+ "moderate|4": -0.2365,
+ "moderate|5": -0.3193,
+ "a request|3": -0.3249,
+ "a request|4": -0.4617,
+ "a request|5": -0.5332
+ }
+ },
+ "first_slot": {
+ "metric": 0.2713,
+ "leave_one_out": [
+ 0.2699,
+ 0.2722
+ ],
+ "gate": 0.15,
+ "passed": true,
+ "decisions": 1740,
+ "argmax_by_slot": [
+ 472,
+ 598,
+ 670
+ ]
+ }
+ },
+ "choice": {
+ "identical": {
+ "metric": -0.1006,
+ "leave_one_out": [
+ -0.1024,
+ -0.0991
+ ],
+ "gate": -0.2,
+ "passed": true,
+ "by_config": {
+ "a team|3": -0.0569,
+ "a team|4": -0.1228,
+ "a team|5": -0.2804,
+ "a request|3": 0.0892,
+ "a request|4": -0.0295,
+ "a request|5": -0.2033
+ }
+ },
+ "first_slot": {
+ "metric": 0.3333,
+ "leave_one_out": [
+ 0.3328,
+ 0.3339
+ ],
+ "gate": 0.15,
+ "passed": true,
+ "decisions": 1740,
+ "argmax_by_slot": [
+ 580,
+ 581,
+ 579
+ ]
+ }
+ }
+ }
+}
diff --git a/docs/DECISIONS.md b/docs/DECISIONS.md
index 37506df..b6e498c 100644
--- a/docs/DECISIONS.md
+++ b/docs/DECISIONS.md
@@ -601,7 +601,39 @@ Serving now skips by default; training does not, so Mode B keeps its data. Mode
Every result is reported whichever way it goes, including accuracy that does not improve.
-**Evidence.** Pending the runs above.
+**Evidence.** Run 2026-09-29 on `interfaze-ai/lev` (HF f8ef711), bf16, one RTX 5090. The numbers behind every table are in [`data/adr-029-summary.json`](../data/adr-029-summary.json), and the raw per-run reports are available on request. The committed report CI compares against is [`data/presentation-checks.json`](../data/presentation-checks.json). `bench_ja` is Japanese, outside Lev's target language, so its rows are a reference only; `bench_en` is the one that bears on the decision.
+
+Issue #1 reproduced with averaging off (Score probe alone, 12 questions per call): identical control -0.332 calibrated / -0.931 raw, first-slot rate 0.270. With averaging off, every answer on 20 items of each bench, the probe included, is identical to the current code.
+
+Score, identical control (slot 0 minus the mean; gate -0.20 raw) and first-slot rate (1/3 is order-free):
+
+```text
+ bench_en (290) bench_ja (300, reference)
+ calibrated raw first slot calibrated raw first slot
+ off -0.332 -0.931 0.270 -0.518 -1.453 0.240
+ reversed -0.010 +0.088 0.321 -0.059 +0.038 0.315
+ cyclic -0.016 -0.037 0.312 -0.056 -0.131 0.301
+```
+
+Choice, which is already read in two orders, with Score averaging off: identical control -0.100 / -0.160 (en) and -0.055 / -0.036 (ja), first-slot rate 0.333 and 0.331.
+
+The benches' three questions per request (4-way Choice, 3-level Score, Noul), shipped calibration:
+
+```text
+ Score acc Score RPS Choice acc Noul acc Noul AUROC ms / request
+ en off 0.583 0.150 0.917 0.938 0.9778 164.6
+ reversed 0.652 0.128 0.917 0.938 0.9780 184.0 (1.12x)
+ cyclic 0.641 0.125 0.917 0.938 0.9784 215.4 (1.31x)
+ ja off (ref.) 0.643 0.138 0.887 0.817 0.8890 150.9
+ reversed 0.730 0.108 0.887 0.813 0.8877 171.4 (1.14x)
+ cyclic 0.717 0.108 0.887 0.813 0.8876 199.9 (1.32x)
+```
+
+Choice does not move. Noul moves by one item on bench_ja, and asked alone it is identical under all three modes on all 300 items: the change comes from the batch the extra Score rows make (bf16), not from the Noul readout.
+
+Two caveats. Asking the probe's questions together or one at a time differs by up to |Δp| 0.18 (raw, bench_ja, off) on the three states checked per run, so a metric is tied to how many questions share a call; `lev presentation-checks` fixes that at one call of at most 24 Score rows. And `cyclic` needs that cap: 42 Score rows per call exhausted a 32 GB card. Even capped, PyTorch's reserved memory went past the card (44-49 GiB for `cyclic`, 62 GiB for the default probe of both kinds), which Windows backs with shared system memory.
+
+**Reading, for the decision.** `reversed` removes most of the slot-0 deficit and lifts Score accuracy by 6.9 points on bench_en (8.7 on the Japanese reference bench) at 1.12-1.14x the time. `cyclic` is no better than `reversed` on either bench and costs 1.31-1.32x. Neither was tested on helpsteer2, the case that kept Score out of averaging in ADR-020, and both benches are one family of synthetic support messages. The proposal keeps the default off and leaves the choice between (a) and (b) to the maintainer.
---
From b4572f454b460b1076eba3664213eb84f5b92db5 Mon Sep 17 00:00:00 2001
From: hiroki-abe-58
Date: Tue, 29 Sep 2026 18:05:12 +0900
Subject: [PATCH 04/39] presentation-checks --compare refuses a report measured
under other settings (exit 2)
Co-Authored-By: Claude Opus 5.5 (1M context)
---
data/presentation-checks.json | 2 +
packages/lev/src/lev/cli.py | 49 ++++++++++++++++++++-----
packages/lev/src/lev/presentation.py | 23 ++++++++++++
packages/lev/tests/test_presentation.py | 46 +++++++++++++++++++++++
4 files changed, 110 insertions(+), 10 deletions(-)
diff --git a/data/presentation-checks.json b/data/presentation-checks.json
index ebfecbc..aa3f4b3 100644
--- a/data/presentation-checks.json
+++ b/data/presentation-checks.json
@@ -12,6 +12,8 @@
"score_order_average": "off",
"split_checks": false,
"max_score_rows": 24,
+ "checkpoint_revision": "f8ef71157ec06a7d3b6435bc0756f9d735c33748",
+ "dtype": "torch.bfloat16",
"packed_max_abs_diff": 0.0207,
"checks": {
"score": {
diff --git a/packages/lev/src/lev/cli.py b/packages/lev/src/lev/cli.py
index 2710bc8..0cbc824 100644
--- a/packages/lev/src/lev/cli.py
+++ b/packages/lev/src/lev/cli.py
@@ -232,9 +232,24 @@ def _cuda():
return torch.cuda if torch.cuda.is_available() else None
+def _snapshot_revision(checkpoint) -> str | None:
+ """The Hub commit a checkpoint was loaded from: the `snapshots/` directory name."""
+ parts = Path(checkpoint).parts if checkpoint else ()
+ return parts[parts.index("snapshots") + 1] if "snapshots" in parts[:-1] else None
+
+
+def _weights_dtype(engine) -> str | None:
+ model = getattr(engine, "_eager_model", None) or getattr(engine, "model", None)
+ try:
+ return str(next(model.parameters()).dtype)
+ except (AttributeError, StopIteration, TypeError):
+ return None
+
+
def cmd_presentation_checks(args: argparse.Namespace) -> None:
- """Label-free presentation checks (ADR-029). Exits non-zero when a metric drifts
- from `--compare` beyond `--tolerance`, or, with `--enforce-gate`, below a gate."""
+ """Label-free presentation checks (ADR-029). With `--compare`, exits 2 when the run's
+ settings differ from the report's, and 1 when a metric drifts beyond `--tolerance`;
+ `--enforce-gate` also fails a check below its gate."""
import time
from . import presentation
@@ -257,6 +272,25 @@ def cmd_presentation_checks(args: argparse.Namespace) -> None:
"max_score_rows": args.max_score_rows,
"score_order_average": args.score_order_average,
}
+ settings = {
+ "n_states": len(states),
+ "lang": args.lang,
+ "raw": args.raw,
+ "kinds": kinds,
+ **grouping,
+ "checkpoint_revision": _snapshot_revision(engine.checkpoint),
+ "dtype": _weights_dtype(engine),
+ }
+ saved = json.loads(Path(args.compare).read_text("utf-8")) if args.compare else None
+ if saved is not None:
+ # Metrics move with the order mode, the batching and the weights; compare like with like.
+ differ = presentation.settings_mismatch(saved, settings)
+ if differ:
+ print(
+ "presentation settings differ from the committed report: " + "; ".join(differ),
+ file=sys.stderr,
+ )
+ sys.exit(2)
cuda = _cuda()
if cuda:
cuda.empty_cache()
@@ -284,11 +318,7 @@ def cmd_presentation_checks(args: argparse.Namespace) -> None:
payload = {
"checkpoint": args.checkpoint,
"states": args.states,
- "n_states": len(states),
- "lang": args.lang,
- "raw": args.raw,
- "kinds": kinds,
- **grouping,
+ **settings,
"seconds": round(seconds, 1),
"peak_vram": peak,
"packed_max_abs_diff": packed,
@@ -298,9 +328,8 @@ def cmd_presentation_checks(args: argparse.Namespace) -> None:
text = json.dumps(payload, indent=1, ensure_ascii=False) + "\n"
Path(args.out).write_text(text, "utf-8")
failures = []
- if args.compare:
- committed = json.loads(Path(args.compare).read_text("utf-8"))["checks"]
- failures += presentation.drift(report, committed, args.tolerance)
+ if saved is not None:
+ failures += presentation.drift(report, saved["checks"], args.tolerance)
if args.enforce_gate:
failures += [
f"{kind}.{check}: {r[check]['metric']:+.3f} below {r[check]['gate']:+.2f}"
diff --git a/packages/lev/src/lev/presentation.py b/packages/lev/src/lev/presentation.py
index 5a06774..d8c94e6 100644
--- a/packages/lev/src/lev/presentation.py
+++ b/packages/lev/src/lev/presentation.py
@@ -246,6 +246,29 @@ def packed_consistency(
return worst
+SETTINGS = (
+ "n_states",
+ "lang",
+ "raw",
+ "kinds",
+ "score_order_average",
+ "split_checks",
+ "max_score_rows",
+ "checkpoint_revision",
+ "dtype",
+)
+
+
+def settings_mismatch(saved: dict, run: dict) -> list[str]:
+ """Every recorded setting of `saved` that this run does not share, as
+ ` (report: X, run: Y)`. A setting the report never recorded is not checked."""
+ return [
+ f"{key} (report: {saved[key]!r}, run: {run.get(key)!r})"
+ for key in SETTINGS
+ if key in saved and saved[key] != run.get(key)
+ ]
+
+
def drift(report: dict, snapshot: dict, tolerance: float) -> list[str]:
"""Every metric that moved more than `tolerance` from the committed report."""
moved = []
diff --git a/packages/lev/tests/test_presentation.py b/packages/lev/tests/test_presentation.py
index 9b5638f..4afa056 100644
--- a/packages/lev/tests/test_presentation.py
+++ b/packages/lev/tests/test_presentation.py
@@ -48,6 +48,7 @@ class FakeEngine:
def __init__(self, slot0: float = 0.0, content: dict | None = None, packing: float = 0.0):
self.slot0, self.content, self.packing, self.calls = slot0, content or {}, packing, 0
+ self.checkpoint = None # a DecisionEngine's resolved checkpoint; None for a frozen base
def system_one(self, state, questions):
request = SystemOneRequest(state=state, questions=questions)
@@ -287,3 +288,48 @@ def test_it_parses_and_carries_every_metric(self):
for check in ("identical", "first_slot"):
assert isinstance(committed["checks"][kind][check]["metric"], float)
assert drift(committed["checks"], committed["checks"], 0.0) == []
+
+
+class TestCompareGuardsItsSettings:
+ """`--compare` compares like with like: settings first (exit 2), then metrics (exit 1)."""
+
+ def cli(self, monkeypatch, tmp_path, engine, *extra):
+ from lev.cli import main
+
+ monkeypatch.setattr("lev.model.load", lambda *a, **k: engine)
+ monkeypatch.setattr("lev.cli._cuda", lambda: None)
+ states = tmp_path / "states.jsonl"
+ states.write_text('{"state": "a"}\n{"state": "b"}\n', "utf-8")
+ main(["presentation-checks", str(states), "--packed", "0", *extra])
+
+ def committed(self, monkeypatch, tmp_path, engine) -> dict:
+ out = tmp_path / "report.json"
+ self.cli(monkeypatch, tmp_path, engine, "--out", str(out))
+ return json.loads(out.read_text("utf-8"))
+
+ def test_matching_settings_are_compared_metric_by_metric(self, monkeypatch, tmp_path):
+ engine = FakeEngine(slot0=-1.0)
+ saved = self.committed(monkeypatch, tmp_path, engine)
+ assert saved["score_order_average"] == "off" and saved["max_score_rows"] == 24
+ saved["checks"]["score"]["identical"]["metric"] += 0.5
+ path = tmp_path / "committed.json"
+ path.write_text(json.dumps(saved), "utf-8")
+ with pytest.raises(SystemExit) as exc:
+ self.cli(monkeypatch, tmp_path, engine, "--compare", str(path))
+ assert "score.identical" in str(exc.value.code)
+
+ def test_different_settings_are_refused_before_any_run(self, monkeypatch, tmp_path, capsys):
+ engine = FakeEngine(slot0=-1.0)
+ saved = self.committed(monkeypatch, tmp_path, engine)
+ saved["score_order_average"] = "reversed"
+ path = tmp_path / "committed.json"
+ path.write_text(json.dumps(saved), "utf-8")
+ calls = engine.calls
+ with pytest.raises(SystemExit) as exc:
+ self.cli(monkeypatch, tmp_path, engine, "--compare", str(path))
+ assert exc.value.code == 2
+ assert engine.calls == calls
+ assert (
+ "presentation settings differ from the committed report: "
+ "score_order_average (report: 'reversed', run: 'off')"
+ ) in capsys.readouterr().err
From 4c27d72e7aa37ea703df59a3c0f0a491b1f94cbe Mon Sep 17 00:00:00 2001
From: hiroki-abe-58
Date: Sun, 4 Oct 2026 23:35:13 +0900
Subject: [PATCH 05/39] Address review: one-read identical control,
engine-counted rows and a request row limit
Presentation checks (ADR-029):
- The identical-option control reads every question once (order_average=False):
averaged orders cancel it by construction (cyclic) or hide the middle slots
(reversed), so it measures the bias averaging has to remove.
- Choice control keys are unordered shapes, and every assignment of keys to slots
is read, so a key's own pull cancels.
- First-slot rate: a tie gives each of its m slots 1/m. The report adds
`consistent`, the states whose six orders all give one answer.
- Calls are cut by the engine's own row count (DecisionEngine.question_rows), for
Choice too; --max-rows replaces --max-score-rows and --split-checks.
- packed_consistency reuses the answers the run computed.
- --compare checks the model-free settings before the load, and only revision and
dtype after it.
Engine:
- order_average=False reads every question once, Score included.
- question_rows: the rows each question adds, from the same _orders prepare renders.
- max_request_rows (default 32, about the batcher's 16384-token budget at bench-length
rows): prepare refuses a larger request with a ValueError
(HTTP 422) before any GPU work; `lev serve --max-request-rows`, /health reports it.
Co-Authored-By: Claude Opus 5.5
Claude-Session: https://claude.ai/code/session_01VyLpqwg29AwAk4YfyeTqxP
---
packages/lev/src/lev/cli.py | 85 ++---
packages/lev/src/lev/model.py | 36 ++-
packages/lev/src/lev/presentation.py | 296 +++++++++--------
packages/lev/src/lev/server.py | 5 +-
packages/lev/tests/test_presentation.py | 401 +++++++++++++++++-------
5 files changed, 535 insertions(+), 288 deletions(-)
diff --git a/packages/lev/src/lev/cli.py b/packages/lev/src/lev/cli.py
index 0cbc824..9a3ff09 100644
--- a/packages/lev/src/lev/cli.py
+++ b/packages/lev/src/lev/cli.py
@@ -217,6 +217,11 @@ def cmd_serve(args: argparse.Namespace) -> None:
max_pending=args.max_pending,
max_batch_tokens=args.max_batch_tokens,
score_order_average=args.score_order_average,
+ **(
+ {"max_request_rows": args.max_request_rows}
+ if args.max_request_rows is not None
+ else {}
+ ),
),
host=args.host,
port=args.port,
@@ -254,54 +259,44 @@ def cmd_presentation_checks(args: argparse.Namespace) -> None:
from . import presentation
from .calibrate import CalibrationProfile
- from .model import load
+ from .model import MAX_REQUEST_ROWS, load
states = presentation.read_states(args.states)[: args.limit]
+ kinds = [k.strip() for k in args.kinds.split(",") if k.strip()]
+ max_rows = args.max_rows or MAX_REQUEST_ROWS
+ settings = {
+ "n_states": len(states),
+ "lang": args.lang,
+ "raw": args.raw,
+ "kinds": kinds,
+ "score_order_average": args.score_order_average,
+ "max_rows": max_rows,
+ }
+ saved = json.loads(Path(args.compare).read_text("utf-8")) if args.compare else None
+ _refuse_other_settings(saved, settings, presentation.PRE_LOAD)
engine = load(
args.checkpoint,
model_id=args.model,
cache_dir=args.model_cache,
calibration=args.calibration,
score_order_average=args.score_order_average,
+ max_request_rows=max(max_rows, MAX_REQUEST_ROWS),
)
if args.raw:
engine.calibration = CalibrationProfile()
- kinds = [k.strip() for k in args.kinds.split(",") if k.strip()]
- grouping = {
- "split_checks": args.split_checks,
- "max_score_rows": args.max_score_rows,
- "score_order_average": args.score_order_average,
- }
- settings = {
- "n_states": len(states),
- "lang": args.lang,
- "raw": args.raw,
- "kinds": kinds,
- **grouping,
- "checkpoint_revision": _snapshot_revision(engine.checkpoint),
- "dtype": _weights_dtype(engine),
- }
- saved = json.loads(Path(args.compare).read_text("utf-8")) if args.compare else None
- if saved is not None:
- # Metrics move with the order mode, the batching and the weights; compare like with like.
- differ = presentation.settings_mismatch(saved, settings)
- if differ:
- print(
- "presentation settings differ from the committed report: " + "; ".join(differ),
- file=sys.stderr,
- )
- sys.exit(2)
+ settings["checkpoint_revision"] = _snapshot_revision(engine.checkpoint)
+ settings["dtype"] = _weights_dtype(engine)
+ _refuse_other_settings(saved, settings, presentation.POST_LOAD)
cuda = _cuda()
if cuda:
cuda.empty_cache()
cuda.reset_peak_memory_stats()
started = time.perf_counter()
- report = presentation.run(
- engine.system_one, states, args.lang, kinds, args.slot0_min, **grouping
- )
+ answers = presentation.ask(engine, states, args.lang, kinds, max_rows)
seconds = time.perf_counter() - started
+ report = presentation.report(answers, args.lang, kinds, args.slot0_min)
packed = presentation.packed_consistency(
- engine.system_one, states[: args.packed], args.lang, kinds, **grouping
+ engine, states[: args.packed], answers, args.lang, kinds
)
peak = (
{
@@ -341,6 +336,19 @@ def cmd_presentation_checks(args: argparse.Namespace) -> None:
sys.exit("presentation checks failed:\n " + "\n ".join(failures))
+def _refuse_other_settings(saved: dict | None, settings: dict, keys) -> None:
+ """Metrics move with the order mode, the batching and the weights; compare like with like."""
+ from .presentation import settings_mismatch
+
+ differ = settings_mismatch(saved, settings, keys) if saved is not None else []
+ if differ:
+ print(
+ "presentation settings differ from the committed report: " + "; ".join(differ),
+ file=sys.stderr,
+ )
+ sys.exit(2)
+
+
def main(argv: list[str] | None = None) -> None:
parser = argparse.ArgumentParser(prog="lev", description=__doc__)
sub = parser.add_subparsers(dest="command", required=True)
@@ -415,6 +423,12 @@ def main(argv: list[str] | None = None) -> None:
default=16384,
help="padded tokens per batched forward across requests (default 16384)",
)
+ serve_parser.add_argument(
+ "--max-request-rows",
+ type=int,
+ default=None,
+ help="batch rows one request may need before it gets 422 (default: the engine's)",
+ )
serve_parser.add_argument(
"--score-order-average",
choices=["off", "reversed", "cyclic"],
@@ -447,15 +461,10 @@ def main(argv: list[str] | None = None) -> None:
"--packed", type=int, default=3, help="states for the packed-vs-alone check"
)
presentation_parser.add_argument(
- "--split-checks",
- action="store_true",
- help="ask the identical and permuted questions in separate calls",
- )
- presentation_parser.add_argument(
- "--max-score-rows",
+ "--max-rows",
type=int,
- default=24,
- help="Score rows per call before the probe is split (cyclic reads a level K times)",
+ default=None,
+ help="batch rows per call, as the engine counts them (default: max_request_rows)",
)
presentation_parser.add_argument("--slot0-min", type=float, default=-0.20)
presentation_parser.add_argument("--out", default=None, help="write the report JSON here")
diff --git a/packages/lev/src/lev/model.py b/packages/lev/src/lev/model.py
index 9b6cb04..b936a9f 100644
--- a/packages/lev/src/lev/model.py
+++ b/packages/lev/src/lev/model.py
@@ -33,6 +33,9 @@
_QUESTIONS = TypeAdapter(dict[str, Question])
+# Bench-length rows (~500 tokens) x 32 is about the batcher's 16384-token budget (ADR-029).
+MAX_REQUEST_ROWS = 32
+
@dataclass
class EngineConfig:
@@ -50,6 +53,9 @@ class EngineConfig:
# adds the reversed order (2 rows per Score, as Choice), "cyclic" every rotation (K rows).
# Both show the model level orders training never produces (ADR-020, FINDINGS §12).
score_order_average: Literal["off", "reversed", "cyclic"] = "off"
+ # Rows one request may put into a forward (every question's every order); `prepare`
+ # refuses more with a ValueError (HTTP 422). None: no limit.
+ max_request_rows: int | None = MAX_REQUEST_ROWS
# Serving can skip split codes; training keeps large sets for Mode B (ADR-028).
skip_multi_token_codes: bool = True
# Forking avoids repeated prefix work but adds a forward and cache copies (ADR-023).
@@ -213,8 +219,23 @@ def prepare(self, state, questions: Mapping[str, Question | dict]) -> Prepared:
with self._tokenizer_lock:
return self._route_and_tokenise(state, questions)
+ def question_rows(self, questions: Mapping[str, Question | dict]) -> dict[str, int]:
+ """Batch rows each question adds to a request under this engine's config:
+ the orders `prepare` renders it in. Depends on routing, not on the state."""
+ questions = _QUESTIONS.validate_python(questions)
+ with self._tokenizer_lock:
+ routes = serving_routes(questions, self.tokenizer, self.config)
+ return {name: len(self._orders(q, routes[name])) for name, q in questions.items()}
+
def _route_and_tokenise(self, state, questions: dict[str, Question]) -> Prepared:
routes = serving_routes(questions, self.tokenizer, self.config)
+ limit = self.config.max_request_rows
+ rows = sum(len(self._orders(q, routes[name])) for name, q in questions.items())
+ if limit is not None and rows > limit:
+ raise ValueError(
+ f"this request needs {rows} batch rows, above max_request_rows={limit}; "
+ "send fewer questions per request, or average fewer orders"
+ )
unsupported = [
name
@@ -285,8 +306,13 @@ def _orders(self, question: Question, route: Route) -> list[list[int] | None]:
Choice and binary Noul in state-first layout are reversed (ordered scales
keep training's low-to-high order; schema-first shares options), unless
`score_order_average` opts a Score in: "reversed" adds one row, "cyclic"
- K-1, both in orders training never shows (ADR-029)."""
- if route.mode is not Mode.LABEL_TOKEN or self.config.layout is not Layout.STATE_FIRST:
+ K-1, both in orders training never shows (ADR-029). `order_average=False`
+ reads every question once."""
+ if (
+ not self.config.order_average
+ or route.mode is not Mode.LABEL_TOKEN
+ or self.config.layout is not Layout.STATE_FIRST
+ ):
return [None]
n = len(route.codes or [])
if isinstance(question, Score):
@@ -295,9 +321,7 @@ def _orders(self, question: Question, route: Route) -> list[list[int] | None]:
if self.config.score_order_average == "reversed":
return [None, list(reversed(range(n)))]
return [None] + [[(j + r) % n for j in range(n)] for r in range(1, n)]
- if not self.config.order_average or (
- isinstance(question, Noul) and route.reason != BINARY_NOUL
- ):
+ if isinstance(question, Noul) and route.reason != BINARY_NOUL:
return [None]
return [None, list(reversed(range(n)))] if n >= 2 else [None]
@@ -531,6 +555,7 @@ def load(
max_label_options: int | None = None,
skip_multi_token_codes: bool = True,
score_order_average: Literal["off", "reversed", "cyclic"] = "off",
+ max_request_rows: int | None = MAX_REQUEST_ROWS,
) -> DecisionEngine:
"""Load a checkpoint -- a release directory, a training output or a Hub id --
into a ready `DecisionEngine`. With no checkpoint, serves `model_id` frozen.
@@ -607,6 +632,7 @@ def load(
max_label_options=max_label_options,
skip_multi_token_codes=skip_multi_token_codes,
score_order_average=score_order_average,
+ max_request_rows=max_request_rows,
)
engine = DecisionEngine(model, tokenizer, config, profile, mode_b_head=head)
engine.checkpoint = resolved
diff --git a/packages/lev/src/lev/presentation.py b/packages/lev/src/lev/presentation.py
index d8c94e6..6e3d68b 100644
--- a/packages/lev/src/lev/presentation.py
+++ b/packages/lev/src/lev/presentation.py
@@ -3,18 +3,21 @@
The design of NandhaKishorM/laya#259, over `system_one` (ADR-029). Per question type
(Score and Choice), on any list of states, with no labels:
-- **identical**: every option carries the same text, so options differ only by
- position and code. Metric: slot 0's log-probability minus the mean over the slots,
- averaged over states and configurations. 0 is no position effect; below 0, slot 0
- is disfavoured. Choice keys must be unique, so they are numbered (`1`, `2`, ...)
- and share one description, as Score levels share one text.
-- **first_slot**: three real options in all 3! = 6 orders per state. Every option
- sits in every slot exactly twice, so an order-free readout picks slot 0 in exactly
- 1/3 of the decisions.
-- **packed**: the probe asks its questions in as few `system_one` calls as fit
- `MAX_SCORE_ROWS` (one, by default); this asks each question alone on a few states and
- reports the largest probability difference, so what is measured is the presentation
- and not the packing.
+- **identical**: the position bias of one read, before any order averaging. Every
+ option carries the same text, so options differ only by position and code. Metric:
+ slot 0's log-probability minus the mean over the slots, averaged over states and
+ configurations; 0 is no position effect. It is read with `order_average=False`,
+ because averaged orders cancel it by construction (cyclic) or hide the middle slots
+ (reversed): it measures the bias averaging has to remove, not what is left after it.
+ Choice keys must be unique, so they are shapes with no order (`KEYS`), and every
+ assignment of keys to slots is read, so a key's own pull cancels.
+- **first_slot**: three real options in all 3! = 6 orders per state, read as the engine
+ is configured. An order-free readout picks slot 0 in exactly 1/3 of the decisions; a
+ tie gives each of its m slots 1/m. `consistent` is the share of states whose six orders
+ all give one answer. In the chat style a Score line names its level by its place in
+ the request (`(level i of K)`), so a permuted listing renumbers the levels too.
+- **packed**: the largest probability difference between the probe's answers and each
+ question asked alone, so what is measured is the presentation and not the packing.
Probabilities are whatever `system_one` returns: calibrated, unless the engine carries
an empty `CalibrationProfile`. Within one question, calibrated log-probabilities are the
@@ -28,16 +31,16 @@
import itertools
import json
import math
-from collections.abc import Callable, Mapping, Sequence
+from collections.abc import Iterator, Mapping, Sequence
+from contextlib import contextmanager
+from dataclasses import replace
from typing import Any
KINDS = ("score", "choice")
IDENTICAL_KS = (3, 4, 5)
+KEYS = ("■", "●", "◆", "▲", "◎")
SLOT0_MIN = -0.20
FIRST_SLOT_MIN = 0.15
-# Score rows per `system_one` call. The default probe is 12 rows off and 24 reversed, one call
-# either way; cyclic reads a K-level question K times (42 rows), which filled a 32 GB GPU.
-MAX_SCORE_ROWS = 24
# en follows Laya's presentation_checks.py; ja the same wording in Japanese.
TEXT: dict[str, dict[str, Any]] = {
@@ -75,30 +78,36 @@
},
}
-SystemOne = Callable[[Any, Mapping[str, dict]], Any]
+# Settings a report records. The first group is known before the model loads.
+PRE_LOAD = ("n_states", "lang", "raw", "kinds", "score_order_average", "max_rows")
+POST_LOAD = ("checkpoint_revision", "dtype")
+SETTINGS = PRE_LOAD + POST_LOAD
-def identical_question(kind: str, text: str, k: int, instructions: str) -> dict:
+def identical_questions(kind: str, text: str, k: int, instructions: str) -> dict[str, dict]:
+ """The identical-option questions of one configuration, keyed by key assignment."""
if kind == "score":
- return {"type": "score", "instructions": instructions, "criteria": [text] * k}
+ return {"": {"type": "score", "instructions": instructions, "criteria": [text] * k}}
return {
- "type": "choice",
- "instructions": instructions,
- "criteria": {str(i + 1): text for i in range(k)},
+ "".join(map(str, keys)): {
+ "type": "choice",
+ "instructions": instructions,
+ "criteria": {KEYS[i]: text for i in keys},
+ }
+ for keys in itertools.permutations(range(k))
}
def probe_questions(lang: str = "en", kinds: Sequence[str] = KINDS) -> dict[str, dict]:
- """Every probe question for one state, keyed `kind|identical|text|k` or
- `kind|perm|012`. One `system_one` call answers them all."""
+ """Every probe question for one state, keyed `kind|identical|text|k|keys` or
+ `kind|perm|012`."""
questions: dict[str, dict] = {}
for kind in kinds:
spec = TEXT[lang][kind]
for text in spec["identical"]:
for k in IDENTICAL_KS:
- questions[f"{kind}|identical|{text}|{k}"] = identical_question(
- kind, text, k, spec["instructions"]
- )
+ for keys, q in identical_questions(kind, text, k, spec["instructions"]).items():
+ questions[f"{kind}|identical|{text}|{k}|{keys}"] = q
options = list(spec["options"])
for order in itertools.permutations(range(len(options))):
shown = [options[i] for i in order]
@@ -111,6 +120,68 @@ def probe_questions(lang: str = "en", kinds: Sequence[str] = KINDS) -> dict[str,
return questions
+def is_identical(qid: str) -> bool:
+ return qid.split("|")[1] == "identical"
+
+
+@contextmanager
+def single_order(engine) -> Iterator[None]:
+ """Read every question once, in the order given."""
+ config = engine.config
+ engine.config = replace(config, order_average=False)
+ try:
+ yield
+ finally:
+ engine.config = config
+
+
+def call_groups(engine, questions: dict[str, dict], max_rows: int) -> list[dict[str, dict]]:
+ """`questions` split into `system_one` calls of at most `max_rows` batch rows, as the
+ engine counts them under its current config."""
+ rows = engine.question_rows(questions)
+ groups: list[dict[str, dict]] = []
+ current: dict[str, dict] = {}
+ total = 0
+ for qid, question in questions.items():
+ if current and total + rows[qid] > max_rows:
+ groups.append(current)
+ current, total = {}, 0
+ current[qid] = question
+ total += rows[qid]
+ if current:
+ groups.append(current)
+ return groups
+
+
+def ask(
+ engine,
+ states: Sequence[Any],
+ lang: str = "en",
+ kinds: Sequence[str] = KINDS,
+ max_rows: int | None = None,
+) -> list[dict[str, Any]]:
+ """Every probe answer for every state: the identical questions in one order, the
+ permuted ones as the engine is configured. Calls hold at most `max_rows` batch rows,
+ the engine's `max_request_rows` by default."""
+ max_rows = max_rows or engine.config.max_request_rows
+ questions = probe_questions(lang, kinds)
+ identical = {q: v for q, v in questions.items() if is_identical(q)}
+ permuted = {q: v for q, v in questions.items() if not is_identical(q)}
+ with single_order(engine):
+ identical_groups = call_groups(engine, identical, max_rows)
+ permuted_groups = call_groups(engine, permuted, max_rows)
+ answers = []
+ for state in states:
+ got: dict[str, Any] = {}
+ with single_order(engine):
+ for group in identical_groups:
+ got.update(engine.system_one(state, group).answers)
+ for group in permuted_groups:
+ got.update(engine.system_one(state, group).answers)
+ answers.append(got)
+ return answers
+
+
def slot_probs(answer: Any, question: dict) -> list[float]:
"""An answer's probabilities in the order the options were shown."""
probs = answer.probabilities
@@ -119,126 +190,100 @@ def slot_probs(answer: Any, question: dict) -> list[float]:
return [float(probs[key]) for key in question["criteria"]]
+def top_slots(p: Sequence[float]) -> list[int]:
+ best = max(p)
+ return [i for i, x in enumerate(p) if x == best]
+
+
def leave_one_out(per_state: Sequence[float]) -> list[float]:
n, total = len(per_state), sum(per_state)
loo = [(total - x) / (n - 1) for x in per_state] if n > 1 else list(per_state)
return [min(loo), max(loo)]
-def score_rows(question: dict, score_order_average: str) -> int:
- """Batch rows a Score question adds: one per order it is read in (`_orders`)."""
- k = len(question["criteria"])
- return {"off": 1, "reversed": 2, "cyclic": k}[score_order_average] if k >= 2 else 1
-
-
-def call_groups(
- questions: dict[str, dict],
- split_checks: bool = False,
- max_score_rows: int = MAX_SCORE_ROWS,
- score_order_average: str = "off",
-) -> list[dict[str, dict]]:
- """The probe split into `system_one` calls. By default one call, cut only where the
- Score rows would pass `max_score_rows`, which bounds memory under `cyclic`; with
- `split_checks` the identical and permuted questions never share a call."""
- groups: list[dict[str, dict]] = []
- for check in ("identical", "perm") if split_checks else (None,):
- current: dict[str, dict] = {}
- rows = 0
- for qid, question in questions.items():
- if check is not None and qid.split("|")[1] != check:
- continue
- added = score_rows(question, score_order_average) if question["type"] == "score" else 0
- if current and rows + added > max_score_rows:
- groups.append(current)
- current, rows = {}, 0
- current[qid] = question
- rows += added
- if current:
- groups.append(current)
- return groups
+def mean(values: Sequence[float]) -> float:
+ return sum(values) / len(values)
-def run(
- system_one: SystemOne,
- states: Sequence[Any],
+def report(
+ answers: Sequence[Mapping[str, Any]],
lang: str = "en",
kinds: Sequence[str] = KINDS,
slot0_min: float = SLOT0_MIN,
- split_checks: bool = False,
- max_score_rows: int = MAX_SCORE_ROWS,
- score_order_average: str = "off",
) -> dict[str, dict]:
- """Both checks for each kind, from the `call_groups` calls per state (one by default)."""
+ """Both checks for each kind from `ask`'s answers."""
questions = probe_questions(lang, kinds)
- groups = call_groups(questions, split_checks, max_score_rows, score_order_average)
- slot0: dict[str, list[float]] = {kind: [] for kind in kinds}
- first: dict[str, list[float]] = {kind: [] for kind in kinds}
- by_config: dict[str, dict[str, list[float]]] = {kind: {} for kind in kinds}
- by_slot = {kind: [0] * len(TEXT[lang][kind]["options"]) for kind in kinds}
- for state in states:
- answers = {}
- for group in groups:
- answers.update(system_one(state, group).answers)
- centred: dict[str, list[float]] = {kind: [] for kind in kinds}
- firsts: dict[str, list[int]] = {kind: [] for kind in kinds}
- for qid, question in questions.items():
- kind, check, *rest = qid.split("|")
- p = slot_probs(answers[qid], question)
- if check == "identical":
- logp = [math.log(max(x, 1e-12)) for x in p]
- value = logp[0] - sum(logp) / len(logp)
- centred[kind].append(value)
- by_config[kind].setdefault("|".join(rest), []).append(value)
- else:
- slot = max(range(len(p)), key=p.__getitem__)
- firsts[kind].append(int(slot == 0))
- by_slot[kind][slot] += 1
- for kind in kinds:
- slot0[kind].append(sum(centred[kind]) / len(centred[kind]))
- first[kind].append(sum(firsts[kind]) / len(firsts[kind]))
- report: dict[str, dict] = {}
+ out: dict[str, dict] = {}
for kind in kinds:
- identical = sum(slot0[kind]) / len(slot0[kind])
- rate = sum(first[kind]) / len(first[kind])
- report[kind] = {
+ options = TEXT[lang][kind]["options"]
+ slot0, first, consistent = [], [], 0
+ by_config: dict[str, list[float]] = {}
+ by_slot = [0.0] * len(options)
+ for got in answers:
+ centred: dict[str, list[float]] = {}
+ firsts, picks = [], set()
+ for qid, question in questions.items():
+ if not qid.startswith(kind + "|"):
+ continue
+ p = slot_probs(got[qid], question)
+ if is_identical(qid):
+ logp = [math.log(max(x, 1e-12)) for x in p]
+ config = "|".join(qid.split("|")[2:4])
+ centred.setdefault(config, []).append(logp[0] - mean(logp))
+ continue
+ order = [int(c) for c in qid.split("|")[2]]
+ top = top_slots(p)
+ for slot in top:
+ by_slot[slot] += 1 / len(top)
+ firsts.append(1 / len(top) if 0 in top else 0.0)
+ picks.add(frozenset(order[slot] for slot in top))
+ per_config = {c: mean(v) for c, v in centred.items()}
+ for c, v in per_config.items():
+ by_config.setdefault(c, []).append(v)
+ slot0.append(mean(list(per_config.values())))
+ first.append(mean(firsts))
+ consistent += int(len(picks) == 1)
+ identical, rate = mean(slot0), mean(first)
+ out[kind] = {
"identical": {
"metric": identical,
- "leave_one_out": leave_one_out(slot0[kind]),
+ "leave_one_out": leave_one_out(slot0),
"gate": slot0_min,
"passed": identical >= slot0_min,
- "by_config": {c: sum(v) / len(v) for c, v in by_config[kind].items()},
+ "read": "single order",
+ "by_config": {c: mean(v) for c, v in by_config.items()},
},
"first_slot": {
"metric": rate,
- "leave_one_out": leave_one_out(first[kind]),
+ "leave_one_out": leave_one_out(first),
"gate": FIRST_SLOT_MIN,
"passed": rate >= FIRST_SLOT_MIN,
- "decisions": len(states) * math.factorial(len(TEXT[lang][kind]["options"])),
- "argmax_by_slot": by_slot[kind],
+ "decisions": len(answers) * math.factorial(len(options)),
+ "argmax_by_slot": by_slot,
+ "consistent": consistent / len(answers),
},
}
- return report
+ return out
def packed_consistency(
- system_one: SystemOne,
+ engine,
states: Sequence[Any],
+ answers: Sequence[Mapping[str, Any]],
lang: str = "en",
kinds: Sequence[str] = KINDS,
- split_checks: bool = False,
- max_score_rows: int = MAX_SCORE_ROWS,
- score_order_average: str = "off",
) -> float:
- """Largest |p| difference between the probe as `run` asks it and each question alone."""
+ """Largest |p| difference between `ask`'s answers for `states` and each question asked
+ alone, under the same order setting."""
questions = probe_questions(lang, kinds)
- groups = call_groups(questions, split_checks, max_score_rows, score_order_average)
worst = 0.0
- for state in states:
- packed = {}
- for group in groups:
- packed.update(system_one(state, group).answers)
+ for state, packed in zip(states, answers, strict=False):
for qid, question in questions.items():
- alone = system_one(state, {qid: question}).answers[qid]
+ if is_identical(qid):
+ with single_order(engine):
+ alone = engine.system_one(state, {qid: question}).answers[qid]
+ else:
+ alone = engine.system_one(state, {qid: question}).answers[qid]
for a, b in zip(
slot_probs(packed[qid], question), slot_probs(alone, question), strict=True
):
@@ -246,25 +291,12 @@ def packed_consistency(
return worst
-SETTINGS = (
- "n_states",
- "lang",
- "raw",
- "kinds",
- "score_order_average",
- "split_checks",
- "max_score_rows",
- "checkpoint_revision",
- "dtype",
-)
-
-
-def settings_mismatch(saved: dict, run: dict) -> list[str]:
- """Every recorded setting of `saved` that this run does not share, as
+def settings_mismatch(saved: dict, run: dict, keys: Sequence[str] = SETTINGS) -> list[str]:
+ """Every setting among `keys` that `saved` recorded and this run does not share, as
` (report: X, run: Y)`. A setting the report never recorded is not checked."""
return [
f"{key} (report: {saved[key]!r}, run: {run.get(key)!r})"
- for key in SETTINGS
+ for key in keys
if key in saved and saved[key] != run.get(key)
]
@@ -284,8 +316,9 @@ def drift(report: dict, snapshot: dict, tolerance: float) -> list[str]:
def to_markdown(report: dict) -> str:
lines = [
- "| kind | identical (slot 0 - mean) | leave-one-out | first-slot rate | leave-one-out |",
- "|---|---|---|---|---|",
+ "| kind | identical, one read (slot 0 - mean) | leave-one-out "
+ "| first-slot rate | leave-one-out | consistent |",
+ "|---|---|---|---|---|---|",
]
for kind, r in report.items():
ident, first = r["identical"], r["first_slot"]
@@ -293,7 +326,8 @@ def to_markdown(report: dict) -> str:
f"| {kind} | {ident['metric']:+.3f} {'PASS' if ident['passed'] else 'FAIL'} "
f"| {ident['leave_one_out'][0]:+.3f} .. {ident['leave_one_out'][1]:+.3f} "
f"| {first['metric']:.3f} {'PASS' if first['passed'] else 'FAIL'} "
- f"| {first['leave_one_out'][0]:.3f} .. {first['leave_one_out'][1]:.3f} |"
+ f"| {first['leave_one_out'][0]:.3f} .. {first['leave_one_out'][1]:.3f} "
+ f"| {first['consistent']:.3f} |"
)
return "\n".join(lines)
diff --git a/packages/lev/src/lev/server.py b/packages/lev/src/lev/server.py
index 0ce5dad..90c045b 100644
--- a/packages/lev/src/lev/server.py
+++ b/packages/lev/src/lev/server.py
@@ -13,7 +13,7 @@
from fastapi import Request
from .batcher import Batcher, ClientGone, Overloaded, WorkerStopped
-from .model import load
+from .model import MAX_REQUEST_ROWS, load
from .types import SystemOneRequest, SystemOneResponse
@@ -31,6 +31,7 @@ def create_app(
max_batch_tokens: int = 16384,
timeout: float = 30.0,
score_order_average: Literal["off", "reversed", "cyclic"] = "off",
+ max_request_rows: int | None = MAX_REQUEST_ROWS,
) -> Any:
from fastapi import FastAPI, HTTPException, Response
from starlette.concurrency import run_in_threadpool
@@ -53,6 +54,7 @@ def _load() -> None:
max_label_options=max_label_options,
skip_multi_token_codes=skip_multi_token_codes,
score_order_average=score_order_average,
+ max_request_rows=max_request_rows,
)
state["batcher"] = Batcher(
engine, max_pending=max_pending, max_batch_tokens=max_batch_tokens, timeout=timeout
@@ -76,6 +78,7 @@ async def health() -> dict:
"max_label_options": config.max_label_options,
"order_average": config.order_average,
"score_order_average": config.score_order_average,
+ "max_request_rows": config.max_request_rows,
"prefix_mode": config.prefix_mode,
"compiled": config.compile,
"prompt_style": config.prompt_style,
diff --git a/packages/lev/tests/test_presentation.py b/packages/lev/tests/test_presentation.py
index 4afa056..410a65e 100644
--- a/packages/lev/tests/test_presentation.py
+++ b/packages/lev/tests/test_presentation.py
@@ -1,39 +1,41 @@
-"""Presentation checks (ADR-029) and opt-in Score order averaging.
+"""Presentation checks (ADR-029), opt-in Score order averaging and the request row limit.
-The checks run against a fake `system_one` that validates every request with
-lev's own `SystemOneRequest` and answers with lev's answer types, so a probe lev
-would reject, or an answer shape the checks misread, fails here without weights.
+The checks run against `FakeEngine`: a real `DecisionEngine` without weights routes,
+renders and counts every request, as serving does, and only the logits are scripted.
+So a probe lev would reject, a row count that disagrees with the engine, or an answer
+shape the checks misread fails here without weights.
"""
from __future__ import annotations
import json
import math
+from pathlib import Path
+from string import ascii_uppercase
import pytest
-from lev import (
- ChoiceAnswer,
- Score,
- ScoreAnswer,
- SystemOneRequest,
- SystemOneResponse,
- Usage,
-)
-from lev.model import DecisionEngine, EngineConfig, average_orders
+from lev import ChoiceAnswer, Score, ScoreAnswer, SystemOneRequest, SystemOneResponse, Usage
+from lev.model import MAX_REQUEST_ROWS, DecisionEngine, EngineConfig, average_orders
from lev.presentation import (
IDENTICAL_KS,
+ KEYS,
KINDS,
+ POST_LOAD,
+ PRE_LOAD,
SLOT0_MIN,
TEXT,
+ ask,
+ call_groups,
drift,
packed_consistency,
probe_questions,
read_states,
- run,
+ report,
+ single_order,
to_markdown,
)
from lev.router import route
-from lev.types import Choice
+from lev.types import Choice, Noul
def softmax(z: list[float]) -> list[float]:
@@ -42,27 +44,59 @@ def softmax(z: list[float]) -> list[float]:
return [v / sum(e) for v in e]
+class Tokenizer:
+ """Space-prefixed A-Z and 0-8 are single tokens, as `rich_tokenizer`."""
+
+ single = {f" {c}" for c in ascii_uppercase} | {f" {i}" for i in range(9)}
+
+ def encode(self, text: str, add_special_tokens: bool = True) -> list[int]:
+ return [1] if text in self.single else [1] * (len(text) + 1)
+
+
class FakeEngine:
- """logit = content[option text] + slot_bias[slot]. `packing` adds a term that
- depends on how many questions share the call, which a packed readout must not."""
+ """logit = content[text] + key_pull[key] + slot0 (in slot 0) + a packing term that
+ grows with the questions per call, read in every order the engine renders and
+ averaged as lev does. Routing, orders, row counts and the row limit are the real
+ engine's."""
+
+ def __init__(self, slot0=0.0, content=None, packing=0.0, key_pull=None, **config):
+ self.real = DecisionEngine(model=None, tokenizer=Tokenizer(), config=EngineConfig(**config))
+ self.slot0, self.content, self.packing = slot0, content or {}, packing
+ self.key_pull = key_pull or {}
+ self.calls = 0
+ self.checkpoint = None
+
+ @property
+ def config(self) -> EngineConfig:
+ return self.real.config
+
+ @config.setter
+ def config(self, value: EngineConfig) -> None:
+ self.real.config = value
- def __init__(self, slot0: float = 0.0, content: dict | None = None, packing: float = 0.0):
- self.slot0, self.content, self.packing, self.calls = slot0, content or {}, packing, 0
- self.checkpoint = None # a DecisionEngine's resolved checkpoint; None for a frozen base
+ def question_rows(self, questions):
+ return self.real.question_rows(questions)
def system_one(self, state, questions):
- request = SystemOneRequest(state=state, questions=questions)
+ prepared = self.real.prepare(state, questions)
+ SystemOneRequest(state=state, questions=questions)
self.calls += 1
- answers = {}
- for name, q in request.questions.items():
- texts = list(q.criteria)
+ rows: dict[str, list] = {}
+ for variant in prepared.variants:
+ q = prepared.questions[variant.name]
+ items = list(q.criteria.items()) if q.type == "choice" else list(enumerate(q.criteria))
+ shown = items if variant.order is None else [items[i] for i in variant.order]
z = [
- self.content.get(t, 0.0)
- + (self.slot0 if i == 0 else 0.0)
- + self.packing * len(request.questions) * i
- for i, t in enumerate(texts)
+ self.content.get(text, 0.0)
+ + self.key_pull.get(key, 0.0)
+ + (self.slot0 if j == 0 else 0.0)
+ + self.packing * len(questions) * j
+ for j, (key, text) in enumerate(shown)
]
- p = softmax(z)
+ rows.setdefault(variant.name, []).append((softmax(z), variant.order))
+ answers = {}
+ for name, q in prepared.questions.items():
+ p = average_orders([r[0] for r in rows[name]], [r[1] for r in rows[name]])
if q.type == "score":
answers[name] = ScoreAnswer(
score=0.0,
@@ -80,13 +114,18 @@ def system_one(self, state, questions):
return SystemOneResponse(model="fake", answers=answers, usage=Usage(input_tokens=1))
+def run(engine, states, lang="en", kinds=KINDS, max_rows=MAX_REQUEST_ROWS, slot0_min=SLOT0_MIN):
+ return report(ask(engine, states, lang, kinds, max_rows), lang, kinds, slot0_min)
+
+
class TestProbeQuestions:
@pytest.mark.parametrize("lang", sorted(TEXT))
def test_every_probe_is_a_valid_request(self, lang):
questions = probe_questions(lang)
assert SystemOneRequest(state="s", questions=questions)
- per_kind = 2 * len(IDENTICAL_KS) + 6
- assert len(questions) == per_kind * len(KINDS)
+ score = 2 * len(IDENTICAL_KS) + 6
+ choice = 2 * sum(math.factorial(k) for k in IDENTICAL_KS) + 6
+ assert len(questions) == score + choice
@pytest.mark.parametrize("lang", sorted(TEXT))
def test_identical_options_differ_only_by_position(self, lang):
@@ -96,7 +135,20 @@ def test_identical_options_differ_only_by_position(self, lang):
texts = q["criteria"] if q["type"] == "score" else list(q["criteria"].values())
assert len(set(texts)) == 1
if q["type"] == "choice":
- assert list(q["criteria"]) == [str(i + 1) for i in range(len(q["criteria"]))]
+ assert set(q["criteria"]) <= set(KEYS)
+ assert not any(c.isdigit() for key in q["criteria"] for c in key)
+
+ def test_every_choice_key_sits_in_every_slot_equally_often(self):
+ for k in IDENTICAL_KS:
+ listings = [
+ list(q["criteria"])
+ for qid, q in probe_questions("en", ["choice"]).items()
+ if qid.startswith(f"choice|identical|a team|{k}|")
+ ]
+ assert len(listings) == math.factorial(k)
+ for slot in range(k):
+ counts = {key: sum(lst[slot] == key for lst in listings) for key in KEYS[:k]}
+ assert set(counts.values()) == {math.factorial(k - 1)}
@pytest.mark.parametrize("lang", sorted(TEXT))
def test_every_option_sits_in_every_slot_twice(self, lang):
@@ -113,46 +165,114 @@ def test_every_option_sits_in_every_slot_twice(self, lang):
class TestChecks:
def test_a_flat_readout_sits_at_zero_and_one_third(self):
- content = {"Soon": 3.0, "Technical": 3.0}
- report = run(FakeEngine(content=content).system_one, ["a", "b"])
+ content = {"Soon": 3.0, "payments, refunds, invoices": 3.0}
+ r = run(FakeEngine(content=content), ["a", "b"])
for kind in KINDS:
- assert report[kind]["identical"]["metric"] == pytest.approx(0.0)
- assert report[kind]["identical"]["passed"]
- assert report[kind]["first_slot"]["metric"] == pytest.approx(1 / 3)
+ assert r[kind]["identical"]["metric"] == pytest.approx(0.0)
+ assert r[kind]["identical"]["passed"]
+ assert r[kind]["first_slot"]["metric"] == pytest.approx(1 / 3)
+ assert r[kind]["first_slot"]["consistent"] == 1.0
def test_a_planted_slot_zero_penalty_is_measured_exactly(self):
- report = run(FakeEngine(slot0=-2.0).system_one, ["a", "b", "c"])
+ r = run(FakeEngine(slot0=-2.0, order_average=False), ["a", "b", "c"])
# centred slot 0 = -2 - (-2/K), averaged over K = 3, 4, 5 and both texts
expected = sum(-2.0 * (1 - 1 / k) for k in IDENTICAL_KS) / len(IDENTICAL_KS)
for kind in KINDS:
- assert report[kind]["identical"]["metric"] == pytest.approx(expected)
- assert not report[kind]["identical"]["passed"]
- assert report[kind]["first_slot"]["argmax_by_slot"][0] == 0
- assert report[kind]["first_slot"]["decisions"] == 18
+ assert r[kind]["identical"]["metric"] == pytest.approx(expected)
+ assert not r[kind]["identical"]["passed"]
+ assert r[kind]["first_slot"]["argmax_by_slot"][0] == 0
+ assert r[kind]["first_slot"]["decisions"] == 18
+
+ def test_ties_are_split_between_the_tied_slots(self):
+ """A flat readout ties all three slots; argmax would credit slot 0 every time."""
+ r = run(FakeEngine(), ["a", "b"])
+ for kind in KINDS:
+ assert r[kind]["first_slot"]["metric"] == pytest.approx(1 / 3)
+ assert r[kind]["first_slot"]["argmax_by_slot"] == pytest.approx([4.0, 4.0, 4.0])
+
+ def test_a_choice_key_pull_cancels_and_a_position_effect_does_not(self):
+ pulled = run(FakeEngine(key_pull={KEYS[0]: 2.0}), ["a"], kinds=["choice"])
+ assert pulled["choice"]["identical"]["metric"] == pytest.approx(0.0, abs=1e-12)
+ both = run(FakeEngine(key_pull={KEYS[0]: 2.0}, slot0=-1.0, order_average=False), ["a"])
+ alone = run(FakeEngine(slot0=-1.0, order_average=False), ["a"])
+ assert both["choice"]["identical"]["metric"] == pytest.approx(
+ alone["choice"]["identical"]["metric"]
+ )
- def test_one_call_per_state(self):
- engine = FakeEngine()
- run(engine.system_one, ["a", "b", "c"])
- assert engine.calls == 3
+ @pytest.mark.parametrize("mode", ["reversed", "cyclic"])
+ def test_the_identical_check_reads_one_order_whatever_the_engine_averages(self, mode):
+ """Averaged, a pure slot bias cancels (cyclic) or hides the middle slots (reversed)."""
+ single = run(FakeEngine(slot0=-2.0, order_average=False), ["a"])
+ averaged = run(FakeEngine(slot0=-2.0, score_order_average=mode), ["a"])
+ for kind in KINDS:
+ assert averaged[kind]["identical"]["metric"] == pytest.approx(
+ single[kind]["identical"]["metric"]
+ )
+ engine = FakeEngine(score_order_average=mode)
+ with single_order(engine):
+ assert engine.config.order_average is False
+ assert engine.config.order_average and engine.config.score_order_average == mode
def test_the_gate_is_the_proposal_and_can_be_moved(self):
- engine = FakeEngine(slot0=-0.3)
- assert not run(engine.system_one, ["a"])["score"]["identical"]["passed"]
- assert run(engine.system_one, ["a"], slot0_min=-1.0)["score"]["identical"]["passed"]
+ engine = FakeEngine(slot0=-0.3, order_average=False)
+ assert not run(engine, ["a"])["score"]["identical"]["passed"]
+ assert run(engine, ["a"], slot0_min=-1.0)["score"]["identical"]["passed"]
assert SLOT0_MIN == -0.20
+class TestCallGroups:
+ """Calls hold at most `max_rows` rows, counted by the engine for every question."""
+
+ @pytest.mark.parametrize("mode", ["off", "reversed", "cyclic"])
+ def test_every_call_fits_the_limit_and_asks_everything_once(self, mode):
+ engine = FakeEngine(score_order_average=mode)
+ questions = probe_questions("en")
+ groups = call_groups(engine, questions, 24)
+ rows = engine.question_rows(questions)
+ assert all(sum(rows[q] for q in g) <= 24 for g in groups)
+ assert sorted(q for g in groups for q in g) == sorted(questions)
+
+ def test_choice_rows_count(self):
+ engine = FakeEngine()
+ rows = engine.question_rows(probe_questions("en", ["choice"]))
+ assert set(rows.values()) == {2}
+ assert len(call_groups(engine, probe_questions("en", ["choice"]), 24)) > 1
+
+ def test_the_grouping_does_not_change_an_isolated_readout(self):
+ engine = FakeEngine(slot0=-1.0, content={"Soon": 1.0}, score_order_average="cyclic")
+ assert run(engine, ["a", "b"], max_rows=5) == run(engine, ["a", "b"])
+
+ def test_identical_and_permuted_questions_never_share_a_call(self):
+ engine = FakeEngine()
+ seen = []
+ real = engine.system_one
+ engine.system_one = lambda state, qs: seen.append(set(qs)) or real(state, qs)
+ ask(engine, ["a"])
+ for qids in seen:
+ assert len({q.split("|")[1] for q in qids}) == 1
+
+
class TestPackedConsistency:
def test_an_isolated_readout_agrees_with_itself_alone(self):
- assert packed_consistency(FakeEngine(slot0=-1.0).system_one, ["a"]) == pytest.approx(0.0)
+ engine = FakeEngine(slot0=-1.0)
+ answers = ask(engine, ["a"])
+ assert packed_consistency(engine, ["a"], answers) == pytest.approx(0.0)
def test_a_readout_that_depends_on_packing_is_caught(self):
- assert packed_consistency(FakeEngine(packing=0.01).system_one, ["a"]) > 0.01
+ engine = FakeEngine(packing=0.01)
+ assert packed_consistency(engine, ["a"], ask(engine, ["a"])) > 0.01
+
+ def test_the_packed_answers_are_reused_not_asked_again(self):
+ engine = FakeEngine()
+ answers = ask(engine, ["a", "b"])
+ before = engine.calls
+ packed_consistency(engine, ["a"], answers)
+ assert engine.calls - before == len(probe_questions("en"))
class TestReport:
def test_drift_flags_only_what_moved_past_the_tolerance(self):
- now = run(FakeEngine(slot0=-2.0).system_one, ["a"])
+ now = run(FakeEngine(slot0=-2.0), ["a"])
committed = json.loads(json.dumps(now))
assert drift(now, committed, 0.05) == []
committed["score"]["identical"]["metric"] += 0.2
@@ -164,7 +284,7 @@ def test_drift_flags_only_what_moved_past_the_tolerance(self):
assert "choice.identical: missing from this run" in drift(now, committed, 0.05)
def test_markdown_has_a_row_per_kind(self):
- text = to_markdown(run(FakeEngine().system_one, ["a"]))
+ text = to_markdown(run(FakeEngine(), ["a"]))
assert text.count("\n") == 1 + len(KINDS)
def test_states_are_read_from_jsonl(self, tmp_path):
@@ -203,11 +323,17 @@ def test_cyclic_is_every_rotation(self, rich_tokenizer):
[2, 0, 1],
]
- def test_independent_of_the_choice_switch(self, rich_tokenizer):
- q = Score(criteria=["low", "high"])
- assert self.orders(
- q, rich_tokenizer, score_order_average="reversed", order_average=False
- ) == [None, [1, 0]]
+ @pytest.mark.parametrize("mode", ["off", "reversed", "cyclic"])
+ def test_order_average_off_reads_every_question_once(self, rich_tokenizer, mode):
+ for q in (
+ Score(criteria=["low", "mid", "high"]),
+ Choice(criteria={"a": None, "b": None, "c": None}),
+ ):
+ assert self.orders(
+ q, rich_tokenizer, score_order_average=mode, order_average=False
+ ) == [None]
+
+ def test_choice_does_not_follow_the_score_switch(self, rich_tokenizer):
c = Choice(criteria={"a": None, "b": None, "c": None})
for mode in ("off", "reversed", "cyclic"):
assert self.orders(c, rich_tokenizer, score_order_average=mode) == [None, [2, 1, 0]]
@@ -224,67 +350,95 @@ def test_cyclic_averaging_cancels_a_slot_bias_on_equal_content(self):
"""Each level sits in each slot once, so a pure position bias averages out."""
k = 4
orders = [None] + [[(j + r) % k for j in range(k)] for r in range(1, k)]
- biased = softmax([2.0, 0.0, 0.0, 0.0]) # slot 0 favoured, content equal
+ biased = softmax([2.0, 0.0, 0.0, 0.0])
assert average_orders([biased] * k, orders) == pytest.approx([1 / k] * k)
-class TestCallGroups:
- """How the probe is split into calls: one by default, bounded Score rows under cyclic."""
+QUESTIONS = {
+ "team": Choice(criteria={"a": None, "b": None, "c": None}),
+ "urgency": Score(criteria=["low", "mid", "high", "blocked"]),
+ "churn": Noul(instructions="leaving?"),
+ "many": Choice(criteria={f"o{i}": None for i in range(40)}),
+}
+
+
+class TestQuestionRows:
+ """`question_rows` is what `prepare` renders, so callers never recount rows."""
+
+ @pytest.mark.parametrize("mode", ["off", "reversed", "cyclic"])
+ @pytest.mark.parametrize("order_average", [True, False])
+ @pytest.mark.parametrize("noul_readout", ["rating", "binary"])
+ def test_it_matches_prepare(self, poor_tokenizer, mode, order_average, noul_readout):
+ engine = DecisionEngine(
+ model=None,
+ tokenizer=poor_tokenizer,
+ config=EngineConfig(
+ score_order_average=mode,
+ order_average=order_average,
+ noul_readout=noul_readout,
+ max_request_rows=None,
+ ),
+ mode_b_head=object(),
+ )
+ rows = engine.question_rows(QUESTIONS)
+ assert sum(rows.values()) == engine.prepare("s", QUESTIONS).rows
+ assert rows["many"] == 1 # Mode B: one row whatever the averaging
- def rows(self, group, mode):
- from lev.presentation import score_rows
+ def test_schema_first_is_one_row_per_question(self, rich_tokenizer):
+ from lev.prompt import Layout
- return sum(score_rows(q, mode) for q in group.values() if q["type"] == "score")
+ engine = DecisionEngine(
+ model=None,
+ tokenizer=rich_tokenizer,
+ config=EngineConfig(layout=Layout.SCHEMA_FIRST, score_order_average="cyclic"),
+ mode_b_head=object(),
+ )
+ assert set(engine.question_rows(QUESTIONS).values()) == {1}
+ assert engine.prepare("s", QUESTIONS).rows == len(QUESTIONS)
- def test_off_and_reversed_stay_one_call(self):
- from lev.presentation import call_groups
- questions = probe_questions("en", ["score"])
- assert len(call_groups(questions)) == 1
- assert self.rows(questions, "reversed") == 24
- assert len(call_groups(questions, score_order_average="reversed")) == 1
+class TestRequestRowLimit:
+ def engine(self, tokenizer, **config) -> DecisionEngine:
+ return DecisionEngine(model=None, tokenizer=tokenizer, config=EngineConfig(**config))
- def test_cyclic_is_cut_at_the_row_limit_and_asks_everything_once(self):
- from lev.presentation import call_groups
+ def test_a_request_past_the_limit_is_refused_before_any_forward(self, rich_tokenizer):
+ questions = {f"q{i}": Score(criteria=["a", "b", "c"]) for i in range(4)}
+ engine = self.engine(rich_tokenizer, score_order_average="cyclic", max_request_rows=11)
+ with pytest.raises(ValueError, match="needs 12 batch rows, above max_request_rows=11"):
+ engine.prepare("s", questions)
+ assert self.engine(rich_tokenizer, max_request_rows=11).prepare("s", questions).rows == 4
- questions = probe_questions("en")
- assert self.rows(questions, "cyclic") == 42
- groups = call_groups(questions, score_order_average="cyclic")
- assert len(groups) > 1
- assert all(self.rows(g, "cyclic") <= 24 for g in groups)
- asked = [qid for g in groups for qid in g]
- assert sorted(asked) == sorted(questions)
-
- def test_split_checks_never_mixes_the_two_checks(self):
- from lev.presentation import call_groups
-
- for group in call_groups(probe_questions("ja"), split_checks=True):
- assert len({qid.split("|")[1] for qid in group}) == 1
-
- def test_the_split_does_not_change_an_isolated_readout(self):
- engine = FakeEngine(slot0=-1.0, content={"Soon": 1.0})
- one = run(engine.system_one, ["a", "b"])
- split = run(
- engine.system_one,
- ["a", "b"],
- split_checks=True,
- max_score_rows=5,
- score_order_average="cyclic",
- )
- assert split == one
- assert packed_consistency(engine.system_one, ["a"], split_checks=True) == pytest.approx(0.0)
+ def test_none_is_no_limit(self, rich_tokenizer):
+ questions = {f"q{i}": Score(criteria=["a", "b", "c"]) for i in range(30)}
+ engine = self.engine(rich_tokenizer, score_order_average="cyclic", max_request_rows=None)
+ assert engine.prepare("s", questions).rows == 90
+ assert EngineConfig().max_request_rows == MAX_REQUEST_ROWS
+
+ def test_the_server_answers_422(self, monkeypatch, rich_tokenizer):
+ pytest.importorskip("fastapi")
+ from fastapi.testclient import TestClient
+ from lev import server
+
+ engine = self.engine(rich_tokenizer, score_order_average="cyclic", max_request_rows=5)
+ monkeypatch.setattr(server, "load", lambda *a, **k: engine)
+ questions = {f"q{i}": {"type": "score", "criteria": ["a", "b", "c"]} for i in range(2)}
+ with TestClient(server.create_app()) as client:
+ response = client.post("/v1/systemone", json={"state": "s", "questions": questions})
+ assert response.status_code == 422
+ assert "max_request_rows=5" in response.json()["detail"]
class TestCommittedReport:
"""`data/presentation-checks.json` is what `--compare` reads in CI."""
def test_it_parses_and_carries_every_metric(self):
- from pathlib import Path
path = Path(__file__).resolve().parents[3] / "data" / "presentation-checks.json"
committed = json.loads(path.read_text("utf-8"))
assert committed["score_order_average"] == "off" and committed["n_states"] == 290
+ assert committed["max_rows"] == MAX_REQUEST_ROWS
for kind in KINDS:
+ assert committed["checks"][kind]["identical"]["read"] == "single order"
for check in ("identical", "first_slot"):
assert isinstance(committed["checks"][kind][check]["metric"], float)
assert drift(committed["checks"], committed["checks"], 0.0) == []
@@ -302,34 +456,55 @@ def cli(self, monkeypatch, tmp_path, engine, *extra):
states.write_text('{"state": "a"}\n{"state": "b"}\n', "utf-8")
main(["presentation-checks", str(states), "--packed", "0", *extra])
- def committed(self, monkeypatch, tmp_path, engine) -> dict:
+ def committed(self, monkeypatch, tmp_path, engine, **edit) -> str:
out = tmp_path / "report.json"
self.cli(monkeypatch, tmp_path, engine, "--out", str(out))
- return json.loads(out.read_text("utf-8"))
+ saved = json.loads(out.read_text("utf-8")) | edit
+ path = tmp_path / "committed.json"
+ path.write_text(json.dumps(saved), "utf-8")
+ return str(path)
+
+ def test_the_settings_split_at_the_load(self):
+ assert set(PRE_LOAD).isdisjoint(POST_LOAD)
+ assert set(POST_LOAD) == {"checkpoint_revision", "dtype"}
def test_matching_settings_are_compared_metric_by_metric(self, monkeypatch, tmp_path):
engine = FakeEngine(slot0=-1.0)
- saved = self.committed(monkeypatch, tmp_path, engine)
- assert saved["score_order_average"] == "off" and saved["max_score_rows"] == 24
+ path = Path(self.committed(monkeypatch, tmp_path, engine))
+ saved = json.loads(path.read_text("utf-8"))
+ assert saved["score_order_average"] == "off" and saved["max_rows"] == MAX_REQUEST_ROWS
saved["checks"]["score"]["identical"]["metric"] += 0.5
- path = tmp_path / "committed.json"
path.write_text(json.dumps(saved), "utf-8")
with pytest.raises(SystemExit) as exc:
self.cli(monkeypatch, tmp_path, engine, "--compare", str(path))
assert "score.identical" in str(exc.value.code)
- def test_different_settings_are_refused_before_any_run(self, monkeypatch, tmp_path, capsys):
- engine = FakeEngine(slot0=-1.0)
- saved = self.committed(monkeypatch, tmp_path, engine)
- saved["score_order_average"] = "reversed"
- path = tmp_path / "committed.json"
- path.write_text(json.dumps(saved), "utf-8")
- calls = engine.calls
+ def test_a_model_free_setting_is_refused_before_the_load(self, monkeypatch, tmp_path, capsys):
+ path = self.committed(monkeypatch, tmp_path, FakeEngine(), score_order_average="reversed")
+
+ def no_load(*a, **k):
+ raise AssertionError("loaded a model for a run it had to refuse")
+
+ monkeypatch.setattr("lev.model.load", no_load)
+ from lev.cli import main
+
+ states = tmp_path / "states.jsonl"
with pytest.raises(SystemExit) as exc:
- self.cli(monkeypatch, tmp_path, engine, "--compare", str(path))
+ main(["presentation-checks", str(states), "--packed", "0", "--compare", path])
assert exc.value.code == 2
- assert engine.calls == calls
assert (
"presentation settings differ from the committed report: "
"score_order_average (report: 'reversed', run: 'off')"
) in capsys.readouterr().err
+
+ def test_another_revision_is_refused_after_the_load_and_before_any_run(
+ self, monkeypatch, tmp_path, capsys
+ ):
+ engine = FakeEngine()
+ path = self.committed(monkeypatch, tmp_path, engine, checkpoint_revision="abc123")
+ calls = engine.calls
+ with pytest.raises(SystemExit) as exc:
+ self.cli(monkeypatch, tmp_path, engine, "--compare", str(path))
+ assert exc.value.code == 2
+ assert engine.calls == calls
+ assert "checkpoint_revision (report: 'abc123', run: None)" in capsys.readouterr().err
From 0e5fc6e32017024baef3213df7278d4247a6acf2 Mon Sep 17 00:00:00 2001
From: hiroki-abe-58
Date: Mon, 5 Oct 2026 05:36:30 +0900
Subject: [PATCH 06/39] ADR-029 evidence after review: one-read bias, labelled
and calibration effects, row limit
Re-run 2026-10-04/05 on interfaze-ai/lev (HF 7bdc748, the weights of f8ef711), bf16, one
RTX 5090, on this branch rebased after #4. The identical-option control is reported as the
bias of one read, which no mode changes; what averaging changes is shown by Score accuracy,
RPS and ECE, the first-slot rate with ties split, and order consistency. A held-out refit of
the Score temperature per mode, the row-limit memory table, and a regenerated committed
report (max_rows 32) are included. Accuracy, RPS, Choice and Noul match the earlier runs
exactly.
Co-Authored-By: Claude Opus 5.5
Claude-Session: https://claude.ai/code/session_01VyLpqwg29AwAk4YfyeTqxP
---
data/adr-029-summary.json | 712 +++++++++++++++++++++++++++++++---
data/presentation-checks.json | 73 ++--
docs/ARCHITECTURE.md | 2 +-
docs/DECISIONS.md | 70 +++-
4 files changed, 754 insertions(+), 103 deletions(-)
diff --git a/data/adr-029-summary.json b/data/adr-029-summary.json
index 58c6e72..4638f94 100644
--- a/data/adr-029-summary.json
+++ b/data/adr-029-summary.json
@@ -1,61 +1,321 @@
{
- "about": "Summary of the ADR-029 runs: the numbers behind its tables. interfaze-ai/lev (HF f8ef711), bf16, one RTX 5090, 2026-09-29. Raw per-run reports are not committed and are available on request.",
+ "about": "Summary of the ADR-029 runs after review (2026-10-04/05): the numbers behind its tables. interfaze-ai/lev (HF 7bdc748; weights and calibration identical to f8ef711, only README.md changed), bf16, one RTX 5090, lev branch rebased on main 8ca94b7 (#4).",
"benches": {
"bench_en": "sokudan data/bench_en.jsonl, 290 states (English)",
"bench_ja": "sokudan data/bench_ja.jsonl, 300 states (Japanese, outside Lev's target language; reference only)"
},
- "score_presentation": {
+ "shipped_temperatures": {
+ "choice:B:large": 0.6333,
+ "choice:B": 0.6333,
+ "noul:A": 2.3333,
+ "score:A": 2.8033,
+ "choice:A": 1.7667,
+ "choice:A:small": 1.7898,
+ "choice:A:mid": 1.6086,
+ "choice:A:large": 1.6641
+ },
+ "identical_one_read": {
+ "bench_en": {
+ "score": {
+ "calibrated": -0.3323,
+ "leave_one_out": [
+ -0.3364,
+ -0.3298
+ ],
+ "raw": -0.9317,
+ "by_config_calibrated": {
+ "moderate|3": -0.1191,
+ "moderate|4": -0.2373,
+ "moderate|5": -0.3198,
+ "a request|3": -0.3249,
+ "a request|4": -0.4614,
+ "a request|5": -0.5315
+ },
+ "by_config_raw": {
+ "moderate|3": -0.3338,
+ "moderate|4": -0.6652,
+ "moderate|5": -0.8966,
+ "a request|3": -0.9108,
+ "a request|4": -1.2935,
+ "a request|5": -1.49
+ },
+ "max_abs_diff_across_modes": 0.0
+ },
+ "choice": {
+ "calibrated": -0.454,
+ "leave_one_out": [
+ -0.4556,
+ -0.4515
+ ],
+ "raw": -0.8125,
+ "by_config_calibrated": {
+ "a team|3": -0.5365,
+ "a team|4": -0.534,
+ "a team|5": -0.5335,
+ "a request|3": -0.323,
+ "a request|4": -0.3803,
+ "a request|5": -0.4165
+ },
+ "by_config_raw": {
+ "a team|3": -0.9601,
+ "a team|4": -0.9558,
+ "a team|5": -0.9548,
+ "a request|3": -0.5781,
+ "a request|4": -0.6807,
+ "a request|5": -0.7455
+ },
+ "max_abs_diff_across_modes": 0.0
+ }
+ },
+ "bench_ja": {
+ "score": {
+ "calibrated": -0.5181,
+ "leave_one_out": [
+ -0.5222,
+ -0.5151
+ ],
+ "raw": -1.4524,
+ "by_config_calibrated": {
+ "中程度|3": -0.1889,
+ "中程度|4": -0.2989,
+ "中程度|5": -0.3897,
+ "依頼|3": -0.5875,
+ "依頼|4": -0.7656,
+ "依頼|5": -0.8779
+ },
+ "by_config_raw": {
+ "中程度|3": -0.5294,
+ "中程度|4": -0.838,
+ "中程度|5": -1.0926,
+ "依頼|3": -1.647,
+ "依頼|4": -2.1462,
+ "依頼|5": -2.4609
+ },
+ "max_abs_diff_across_modes": 0.0
+ },
+ "choice": {
+ "calibrated": -0.6602,
+ "leave_one_out": [
+ -0.6624,
+ -0.6577
+ ],
+ "raw": -1.1817,
+ "by_config_calibrated": {
+ "部署|3": -0.611,
+ "部署|4": -0.7262,
+ "部署|5": -0.7265,
+ "依頼|3": -0.5278,
+ "依頼|4": -0.6493,
+ "依頼|5": -0.7206
+ },
+ "by_config_raw": {
+ "部署|3": -1.0936,
+ "部署|4": -1.2998,
+ "部署|5": -1.3002,
+ "依頼|3": -0.9447,
+ "依頼|4": -1.162,
+ "依頼|5": -1.2897
+ },
+ "max_abs_diff_across_modes": 0.0
+ }
+ }
+ },
+ "first_slot": {
"bench_en": {
"off": {
- "identical_calibrated": -0.3321,
- "identical_raw": -0.931,
- "first_slot": 0.2695,
- "first_slot_raw": 0.2695
+ "score": {
+ "metric": 0.2646,
+ "leave_one_out": [
+ 0.2641,
+ 0.2655
+ ],
+ "argmax_by_slot": [
+ 460.3333,
+ 595.8333,
+ 683.8333
+ ],
+ "consistent": 0.7138,
+ "decisions": 1740
+ },
+ "choice": {
+ "metric": 0.331,
+ "leave_one_out": [
+ 0.331,
+ 0.3319
+ ],
+ "argmax_by_slot": [
+ 576.0,
+ 588.0,
+ 576.0
+ ],
+ "consistent": 0.9862,
+ "decisions": 1740
+ }
},
"reversed": {
- "identical_calibrated": -0.0101,
- "identical_raw": 0.0878,
- "first_slot": 0.3207,
- "first_slot_raw": 0.3207
+ "score": {
+ "metric": 0.3191,
+ "leave_one_out": [
+ 0.3184,
+ 0.3202
+ ],
+ "argmax_by_slot": [
+ 555.1667,
+ 575.6667,
+ 609.1667
+ ],
+ "consistent": 0.8724,
+ "decisions": 1740
+ },
+ "choice": {
+ "metric": 0.3317,
+ "leave_one_out": [
+ 0.3317,
+ 0.3323
+ ],
+ "argmax_by_slot": [
+ 577.1667,
+ 585.6667,
+ 577.1667
+ ],
+ "consistent": 0.9793,
+ "decisions": 1740
+ }
},
"cyclic": {
- "identical_calibrated": -0.0161,
- "identical_raw": -0.0365,
- "first_slot": 0.3126,
- "first_slot_raw": 0.3121
+ "score": {
+ "metric": 0.3101,
+ "leave_one_out": [
+ 0.3091,
+ 0.3111
+ ],
+ "argmax_by_slot": [
+ 539.5,
+ 590.5,
+ 610.0
+ ],
+ "consistent": 0.8931,
+ "decisions": 1740
+ },
+ "choice": {
+ "metric": 0.332,
+ "leave_one_out": [
+ 0.3314,
+ 0.3328
+ ],
+ "argmax_by_slot": [
+ 577.6667,
+ 584.6667,
+ 577.6667
+ ],
+ "consistent": 0.9828,
+ "decisions": 1740
+ }
}
},
"bench_ja": {
"off": {
- "identical_calibrated": -0.5184,
- "identical_raw": -1.4532,
- "first_slot": 0.24,
- "first_slot_raw": 0.24
+ "score": {
+ "metric": 0.2422,
+ "leave_one_out": [
+ 0.2414,
+ 0.243
+ ],
+ "argmax_by_slot": [
+ 436.0,
+ 631.5,
+ 732.5
+ ],
+ "consistent": 0.6267,
+ "decisions": 1800
+ },
+ "choice": {
+ "metric": 0.3311,
+ "leave_one_out": [
+ 0.3305,
+ 0.3322
+ ],
+ "argmax_by_slot": [
+ 596.0,
+ 608.0,
+ 596.0
+ ],
+ "consistent": 0.9733,
+ "decisions": 1800
+ }
},
"reversed": {
- "identical_calibrated": -0.0586,
- "identical_raw": 0.038,
- "first_slot": 0.315,
- "first_slot_raw": 0.3139
+ "score": {
+ "metric": 0.3142,
+ "leave_one_out": [
+ 0.3135,
+ 0.3152
+ ],
+ "argmax_by_slot": [
+ 565.5,
+ 606.0,
+ 628.5
+ ],
+ "consistent": 0.87,
+ "decisions": 1800
+ },
+ "choice": {
+ "metric": 0.3306,
+ "leave_one_out": [
+ 0.3305,
+ 0.3317
+ ],
+ "argmax_by_slot": [
+ 595.0,
+ 610.0,
+ 595.0
+ ],
+ "consistent": 0.9767,
+ "decisions": 1800
+ }
},
"cyclic": {
- "identical_calibrated": -0.0558,
- "identical_raw": -0.1313,
- "first_slot": 0.3017,
- "first_slot_raw": 0.3006
+ "score": {
+ "metric": 0.2994,
+ "leave_one_out": [
+ 0.2993,
+ 0.3004
+ ],
+ "argmax_by_slot": [
+ 539.0,
+ 619.0,
+ 642.0
+ ],
+ "consistent": 0.8533,
+ "decisions": 1800
+ },
+ "choice": {
+ "metric": 0.33,
+ "leave_one_out": [
+ 0.3294,
+ 0.3311
+ ],
+ "argmax_by_slot": [
+ 594.0,
+ 612.0,
+ 594.0
+ ],
+ "consistent": 0.9733,
+ "decisions": 1800
+ }
}
}
},
- "choice_presentation_off": {
+ "packed_max_abs_diff": {
"bench_en": {
- "identical_calibrated": -0.1001,
- "identical_raw": -0.1597,
- "first_slot": 0.3333
+ "off": 0.0375,
+ "reversed": 0.0375,
+ "cyclic": 0.0375
},
"bench_ja": {
- "identical_calibrated": -0.0547,
- "identical_raw": -0.0362,
- "first_slot": 0.3306
+ "off": 0.0408,
+ "reversed": 0.0408,
+ "cyclic": 0.0408
}
},
"bench_accuracy": {
@@ -63,60 +323,202 @@
"off": {
"score_acc": 0.5828,
"score_rps": 0.1504,
+ "score_ece15": 0.1238,
+ "score_ece15_equal_mass": 0.1545,
+ "score_mean_confidence": 0.6814,
"choice_acc": 0.9172,
"noul_acc": 0.9379,
"noul_auroc": 0.9778,
- "ms_per_request": 164.6
+ "ms_per_request": 156.5
},
"reversed": {
"score_acc": 0.6517,
"score_rps": 0.1275,
+ "score_ece15": 0.1013,
+ "score_ece15_equal_mass": 0.1066,
+ "score_mean_confidence": 0.6594,
"choice_acc": 0.9172,
"noul_acc": 0.9379,
"noul_auroc": 0.978,
- "ms_per_request": 184.0
+ "ms_per_request": 179.3
},
"cyclic": {
"score_acc": 0.6414,
"score_rps": 0.1249,
+ "score_ece15": 0.0601,
+ "score_ece15_equal_mass": 0.0912,
+ "score_mean_confidence": 0.6453,
"choice_acc": 0.9172,
"noul_acc": 0.9379,
"noul_auroc": 0.9784,
- "ms_per_request": 215.4
+ "ms_per_request": 210.2
}
},
"bench_ja": {
"off": {
"score_acc": 0.6433,
"score_rps": 0.1384,
+ "score_ece15": 0.0934,
+ "score_ece15_equal_mass": 0.113,
+ "score_mean_confidence": 0.7184,
"choice_acc": 0.8867,
"noul_acc": 0.8167,
"noul_auroc": 0.889,
- "ms_per_request": 150.9
+ "ms_per_request": 149.2
},
"reversed": {
"score_acc": 0.73,
"score_rps": 0.1079,
+ "score_ece15": 0.0588,
+ "score_ece15_equal_mass": 0.1072,
+ "score_mean_confidence": 0.6829,
"choice_acc": 0.8867,
"noul_acc": 0.8133,
"noul_auroc": 0.8877,
- "ms_per_request": 171.4
+ "ms_per_request": 169.9
},
"cyclic": {
"score_acc": 0.7167,
"score_rps": 0.108,
+ "score_ece15": 0.0765,
+ "score_ece15_equal_mass": 0.0879,
+ "score_mean_confidence": 0.6813,
"choice_acc": 0.8867,
"noul_acc": 0.8133,
"noul_auroc": 0.8876,
- "ms_per_request": 199.9
+ "ms_per_request": 197.3
}
}
},
- "noul_alone_bench_ja": {
- "requests": 300,
- "identical_across_modes": true
+ "score_temperature_refit": {
+ "bench_en": {
+ "split": "seed 0, stratified by gold level, alternate halves",
+ "off": {
+ "n_fit": 146,
+ "n_eval": 144,
+ "T_shipped": 2.8033,
+ "T_refit": 3.9628,
+ "held_out_shipped": {
+ "acc": 0.5903,
+ "rps": 0.152,
+ "ece15": 0.1296,
+ "ece15_equal_mass": 0.1683,
+ "nll": 0.9348
+ },
+ "held_out_refit": {
+ "acc": 0.5903,
+ "rps": 0.1466,
+ "ece15": 0.108,
+ "ece15_equal_mass": 0.134,
+ "nll": 0.9087
+ }
+ },
+ "reversed": {
+ "n_fit": 146,
+ "n_eval": 144,
+ "T_shipped": 2.8033,
+ "T_refit": 3.1595,
+ "held_out_shipped": {
+ "acc": 0.6736,
+ "rps": 0.1264,
+ "ece15": 0.1108,
+ "ece15_equal_mass": 0.1453,
+ "nll": 0.8388
+ },
+ "held_out_refit": {
+ "acc": 0.6736,
+ "rps": 0.1263,
+ "ece15": 0.1193,
+ "ece15_equal_mass": 0.1668,
+ "nll": 0.8381
+ }
+ },
+ "cyclic": {
+ "n_fit": 146,
+ "n_eval": 144,
+ "T_shipped": 2.8033,
+ "T_refit": 2.9175,
+ "held_out_shipped": {
+ "acc": 0.6597,
+ "rps": 0.1238,
+ "ece15": 0.0775,
+ "ece15_equal_mass": 0.1387,
+ "nll": 0.8279
+ },
+ "held_out_refit": {
+ "acc": 0.6597,
+ "rps": 0.1238,
+ "ece15": 0.0765,
+ "ece15_equal_mass": 0.1382,
+ "nll": 0.8284
+ }
+ }
+ },
+ "bench_ja": {
+ "split": "seed 0, stratified by gold level, alternate halves",
+ "off": {
+ "n_fit": 151,
+ "n_eval": 149,
+ "T_shipped": 2.8033,
+ "T_refit": 3.5967,
+ "held_out_shipped": {
+ "acc": 0.6577,
+ "rps": 0.1321,
+ "ece15": 0.0896,
+ "ece15_equal_mass": 0.1275,
+ "nll": 0.7807
+ },
+ "held_out_refit": {
+ "acc": 0.6577,
+ "rps": 0.13,
+ "ece15": 0.0772,
+ "ece15_equal_mass": 0.0914,
+ "nll": 0.7792
+ }
+ },
+ "reversed": {
+ "n_fit": 151,
+ "n_eval": 149,
+ "T_shipped": 2.8033,
+ "T_refit": 2.3927,
+ "held_out_shipped": {
+ "acc": 0.7584,
+ "rps": 0.1015,
+ "ece15": 0.0888,
+ "ece15_equal_mass": 0.1219,
+ "nll": 0.641
+ },
+ "held_out_refit": {
+ "acc": 0.7584,
+ "rps": 0.1008,
+ "ece15": 0.0611,
+ "ece15_equal_mass": 0.1142,
+ "nll": 0.6267
+ }
+ },
+ "cyclic": {
+ "n_fit": 151,
+ "n_eval": 149,
+ "T_shipped": 2.8033,
+ "T_refit": 2.3974,
+ "held_out_shipped": {
+ "acc": 0.7517,
+ "rps": 0.1013,
+ "ece15": 0.0681,
+ "ece15_equal_mass": 0.0959,
+ "nll": 0.6436
+ },
+ "held_out_refit": {
+ "acc": 0.7517,
+ "rps": 0.1006,
+ "ece15": 0.0865,
+ "ece15_equal_mass": 0.0909,
+ "nll": 0.6286
+ }
+ }
+ }
},
- "off_vs_main_cf104b6": {
+ "off_vs_main_8ca94b7": {
"bench_en": {
"responses": 40,
"identical": true
@@ -126,9 +528,225 @@
"identical": true
}
},
- "issue_1_reproduced": {
- "identical_calibrated": -0.3321,
- "identical_raw": -0.931,
- "first_slot": 0.2695
+ "request_rows_rtx5090": {
+ "what": "one system_one request of R single-order Score rows; peak CUDA memory in GiB (bf16 weights alone are ~8.1). Past ~31 GiB reserved, Windows spills into shared system memory instead of raising out-of-memory, and the request slows 5-10x.",
+ "bench_row_tokens": {
+ "bench_en": {
+ "n": 290,
+ "median": 259,
+ "p95": 449,
+ "max": 509
+ },
+ "bench_ja": {
+ "n": 300,
+ "median": 249,
+ "p95": 419,
+ "max": 526
+ }
+ },
+ "results": [
+ {
+ "state_tokens": 350,
+ "rows": 8,
+ "width": 474,
+ "padded_tokens": 3792,
+ "seconds": 0.376,
+ "peak_allocated_gib": 9.05,
+ "above_weights_gib": 0.94,
+ "peak_reserved_gib": 9.66
+ },
+ {
+ "state_tokens": 350,
+ "rows": 16,
+ "width": 474,
+ "padded_tokens": 7584,
+ "seconds": 0.782,
+ "peak_allocated_gib": 9.99,
+ "above_weights_gib": 1.88,
+ "peak_reserved_gib": 11.21
+ },
+ {
+ "state_tokens": 350,
+ "rows": 24,
+ "width": 474,
+ "padded_tokens": 11376,
+ "seconds": 1.194,
+ "peak_allocated_gib": 10.93,
+ "above_weights_gib": 2.81,
+ "peak_reserved_gib": 12.71
+ },
+ {
+ "state_tokens": 350,
+ "rows": 32,
+ "width": 474,
+ "padded_tokens": 15168,
+ "seconds": 1.598,
+ "peak_allocated_gib": 11.86,
+ "above_weights_gib": 3.75,
+ "peak_reserved_gib": 14.24
+ },
+ {
+ "state_tokens": 350,
+ "rows": 48,
+ "width": 474,
+ "padded_tokens": 22752,
+ "seconds": 2.425,
+ "peak_allocated_gib": 13.74,
+ "above_weights_gib": 5.63,
+ "peak_reserved_gib": 17.29
+ },
+ {
+ "state_tokens": 350,
+ "rows": 64,
+ "width": 474,
+ "padded_tokens": 30336,
+ "seconds": 3.277,
+ "peak_allocated_gib": 15.61,
+ "above_weights_gib": 7.5,
+ "peak_reserved_gib": 20.34
+ },
+ {
+ "state_tokens": 350,
+ "rows": 96,
+ "width": 474,
+ "padded_tokens": 45504,
+ "seconds": 5.254,
+ "peak_allocated_gib": 19.36,
+ "above_weights_gib": 11.25,
+ "peak_reserved_gib": 26.44
+ },
+ {
+ "state_tokens": 700,
+ "rows": 8,
+ "width": 824,
+ "padded_tokens": 6592,
+ "seconds": 0.696,
+ "peak_allocated_gib": 9.65,
+ "above_weights_gib": 1.54,
+ "peak_reserved_gib": 11.18
+ },
+ {
+ "state_tokens": 700,
+ "rows": 16,
+ "width": 824,
+ "padded_tokens": 13184,
+ "seconds": 1.389,
+ "peak_allocated_gib": 11.19,
+ "above_weights_gib": 3.07,
+ "peak_reserved_gib": 14.49
+ },
+ {
+ "state_tokens": 700,
+ "rows": 24,
+ "width": 824,
+ "padded_tokens": 19776,
+ "seconds": 2.098,
+ "peak_allocated_gib": 12.73,
+ "above_weights_gib": 4.61,
+ "peak_reserved_gib": 17.68
+ },
+ {
+ "state_tokens": 700,
+ "rows": 32,
+ "width": 824,
+ "padded_tokens": 26368,
+ "seconds": 2.813,
+ "peak_allocated_gib": 14.26,
+ "above_weights_gib": 6.15,
+ "peak_reserved_gib": 21.05
+ },
+ {
+ "state_tokens": 700,
+ "rows": 48,
+ "width": 824,
+ "padded_tokens": 39552,
+ "seconds": 4.317,
+ "peak_allocated_gib": 17.33,
+ "above_weights_gib": 9.22,
+ "peak_reserved_gib": 27.51
+ },
+ {
+ "state_tokens": 1000,
+ "rows": 8,
+ "width": 1124,
+ "padded_tokens": 8992,
+ "seconds": 0.947,
+ "peak_allocated_gib": 10.39,
+ "above_weights_gib": 2.28,
+ "peak_reserved_gib": 12.9
+ },
+ {
+ "state_tokens": 1000,
+ "rows": 16,
+ "width": 1124,
+ "padded_tokens": 17984,
+ "seconds": 1.946,
+ "peak_allocated_gib": 12.67,
+ "above_weights_gib": 4.56,
+ "peak_reserved_gib": 17.62
+ },
+ {
+ "state_tokens": 1000,
+ "rows": 24,
+ "width": 1124,
+ "padded_tokens": 26976,
+ "seconds": 2.925,
+ "peak_allocated_gib": 14.95,
+ "above_weights_gib": 6.84,
+ "peak_reserved_gib": 22.38
+ },
+ {
+ "state_tokens": 1000,
+ "rows": 32,
+ "width": 1124,
+ "padded_tokens": 35968,
+ "seconds": 3.916,
+ "peak_allocated_gib": 17.22,
+ "above_weights_gib": 9.11,
+ "peak_reserved_gib": 27.12
+ },
+ {
+ "state_tokens": 2000,
+ "rows": 8,
+ "width": 2124,
+ "padded_tokens": 16992,
+ "seconds": 1.956,
+ "peak_allocated_gib": 14.71,
+ "above_weights_gib": 6.6,
+ "peak_reserved_gib": 19.12
+ },
+ {
+ "state_tokens": 2000,
+ "rows": 16,
+ "width": 2124,
+ "padded_tokens": 33984,
+ "seconds": 12.348,
+ "peak_allocated_gib": 21.29,
+ "above_weights_gib": 13.18,
+ "peak_reserved_gib": 30.12
+ },
+ {
+ "state_tokens": 4000,
+ "rows": 8,
+ "width": 4124,
+ "padded_tokens": 32992,
+ "seconds": 42.186,
+ "peak_allocated_gib": 29.8,
+ "above_weights_gib": 21.69,
+ "peak_reserved_gib": 39.3
+ }
+ ],
+ "first_run_past_the_card": [
+ {
+ "state_tokens": 350,
+ "rows": 128,
+ "width": 474,
+ "padded_tokens": 60672,
+ "seconds": 59.979,
+ "peak_allocated_gib": 23.11,
+ "above_weights_gib": 15.0,
+ "peak_reserved_gib": 32.53
+ }
+ ]
}
}
diff --git a/data/presentation-checks.json b/data/presentation-checks.json
index aa3f4b3..748a9f5 100644
--- a/data/presentation-checks.json
+++ b/data/presentation-checks.json
@@ -1,5 +1,5 @@
{
- "about": "Committed report for `lev presentation-checks --compare` (ADR-029). One run: interfaze-ai/lev (HF f8ef711), bf16, one RTX 5090, the shipped calibration, score_order_average off, kinds score and choice, lang en, 2026-09-29.",
+ "about": "Committed report for `lev presentation-checks --compare` (ADR-029). One run: interfaze-ai/lev (HF 7bdc748), bf16, one RTX 5090, the shipped calibration, score_order_average off, kinds score and choice, lang en, 2026-10-05. The identical check reads one order.",
"checkpoint": "interfaze-ai/lev",
"states": "sokudan bench_en.jsonl (github.com/hiroki-abe-58/sokudan, data/bench_en.jsonl, CC BY 4.0), 290 states",
"n_states": 290,
@@ -10,78 +10,81 @@
"choice"
],
"score_order_average": "off",
- "split_checks": false,
- "max_score_rows": 24,
- "checkpoint_revision": "f8ef71157ec06a7d3b6435bc0756f9d735c33748",
+ "max_rows": 32,
+ "checkpoint_revision": "7bdc748dffebd85b57ee0dbea8f994c6354fed31",
"dtype": "torch.bfloat16",
- "packed_max_abs_diff": 0.0207,
+ "packed_max_abs_diff": 0.0375,
"checks": {
"score": {
"identical": {
"metric": -0.3323,
"leave_one_out": [
- -0.3362,
+ -0.3364,
-0.3298
],
"gate": -0.2,
"passed": false,
+ "read": "single order",
"by_config": {
- "moderate|3": -0.1182,
- "moderate|4": -0.2365,
- "moderate|5": -0.3193,
+ "moderate|3": -0.1191,
+ "moderate|4": -0.2373,
+ "moderate|5": -0.3198,
"a request|3": -0.3249,
- "a request|4": -0.4617,
- "a request|5": -0.5332
+ "a request|4": -0.4614,
+ "a request|5": -0.5315
}
},
"first_slot": {
- "metric": 0.2713,
+ "metric": 0.2646,
"leave_one_out": [
- 0.2699,
- 0.2722
+ 0.2641,
+ 0.2655
],
"gate": 0.15,
"passed": true,
"decisions": 1740,
"argmax_by_slot": [
- 472,
- 598,
- 670
- ]
+ 460.3333,
+ 595.8333,
+ 683.8333
+ ],
+ "consistent": 0.7138
}
},
"choice": {
"identical": {
- "metric": -0.1006,
+ "metric": -0.454,
"leave_one_out": [
- -0.1024,
- -0.0991
+ -0.4556,
+ -0.4515
],
"gate": -0.2,
- "passed": true,
+ "passed": false,
+ "read": "single order",
"by_config": {
- "a team|3": -0.0569,
- "a team|4": -0.1228,
- "a team|5": -0.2804,
- "a request|3": 0.0892,
- "a request|4": -0.0295,
- "a request|5": -0.2033
+ "a team|3": -0.5365,
+ "a team|4": -0.534,
+ "a team|5": -0.5335,
+ "a request|3": -0.323,
+ "a request|4": -0.3803,
+ "a request|5": -0.4165
}
},
"first_slot": {
- "metric": 0.3333,
+ "metric": 0.331,
"leave_one_out": [
- 0.3328,
- 0.3339
+ 0.331,
+ 0.3319
],
"gate": 0.15,
"passed": true,
"decisions": 1740,
"argmax_by_slot": [
- 580,
- 581,
- 579
- ]
+ 576.0,
+ 588.0,
+ 576.0
+ ],
+ "consistent": 0.9862
}
}
}
diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md
index b14ecc3..1d2d356 100644
--- a/docs/ARCHITECTURE.md
+++ b/docs/ARCHITECTURE.md
@@ -201,7 +201,7 @@ LoRA r32 on `q,k,v,o,gate,up,down` ≈ 25–40 M trainable parameters. **The Fam
- they get **separate fitted temperatures** — one global scalar would be wrong for at least one of them;
- on option sets where **both** are valid we measure agreement. Material disagreement means the router is a correctness hazard, not merely a capacity switch. This is an explicit eval, not an assumption.
-**Order averaging** (added in the build, not in the original design). Under Mode A with the state-first layout, Choice and binary Noul are read in their shown order and reversed, and the two distributions are averaged in canonical order (ADR-020). Score keeps its level order by default; `EngineConfig.score_order_average` opts it into the reversed order or every rotation, at 2x or Kx rows, and `lev presentation-checks` measures what that changes (ADR-029).
+**Order averaging** (added in the build, not in the original design). Under Mode A with the state-first layout, Choice and binary Noul are read in their shown order and reversed, and the two distributions are averaged in canonical order (ADR-020). Score keeps its level order by default; `EngineConfig.score_order_average` opts it into the reversed order or every rotation, at 2x or Kx rows, and `order_average=False` reads every question once. A request may need at most `max_request_rows` batch rows (32 by default); `prepare` refuses a larger one before any GPU work. `lev presentation-checks` measures the position bias of one read, the bias averaging has to remove (ADR-029).
### 5.4 Output types
diff --git a/docs/DECISIONS.md b/docs/DECISIONS.md
index b6e498c..f557d4e 100644
--- a/docs/DECISIONS.md
+++ b/docs/DECISIONS.md
@@ -601,39 +601,69 @@ Serving now skips by default; training does not, so Mode B keeps its data. Mode
Every result is reported whichever way it goes, including accuracy that does not improve.
-**Evidence.** Run 2026-09-29 on `interfaze-ai/lev` (HF f8ef711), bf16, one RTX 5090. The numbers behind every table are in [`data/adr-029-summary.json`](../data/adr-029-summary.json), and the raw per-run reports are available on request. The committed report CI compares against is [`data/presentation-checks.json`](../data/presentation-checks.json). `bench_ja` is Japanese, outside Lev's target language, so its rows are a reference only; `bench_en` is the one that bears on the decision.
+**Revised after review (2026-10-02).** The first runs read the identical-option control after averaging, where `cyclic` makes it zero by construction and `reversed` cannot see the middle slots, so it was no evidence for averaging. The control now measures the bias of one read, before any averaging (`order_average=False`); what averaging changes is judged by labelled accuracy, RPS and calibration error, and by the first-slot rate on real options. Also changed: Choice keys are shapes with no order (■ ● ◆ ▲ ◎), read under every key-to-slot assignment so a key's own pull cancels; argmax ties split between the tied slots; calls are cut by the engine's own row count; and the engine bounds the rows one request may need.
-Issue #1 reproduced with averaging off (Score probe alone, 12 questions per call): identical control -0.332 calibrated / -0.931 raw, first-slot rate 0.270. With averaging off, every answer on 20 items of each bench, the probe included, is identical to the current code.
+**Evidence.** Run 2026-10-04/05 on `interfaze-ai/lev` (HF 7bdc748: the weights and calibration of f8ef711, only README.md changed), bf16, one RTX 5090, this branch rebased on main after #4. The numbers behind every table are in [`data/adr-029-summary.json`](../data/adr-029-summary.json); the committed report CI compares against is [`data/presentation-checks.json`](../data/presentation-checks.json). `bench_ja` is Japanese, outside Lev's target language, so its rows are a reference only; `bench_en` is the one that bears on the decision.
-Score, identical control (slot 0 minus the mean; gate -0.20 raw) and first-slot rate (1/3 is order-free):
+The bias of one read: identical-option control, slot 0 minus the mean (gate -0.20 raw). Raw is calibrated x T, exact within one temperature bucket. The three runs per bench, one per mode, agree exactly, as they must.
```text
- bench_en (290) bench_ja (300, reference)
- calibrated raw first slot calibrated raw first slot
- off -0.332 -0.931 0.270 -0.518 -1.453 0.240
- reversed -0.010 +0.088 0.321 -0.059 +0.038 0.315
- cyclic -0.016 -0.037 0.312 -0.056 -0.131 0.301
+ bench_en (290) bench_ja (300, reference)
+ calibrated raw calibrated raw
+ Score -0.332 -0.932 -0.518 -1.452
+ Choice -0.454 -0.813 -0.660 -1.182
```
-Choice, which is already read in two orders, with Score averaging off: identical control -0.100 / -0.160 (en) and -0.055 / -0.036 (ja), first-slot rate 0.333 and 0.331.
+Choice, read once, avoids slot 0 more than Score does; it is already read in two orders when served, and the first-slot rate below is what that leaves.
-The benches' three questions per request (4-way Choice, 3-level Score, Noul), shipped calibration:
+What averaging changes. The benches' three questions per request (4-way Choice, 3-level Score, Noul), shipped calibration; Score ECE over 15 equal-width bins. First-slot rate over all six orders of three real options, ties split (1/3 is order-free), and `consistent`, the share of states whose six orders give one answer:
```text
- Score acc Score RPS Choice acc Noul acc Noul AUROC ms / request
- en off 0.583 0.150 0.917 0.938 0.9778 164.6
- reversed 0.652 0.128 0.917 0.938 0.9780 184.0 (1.12x)
- cyclic 0.641 0.125 0.917 0.938 0.9784 215.4 (1.31x)
- ja off (ref.) 0.643 0.138 0.887 0.817 0.8890 150.9
- reversed 0.730 0.108 0.887 0.813 0.8877 171.4 (1.14x)
- cyclic 0.717 0.108 0.887 0.813 0.8876 199.9 (1.32x)
+ Score Choice Noul
+ acc RPS ECE first slot consistent acc first slot consistent acc AUROC ms / request
+ en off 0.583 0.150 0.124 0.265 0.714 0.917 0.331 0.986 0.938 0.9778 156.5
+ reversed 0.652 0.127 0.101 0.319 0.872 0.917 0.332 0.979 0.938 0.9780 179.3 (1.15x)
+ cyclic 0.641 0.125 0.060 0.310 0.893 0.917 0.332 0.983 0.938 0.9784 210.2 (1.34x)
+ ja off 0.643 0.138 0.093 0.242 0.627 0.887 0.331 0.973 0.817 0.8890 149.2
+ reversed 0.730 0.108 0.059 0.314 0.870 0.887 0.331 0.977 0.813 0.8877 169.9 (1.14x)
+ cyclic 0.717 0.108 0.077 0.299 0.853 0.887 0.330 0.973 0.813 0.8876 197.3 (1.32x)
```
-Choice does not move. Noul moves by one item on bench_ja, and asked alone it is identical under all three modes on all 300 items: the change comes from the batch the extra Score rows make (bf16), not from the Noul readout.
+Accuracy, RPS, Choice and Noul are identical, to the last digit, to the runs before the rebase and the review fixes. In the chat style the release uses, a Score line names its level by its place in the request (`(level i of K)`), so the probe's permuted listings renumber the levels as well as move them, and Score's first-slot rate measures both; the engine's own averaging moves each level with its number.
-Two caveats. Asking the probe's questions together or one at a time differs by up to |Δp| 0.18 (raw, bench_ja, off) on the three states checked per run, so a metric is tied to how many questions share a call; `lev presentation-checks` fixes that at one call of at most 24 Score rows. And `cyclic` needs that cap: 42 Score rows per call exhausted a 32 GB card. Even capped, PyTorch's reserved memory went past the card (44-49 GiB for `cyclic`, 62 GiB for the default probe of both kinds), which Windows backs with shared system memory.
+Calibration of the averaged modes. The shipped Score temperature (2.80) was fitted on single-order rows. Refit per mode on half of each bench (stratified by the gold level, seed 0), judged on the other half:
-**Reading, for the decision.** `reversed` removes most of the slot-0 deficit and lifts Score accuracy by 6.9 points on bench_en (8.7 on the Japanese reference bench) at 1.12-1.14x the time. `cyclic` is no better than `reversed` on either bench and costs 1.31-1.32x. Neither was tested on helpsteer2, the case that kept Score out of averaging in ADR-020, and both benches are one family of synthetic support messages. The proposal keeps the default off and leaves the choice between (a) and (b) to the maintainer.
+```text
+ T refit held-out NLL held-out ECE held-out RPS
+ shipped refit shipped refit shipped refit
+ en (144) off 3.96 0.935 0.909 0.130 0.108 0.152 0.147
+ reversed 3.16 0.839 0.838 0.111 0.119 0.126 0.126
+ cyclic 2.92 0.828 0.828 0.077 0.077 0.124 0.124
+ ja (149) off 3.60 0.781 0.779 0.090 0.077 0.132 0.130
+ reversed 2.39 0.641 0.627 0.089 0.061 0.102 0.101
+ cyclic 2.40 0.644 0.629 0.068 0.086 0.101 0.101
+```
+
+On bench_en a refit gains nothing for either averaged mode (held-out NLL within 0.001), and the shipped temperature is if anything a little sharp for `off`. On the Japanese reference the refit goes the other way for the averaged modes. A per-mode temperature is not proposed.
+
+The rows one request may need (`max_request_rows`, default 32). One `system_one` request of R single-order Score rows; peak CUDA memory on the RTX 5090, GiB (the bf16 weights alone are ~8.1). The benches' rows are at most 509 tokens (en) and 526 (ja):
+
+```text
+ row tokens rows reserved seconds
+ 474 32 14.2 1.6
+ 474 64 20.3 3.3
+ 474 96 26.4 5.3
+ 474 128 32.5 60.0 past the card: Windows spills into system memory
+ 824 48 27.5 4.3
+ 1124 32 27.1 3.9
+ 2124 16 30.1 12.3 spilling
+```
+
+Memory grows with rows times row length, and faster for long rows, so the limit is set for bench-length requests: 32 rows of ~510 tokens is about the batcher's 16384-token budget (ADR-030). A full presentation run with 64-row calls filled the card on bench_en (32 GB in use, 100% busy, no result after 22 minutes) and was stopped; with 32-row calls the six runs peaked at 22.5 GiB reserved, about 50 minutes each.
+
+With averaging off, every answer on 20 items of each bench (40 responses each, the probe included) is identical to main at 8ca94b7. Asking the probe's questions together or alone differs by up to |Δp| 0.038 (bench_en) and 0.041 (bench_ja), so the committed report records `max_rows`.
+
+**Reading, for the decision.** `reversed` lifts Score accuracy by 6.9 points on bench_en and lowers its RPS and ECE at 1.15x the time, with the shipped temperature. `cyclic` calibrates best on bench_en (ECE 0.060) but is no more accurate than `reversed` and costs 1.34x. Neither was tested on helpsteer2, the case that kept Score out of averaging in ADR-020, and both benches are one family of synthetic support messages. The proposal keeps the default off and leaves the choice between (a) and (b) to the maintainer.
---
From ff8fbd7057c87a2fa5766ea6a296b6286b80210f Mon Sep 17 00:00:00 2001
From: hiroki-abe-58
Date: Mon, 5 Oct 2026 12:43:58 +0900
Subject: [PATCH 07/39] Row limit: refuse only the rows Score averaging adds,
never a request main accepts
#4's batcher does not split a request: one larger than max_batch_tokens runs alone, in one
forward (20 Choice + 40 Score questions with averaging off: one forward of 80 rows, up to
35,680 padded tokens, on main). So max_request_rows now refuses a request only when Score
averaging added rows to it and the total is past the limit. With averaging off nothing is
refused, and that 80-row request answers exactly as on main (3 bench_en states, real model;
a tiny-model test pins it against an unlimited engine).
Co-Authored-By: Claude Opus 5.5
Claude-Session: https://claude.ai/code/session_01VyLpqwg29AwAk4YfyeTqxP
---
data/adr-029-summary.json | 59 +++++++++++++++++++++++
docs/ARCHITECTURE.md | 2 +-
docs/DECISIONS.md | 6 +--
packages/lev/src/lev/cli.py | 2 +-
packages/lev/src/lev/model.py | 18 ++++---
packages/lev/tests/test_engine_forward.py | 29 +++++++++++
packages/lev/tests/test_presentation.py | 21 +++++++-
7 files changed, 125 insertions(+), 12 deletions(-)
diff --git a/data/adr-029-summary.json b/data/adr-029-summary.json
index 4638f94..bcc3e7f 100644
--- a/data/adr-029-summary.json
+++ b/data/adr-029-summary.json
@@ -528,6 +528,65 @@
"identical": true
}
},
+ "large_request_without_averaging": {
+ "what": "20 Choice + 40 Score questions per request, default settings, through lev.batcher (max_batch_tokens 16384), first 3 bench_en states: every forward's rows and width",
+ "requests": [
+ {
+ "item_id": "bench_en-0000",
+ "rows": 80,
+ "padded_tokens": 26800,
+ "forwards_main": [
+ {
+ "rows": 80,
+ "width": 335
+ }
+ ],
+ "forwards_branch": [
+ {
+ "rows": 80,
+ "width": 335
+ }
+ ],
+ "answers_identical_to_main": true
+ },
+ {
+ "item_id": "bench_en-0001",
+ "rows": 80,
+ "padded_tokens": 15760,
+ "forwards_main": [
+ {
+ "rows": 80,
+ "width": 197
+ }
+ ],
+ "forwards_branch": [
+ {
+ "rows": 80,
+ "width": 197
+ }
+ ],
+ "answers_identical_to_main": true
+ },
+ {
+ "item_id": "bench_en-0002",
+ "rows": 80,
+ "padded_tokens": 35680,
+ "forwards_main": [
+ {
+ "rows": 80,
+ "width": 446
+ }
+ ],
+ "forwards_branch": [
+ {
+ "rows": 80,
+ "width": 446
+ }
+ ],
+ "answers_identical_to_main": true
+ }
+ ]
+ },
"request_rows_rtx5090": {
"what": "one system_one request of R single-order Score rows; peak CUDA memory in GiB (bf16 weights alone are ~8.1). Past ~31 GiB reserved, Windows spills into shared system memory instead of raising out-of-memory, and the request slows 5-10x.",
"bench_row_tokens": {
diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md
index 1d2d356..a355809 100644
--- a/docs/ARCHITECTURE.md
+++ b/docs/ARCHITECTURE.md
@@ -201,7 +201,7 @@ LoRA r32 on `q,k,v,o,gate,up,down` ≈ 25–40 M trainable parameters. **The Fam
- they get **separate fitted temperatures** — one global scalar would be wrong for at least one of them;
- on option sets where **both** are valid we measure agreement. Material disagreement means the router is a correctness hazard, not merely a capacity switch. This is an explicit eval, not an assumption.
-**Order averaging** (added in the build, not in the original design). Under Mode A with the state-first layout, Choice and binary Noul are read in their shown order and reversed, and the two distributions are averaged in canonical order (ADR-020). Score keeps its level order by default; `EngineConfig.score_order_average` opts it into the reversed order or every rotation, at 2x or Kx rows, and `order_average=False` reads every question once. A request may need at most `max_request_rows` batch rows (32 by default); `prepare` refuses a larger one before any GPU work. `lev presentation-checks` measures the position bias of one read, the bias averaging has to remove (ADR-029).
+**Order averaging** (added in the build, not in the original design). Under Mode A with the state-first layout, Choice and binary Noul are read in their shown order and reversed, and the two distributions are averaged in canonical order (ADR-020). Score keeps its level order by default; `EngineConfig.score_order_average` opts it into the reversed order or every rotation, at 2x or Kx rows, and `order_average=False` reads every question once. A request that Score averaging grows past `max_request_rows` batch rows (32 by default) is refused in `prepare`, before any GPU work; one without averaged Score rows never is, since the batcher runs an oversized request alone (ADR-030). `lev presentation-checks` measures the position bias of one read, the bias averaging has to remove (ADR-029).
### 5.4 Output types
diff --git a/docs/DECISIONS.md b/docs/DECISIONS.md
index f557d4e..7994eda 100644
--- a/docs/DECISIONS.md
+++ b/docs/DECISIONS.md
@@ -601,7 +601,7 @@ Serving now skips by default; training does not, so Mode B keeps its data. Mode
Every result is reported whichever way it goes, including accuracy that does not improve.
-**Revised after review (2026-10-02).** The first runs read the identical-option control after averaging, where `cyclic` makes it zero by construction and `reversed` cannot see the middle slots, so it was no evidence for averaging. The control now measures the bias of one read, before any averaging (`order_average=False`); what averaging changes is judged by labelled accuracy, RPS and calibration error, and by the first-slot rate on real options. Also changed: Choice keys are shapes with no order (■ ● ◆ ▲ ◎), read under every key-to-slot assignment so a key's own pull cancels; argmax ties split between the tied slots; calls are cut by the engine's own row count; and the engine bounds the rows one request may need.
+**Revised after review (2026-10-02).** The first runs read the identical-option control after averaging, where `cyclic` makes it zero by construction and `reversed` cannot see the middle slots, so it was no evidence for averaging. The control now measures the bias of one read, before any averaging (`order_average=False`); what averaging changes is judged by labelled accuracy, RPS and calibration error, and by the first-slot rate on real options. Also changed: Choice keys are shapes with no order (■ ● ◆ ▲ ◎), read under every key-to-slot assignment so a key's own pull cancels; argmax ties split between the tied slots; calls are cut by the engine's own row count; and the engine bounds the rows Score averaging may add to a request.
**Evidence.** Run 2026-10-04/05 on `interfaze-ai/lev` (HF 7bdc748: the weights and calibration of f8ef711, only README.md changed), bf16, one RTX 5090, this branch rebased on main after #4. The numbers behind every table are in [`data/adr-029-summary.json`](../data/adr-029-summary.json); the committed report CI compares against is [`data/presentation-checks.json`](../data/presentation-checks.json). `bench_ja` is Japanese, outside Lev's target language, so its rows are a reference only; `bench_en` is the one that bears on the decision.
@@ -646,7 +646,7 @@ Calibration of the averaged modes. The shipped Score temperature (2.80) was fitt
On bench_en a refit gains nothing for either averaged mode (held-out NLL within 0.001), and the shipped temperature is if anything a little sharp for `off`. On the Japanese reference the refit goes the other way for the averaged modes. A per-mode temperature is not proposed.
-The rows one request may need (`max_request_rows`, default 32). One `system_one` request of R single-order Score rows; peak CUDA memory on the RTX 5090, GiB (the bf16 weights alone are ~8.1). The benches' rows are at most 509 tokens (en) and 526 (ja):
+The rows Score averaging may add (`max_request_rows`, default 32). One `system_one` request of R single-order Score rows; peak CUDA memory on the RTX 5090, GiB (the bf16 weights alone are ~8.1). The benches' rows are at most 509 tokens (en) and 526 (ja):
```text
row tokens rows reserved seconds
@@ -659,7 +659,7 @@ The rows one request may need (`max_request_rows`, default 32). One `system_one`
2124 16 30.1 12.3 spilling
```
-Memory grows with rows times row length, and faster for long rows, so the limit is set for bench-length requests: 32 rows of ~510 tokens is about the batcher's 16384-token budget (ADR-030). A full presentation run with 64-row calls filled the card on bench_en (32 GB in use, 100% busy, no result after 22 minutes) and was stopped; with 32-row calls the six runs peaked at 22.5 GiB reserved, about 50 minutes each.
+Memory grows with rows times row length, and faster for long rows, so the limit is set for bench-length requests: 32 rows of ~510 tokens is about the batcher's 16384-token budget (ADR-030). The batcher does not split a request: on main, 20 Choice and 40 Score questions with averaging off ran as one forward of 80 rows, up to 35,680 padded tokens. So the limit refuses only a request that Score averaging grows past it, and never one without averaged Score rows; that 80-row request gets answers identical to main's. A full presentation run with 64-row calls filled the card on bench_en (32 GB in use, 100% busy, no result after 22 minutes) and was stopped; with 32-row calls the six runs peaked at 22.5 GiB reserved, about 50 minutes each.
With averaging off, every answer on 20 items of each bench (40 responses each, the probe included) is identical to main at 8ca94b7. Asking the probe's questions together or alone differs by up to |Δp| 0.038 (bench_en) and 0.041 (bench_ja), so the committed report records `max_rows`.
diff --git a/packages/lev/src/lev/cli.py b/packages/lev/src/lev/cli.py
index 9a3ff09..5e5cb2b 100644
--- a/packages/lev/src/lev/cli.py
+++ b/packages/lev/src/lev/cli.py
@@ -427,7 +427,7 @@ def main(argv: list[str] | None = None) -> None:
"--max-request-rows",
type=int,
default=None,
- help="batch rows one request may need before it gets 422 (default: the engine's)",
+ help="refuse (422) a request Score averaging grows past this many batch rows (default 32)",
)
serve_parser.add_argument(
"--score-order-average",
diff --git a/packages/lev/src/lev/model.py b/packages/lev/src/lev/model.py
index b936a9f..3072f74 100644
--- a/packages/lev/src/lev/model.py
+++ b/packages/lev/src/lev/model.py
@@ -53,8 +53,8 @@ class EngineConfig:
# adds the reversed order (2 rows per Score, as Choice), "cyclic" every rotation (K rows).
# Both show the model level orders training never produces (ADR-020, FINDINGS §12).
score_order_average: Literal["off", "reversed", "cyclic"] = "off"
- # Rows one request may put into a forward (every question's every order); `prepare`
- # refuses more with a ValueError (HTTP 422). None: no limit.
+ # Rows a request may need once Score averaging adds rows to it; `prepare` refuses
+ # more with a ValueError (HTTP 422). Requests without added rows are never refused.
max_request_rows: int | None = MAX_REQUEST_ROWS
# Serving can skip split codes; training keeps large sets for Mode B (ADR-028).
skip_multi_token_codes: bool = True
@@ -229,12 +229,18 @@ def question_rows(self, questions: Mapping[str, Question | dict]) -> dict[str, i
def _route_and_tokenise(self, state, questions: dict[str, Question]) -> Prepared:
routes = serving_routes(questions, self.tokenizer, self.config)
+ orders = {name: self._orders(q, routes[name]) for name, q in questions.items()}
+ rows = sum(len(o) for o in orders.values())
+ added = sum(len(o) - 1 for name, o in orders.items() if isinstance(questions[name], Score))
limit = self.config.max_request_rows
- rows = sum(len(self._orders(q, routes[name])) for name, q in questions.items())
- if limit is not None and rows > limit:
+ # The batcher runs an oversized request alone and unsplit; refuse only the rows
+ # Score averaging adds, so a request main accepts is never refused for its size.
+ if limit is not None and added and rows > limit:
raise ValueError(
- f"this request needs {rows} batch rows, above max_request_rows={limit}; "
- "send fewer questions per request, or average fewer orders"
+ f"this request needs {rows} batch rows, {added} of them from "
+ f"score_order_average={self.config.score_order_average!r}, above "
+ f"max_request_rows={limit}; send fewer Score questions per request, "
+ "or average fewer orders"
)
unsupported = [
diff --git a/packages/lev/tests/test_engine_forward.py b/packages/lev/tests/test_engine_forward.py
index 17eb149..bc679da 100644
--- a/packages/lev/tests/test_engine_forward.py
+++ b/packages/lev/tests/test_engine_forward.py
@@ -47,3 +47,32 @@ def test_single_forward_matches_the_full_logits_at_each_last_token(tiny_model, w
torch.testing.assert_close(hidden[i], full.hidden_states[-1][0, -1])
assert logits.shape == (len(rows), 512), "one vocabulary row per variant, not per position"
assert (hidden is not None) == want_hidden
+
+
+class TinyTokenizer:
+ """Ids inside the tiny vocabulary; space-prefixed A-Z are single tokens, as label codes need."""
+
+ pad_token_id = 0
+ single = {f" {c}": 400 + i for i, c in enumerate("ABCDEFGHIJKLMNOPQRSTUVWXYZ")}
+
+ def encode(self, text, add_special_tokens=True):
+ return (
+ [self.single[text]] if text in self.single else [ord(c) % 300 + 2 for c in text] or [1]
+ )
+
+
+def test_without_score_averaging_a_large_request_answers_as_with_no_limit(tiny_model):
+ """The row limit never touches a request Score averaging did not grow (ADR-029)."""
+ from lev.types import Choice, Score
+
+ questions = {
+ **{
+ f"c{i}": Choice(criteria={"a": None, "b": None, "c": None, "d": None})
+ for i in range(20)
+ },
+ **{f"s{i}": Score(criteria=["low", "mid", "high"]) for i in range(40)},
+ }
+ limited = DecisionEngine(tiny_model, TinyTokenizer(), EngineConfig())
+ unlimited = DecisionEngine(tiny_model, TinyTokenizer(), EngineConfig(max_request_rows=None))
+ assert limited.prepare("state", questions).rows == 80 > limited.config.max_request_rows
+ assert limited.system_one("state", questions) == unlimited.system_one("state", questions)
diff --git a/packages/lev/tests/test_presentation.py b/packages/lev/tests/test_presentation.py
index 410a65e..91ff55c 100644
--- a/packages/lev/tests/test_presentation.py
+++ b/packages/lev/tests/test_presentation.py
@@ -397,6 +397,12 @@ def test_schema_first_is_one_row_per_question(self, rich_tokenizer):
assert engine.prepare("s", QUESTIONS).rows == len(QUESTIONS)
+BIG = {
+ **{f"c{i}": Choice(criteria={"a": None, "b": None, "c": None, "d": None}) for i in range(20)},
+ **{f"s{i}": Score(criteria=["low", "mid", "high"]) for i in range(40)},
+}
+
+
class TestRequestRowLimit:
def engine(self, tokenizer, **config) -> DecisionEngine:
return DecisionEngine(model=None, tokenizer=tokenizer, config=EngineConfig(**config))
@@ -404,10 +410,23 @@ def engine(self, tokenizer, **config) -> DecisionEngine:
def test_a_request_past_the_limit_is_refused_before_any_forward(self, rich_tokenizer):
questions = {f"q{i}": Score(criteria=["a", "b", "c"]) for i in range(4)}
engine = self.engine(rich_tokenizer, score_order_average="cyclic", max_request_rows=11)
- with pytest.raises(ValueError, match="needs 12 batch rows, above max_request_rows=11"):
+ with pytest.raises(ValueError, match="needs 12 batch rows, 8 of them from"):
engine.prepare("s", questions)
assert self.engine(rich_tokenizer, max_request_rows=11).prepare("s", questions).rows == 4
+ def test_without_score_averaging_a_large_request_is_never_refused(self, rich_tokenizer):
+ """#4's batcher runs an oversized request alone and unsplit, so main accepts it."""
+ prepared = self.engine(rich_tokenizer).prepare("s", BIG)
+ assert prepared.rows == 80 > MAX_REQUEST_ROWS
+ assert [v.order for v in prepared.variants] == [None, [3, 2, 1, 0]] * 20 + [None] * 40
+
+ def test_only_rows_score_averaging_adds_can_refuse(self, rich_tokenizer):
+ engine = self.engine(rich_tokenizer, score_order_average="cyclic")
+ with pytest.raises(ValueError, match="needs 160 batch rows, 80 of them"):
+ engine.prepare("s", BIG)
+ choices = {k: q for k, q in BIG.items() if k.startswith("c")}
+ assert engine.prepare("s", choices).rows == 40
+
def test_none_is_no_limit(self, rich_tokenizer):
questions = {f"q{i}": Score(criteria=["a", "b", "c"]) for i in range(30)}
engine = self.engine(rich_tokenizer, score_order_average="cyclic", max_request_rows=None)
From 3136a79ce5473b743df4a93ce7c7592de6390be0 Mon Sep 17 00:00:00 2001
From: hiroki-abe-58
Date: Mon, 5 Oct 2026 12:44:25 +0900
Subject: [PATCH 08/39] ADR-029: Score's first-slot rate is a reference, not
evidence for averaging
In the chat style a Score line names its level by its place in the request, so the probe's
permuted listings renumber the levels as well as move them. The evidence for averaging is
labelled accuracy, RPS and ECE.
Co-Authored-By: Claude Opus 5.5
Claude-Session: https://claude.ai/code/session_01VyLpqwg29AwAk4YfyeTqxP
---
docs/DECISIONS.md | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/docs/DECISIONS.md b/docs/DECISIONS.md
index 7994eda..0b50de8 100644
--- a/docs/DECISIONS.md
+++ b/docs/DECISIONS.md
@@ -601,7 +601,7 @@ Serving now skips by default; training does not, so Mode B keeps its data. Mode
Every result is reported whichever way it goes, including accuracy that does not improve.
-**Revised after review (2026-10-02).** The first runs read the identical-option control after averaging, where `cyclic` makes it zero by construction and `reversed` cannot see the middle slots, so it was no evidence for averaging. The control now measures the bias of one read, before any averaging (`order_average=False`); what averaging changes is judged by labelled accuracy, RPS and calibration error, and by the first-slot rate on real options. Also changed: Choice keys are shapes with no order (■ ● ◆ ▲ ◎), read under every key-to-slot assignment so a key's own pull cancels; argmax ties split between the tied slots; calls are cut by the engine's own row count; and the engine bounds the rows Score averaging may add to a request.
+**Revised after review (2026-10-02).** The first runs read the identical-option control after averaging, where `cyclic` makes it zero by construction and `reversed` cannot see the middle slots, so it was no evidence for averaging. The control now measures the bias of one read, before any averaging (`order_average=False`); what averaging changes is judged by labelled accuracy, RPS and calibration error. Score's first-slot rate on real options is kept as a reference only: in the chat style the release uses, the probe's permuted listings renumber the levels as well as move them. Also changed: Choice keys are shapes with no order (■ ● ◆ ▲ ◎), read under every key-to-slot assignment so a key's own pull cancels; argmax ties split between the tied slots; calls are cut by the engine's own row count; and the engine bounds the rows Score averaging may add to a request.
**Evidence.** Run 2026-10-04/05 on `interfaze-ai/lev` (HF 7bdc748: the weights and calibration of f8ef711, only README.md changed), bf16, one RTX 5090, this branch rebased on main after #4. The numbers behind every table are in [`data/adr-029-summary.json`](../data/adr-029-summary.json); the committed report CI compares against is [`data/presentation-checks.json`](../data/presentation-checks.json). `bench_ja` is Japanese, outside Lev's target language, so its rows are a reference only; `bench_en` is the one that bears on the decision.
@@ -616,7 +616,7 @@ The bias of one read: identical-option control, slot 0 minus the mean (gate -0.2
Choice, read once, avoids slot 0 more than Score does; it is already read in two orders when served, and the first-slot rate below is what that leaves.
-What averaging changes. The benches' three questions per request (4-way Choice, 3-level Score, Noul), shipped calibration; Score ECE over 15 equal-width bins. First-slot rate over all six orders of three real options, ties split (1/3 is order-free), and `consistent`, the share of states whose six orders give one answer:
+What averaging changes. The benches' three questions per request (4-way Choice, 3-level Score, Noul), shipped calibration; Score ECE over 15 equal-width bins. For reference, the first-slot rate over all six orders of three real options, ties split (1/3 is order-free), and `consistent`, the share of states whose six orders give one answer:
```text
Score Choice Noul
From 79d9451f77dd8516be767a97b6b6b26d9e1d9fd3 Mon Sep 17 00:00:00 2001
From: hiroki-abe-58
Date: Mon, 5 Oct 2026 14:13:29 +0900
Subject: [PATCH 09/39] Remove the _eager_model reference that #4's description
pointed out
_weights_dtype read engine._eager_model, which #4 removed; it now reads the dtype from
engine.model only, the fallback that already gave the right value. A test pins bf16.
Co-Authored-By: Claude Opus 5.5
Claude-Session: https://claude.ai/code/session_01VyLpqwg29AwAk4YfyeTqxP
---
packages/lev/src/lev/cli.py | 2 +-
packages/lev/tests/test_presentation.py | 10 ++++++++++
2 files changed, 11 insertions(+), 1 deletion(-)
diff --git a/packages/lev/src/lev/cli.py b/packages/lev/src/lev/cli.py
index 5e5cb2b..4a9fe8c 100644
--- a/packages/lev/src/lev/cli.py
+++ b/packages/lev/src/lev/cli.py
@@ -244,7 +244,7 @@ def _snapshot_revision(checkpoint) -> str | None:
def _weights_dtype(engine) -> str | None:
- model = getattr(engine, "_eager_model", None) or getattr(engine, "model", None)
+ model = getattr(engine, "model", None)
try:
return str(next(model.parameters()).dtype)
except (AttributeError, StopIteration, TypeError):
diff --git a/packages/lev/tests/test_presentation.py b/packages/lev/tests/test_presentation.py
index 91ff55c..7ae072b 100644
--- a/packages/lev/tests/test_presentation.py
+++ b/packages/lev/tests/test_presentation.py
@@ -483,6 +483,16 @@ def committed(self, monkeypatch, tmp_path, engine, **edit) -> str:
path.write_text(json.dumps(saved), "utf-8")
return str(path)
+ def test_the_weights_dtype_comes_from_the_engine_model(self):
+ torch = pytest.importorskip("torch")
+ from types import SimpleNamespace
+
+ from lev.cli import _weights_dtype
+
+ model = torch.nn.Linear(2, 2).to(torch.bfloat16)
+ assert _weights_dtype(SimpleNamespace(model=model)) == "torch.bfloat16"
+ assert _weights_dtype(SimpleNamespace(model=None)) is None
+
def test_the_settings_split_at_the_load(self):
assert set(PRE_LOAD).isdisjoint(POST_LOAD)
assert set(POST_LOAD) == {"checkpoint_revision", "dtype"}
From b61852980cd9f1d86966453f9fed616f304de5a1 Mon Sep 17 00:00:00 2001
From: hiroki-abe-58
Date: Mon, 5 Oct 2026 14:50:50 +0900
Subject: [PATCH 10/39] Bound Score-averaged requests by the batcher's token
budget, not by rows
CodeRabbit (review 5410274149) pointed out that a row count does not bound a request's memory:
long states need fewer rows to fill the card (16 single-order rows of 2,124 tokens already
spilled). `prepare` now refuses a request only when Score averaging added rows to it and its
padded tokens (rows x widest row) pass `max_batch_tokens`, the same 16384-token budget #4's
batcher uses; the server passes its own value to the engine, so the two always agree, and
/health reports it. Requests without averaged Score rows are never refused (an 80-row
request still answers exactly as on main).
Removed: `max_request_rows`, `MAX_REQUEST_ROWS`, `lev serve --max-request-rows`, and its /health
field. Added: `EngineConfig.max_batch_tokens`, `MAX_BATCH_TOKENS`, `lev.load(max_batch_tokens=)`.
The probe sizes its own calls (`--max-rows`, 32) and lifts the serving limit for them.
On an RTX 5090 the largest admitted averaged request peaked at 13.1-19.8 GiB reserved at
474-4,124-token rows and 25.8 GiB at 5,423; at 8,124-token rows it reached the card's edge,
as one unaveraged Choice question does on the same state (ADR-029).
Co-Authored-By: Claude Opus 5.5
Claude-Session: https://claude.ai/code/session_01VyLpqwg29AwAk4YfyeTqxP
---
data/adr-029-summary.json | 121 +++++++++++++++++++++-
docs/ARCHITECTURE.md | 2 +-
docs/DECISIONS.md | 25 +++--
packages/lev/src/lev/batcher.py | 4 +-
packages/lev/src/lev/cli.py | 21 +---
packages/lev/src/lev/model.py | 46 ++++----
packages/lev/src/lev/presentation.py | 28 +++--
packages/lev/src/lev/server.py | 9 +-
packages/lev/tests/test_engine_forward.py | 10 +-
packages/lev/tests/test_presentation.py | 96 +++++++++++------
10 files changed, 262 insertions(+), 100 deletions(-)
diff --git a/data/adr-029-summary.json b/data/adr-029-summary.json
index bcc3e7f..3348833 100644
--- a/data/adr-029-summary.json
+++ b/data/adr-029-summary.json
@@ -587,7 +587,126 @@
}
]
},
- "request_rows_rtx5090": {
+ "averaged_token_budget_rtx5090": {
+ "what": "the largest Score-averaged request max_batch_tokens (16384) admits, by row length: averaged Score questions topped up with one-row Noul; one question more is refused in every case. The value is the batcher's budget; this only checks it.",
+ "results": [
+ {
+ "state_tokens": 350,
+ "mode": "reversed",
+ "score_questions": 17,
+ "noul_questions": 0,
+ "rows": 34,
+ "width": 474,
+ "padded_tokens": 16116,
+ "one_more_refused": true,
+ "seconds": 1.686,
+ "peak_allocated_gib": 12.1,
+ "peak_reserved_gib": 14.62
+ },
+ {
+ "state_tokens": 350,
+ "mode": "cyclic",
+ "score_questions": 11,
+ "noul_questions": 1,
+ "rows": 34,
+ "width": 474,
+ "padded_tokens": 16116,
+ "one_more_refused": true,
+ "seconds": 1.653,
+ "peak_allocated_gib": 12.18,
+ "peak_reserved_gib": 14.63
+ },
+ {
+ "state_tokens": 2000,
+ "mode": "reversed",
+ "score_questions": 3,
+ "noul_questions": 1,
+ "rows": 7,
+ "width": 2124,
+ "padded_tokens": 14868,
+ "one_more_refused": true,
+ "seconds": 1.539,
+ "peak_allocated_gib": 11.7,
+ "peak_reserved_gib": 14.03
+ },
+ {
+ "state_tokens": 2000,
+ "mode": "cyclic",
+ "score_questions": 2,
+ "noul_questions": 1,
+ "rows": 7,
+ "width": 2124,
+ "padded_tokens": 14868,
+ "one_more_refused": true,
+ "seconds": 1.527,
+ "peak_allocated_gib": 11.7,
+ "peak_reserved_gib": 14.03
+ },
+ {
+ "state_tokens": 4000,
+ "mode": "reversed",
+ "score_questions": 1,
+ "noul_questions": 1,
+ "rows": 3,
+ "width": 4124,
+ "padded_tokens": 12372,
+ "one_more_refused": true,
+ "seconds": 1.387,
+ "peak_allocated_gib": 11.09,
+ "peak_reserved_gib": 13.06
+ },
+ {
+ "state_tokens": 4000,
+ "mode": "cyclic",
+ "score_questions": 1,
+ "noul_questions": 0,
+ "rows": 3,
+ "width": 4124,
+ "padded_tokens": 12372,
+ "one_more_refused": true,
+ "seconds": 1.998,
+ "peak_allocated_gib": 16.29,
+ "peak_reserved_gib": 19.84
+ },
+ {
+ "state_tokens": 5300,
+ "mode": "cyclic",
+ "score_questions": 1,
+ "noul_questions": 0,
+ "rows": 3,
+ "width": 5423,
+ "padded_tokens": 16269,
+ "one_more_refused": true,
+ "seconds": 2.776,
+ "peak_allocated_gib": 21.72,
+ "peak_reserved_gib": 25.83
+ },
+ {
+ "state_tokens": 8000,
+ "mode": "reversed",
+ "score_questions": 1,
+ "noul_questions": 0,
+ "rows": 2,
+ "width": 8124,
+ "padded_tokens": 16248,
+ "one_more_refused": true,
+ "seconds": 18.927,
+ "peak_allocated_gib": 27.72,
+ "peak_reserved_gib": 31.68
+ }
+ ],
+ "averaging_off_one_choice_8000_tokens": {
+ "what": "averaging off (default), one Choice question read in two orders",
+ "state_tokens": 8000,
+ "rows": 2,
+ "width": 8109,
+ "padded_tokens": 16218,
+ "seconds": 30.932,
+ "peak_allocated_gib": 27.66,
+ "peak_reserved_gib": 31.63
+ }
+ },
+ "rows_without_a_limit_rtx5090": {
"what": "one system_one request of R single-order Score rows; peak CUDA memory in GiB (bf16 weights alone are ~8.1). Past ~31 GiB reserved, Windows spills into shared system memory instead of raising out-of-memory, and the request slows 5-10x.",
"bench_row_tokens": {
"bench_en": {
diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md
index a355809..770e9bf 100644
--- a/docs/ARCHITECTURE.md
+++ b/docs/ARCHITECTURE.md
@@ -201,7 +201,7 @@ LoRA r32 on `q,k,v,o,gate,up,down` ≈ 25–40 M trainable parameters. **The Fam
- they get **separate fitted temperatures** — one global scalar would be wrong for at least one of them;
- on option sets where **both** are valid we measure agreement. Material disagreement means the router is a correctness hazard, not merely a capacity switch. This is an explicit eval, not an assumption.
-**Order averaging** (added in the build, not in the original design). Under Mode A with the state-first layout, Choice and binary Noul are read in their shown order and reversed, and the two distributions are averaged in canonical order (ADR-020). Score keeps its level order by default; `EngineConfig.score_order_average` opts it into the reversed order or every rotation, at 2x or Kx rows, and `order_average=False` reads every question once. A request that Score averaging grows past `max_request_rows` batch rows (32 by default) is refused in `prepare`, before any GPU work; one without averaged Score rows never is, since the batcher runs an oversized request alone (ADR-030). `lev presentation-checks` measures the position bias of one read, the bias averaging has to remove (ADR-029).
+**Order averaging** (added in the build, not in the original design). Under Mode A with the state-first layout, Choice and binary Noul are read in their shown order and reversed, and the two distributions are averaged in canonical order (ADR-020). Score keeps its level order by default; `EngineConfig.score_order_average` opts it into the reversed order or every rotation, at 2x or Kx rows, and `order_average=False` reads every question once. A request that Score averaging grew is refused in `prepare`, before any GPU work, when its padded tokens pass `max_batch_tokens` (16384, the batcher's budget; the server passes its own value to the engine); one without averaged Score rows never is, since the batcher runs an oversized request alone (ADR-030). `lev presentation-checks` measures the position bias of one read, the bias averaging has to remove (ADR-029).
### 5.4 Output types
diff --git a/docs/DECISIONS.md b/docs/DECISIONS.md
index 0b50de8..9b6e762 100644
--- a/docs/DECISIONS.md
+++ b/docs/DECISIONS.md
@@ -601,7 +601,7 @@ Serving now skips by default; training does not, so Mode B keeps its data. Mode
Every result is reported whichever way it goes, including accuracy that does not improve.
-**Revised after review (2026-10-02).** The first runs read the identical-option control after averaging, where `cyclic` makes it zero by construction and `reversed` cannot see the middle slots, so it was no evidence for averaging. The control now measures the bias of one read, before any averaging (`order_average=False`); what averaging changes is judged by labelled accuracy, RPS and calibration error. Score's first-slot rate on real options is kept as a reference only: in the chat style the release uses, the probe's permuted listings renumber the levels as well as move them. Also changed: Choice keys are shapes with no order (■ ● ◆ ▲ ◎), read under every key-to-slot assignment so a key's own pull cancels; argmax ties split between the tied slots; calls are cut by the engine's own row count; and the engine bounds the rows Score averaging may add to a request.
+**Revised after review (2026-10-02).** The first runs read the identical-option control after averaging, where `cyclic` makes it zero by construction and `reversed` cannot see the middle slots, so it was no evidence for averaging. The control now measures the bias of one read, before any averaging (`order_average=False`); what averaging changes is judged by labelled accuracy, RPS and calibration error. Score's first-slot rate on real options is kept as a reference only: in the chat style the release uses, the probe's permuted listings renumber the levels as well as move them. Also changed: Choice keys are shapes with no order (■ ● ◆ ▲ ◎), read under every key-to-slot assignment so a key's own pull cancels; argmax ties split between the tied slots; calls are cut by the engine's own row count; and the engine refuses a request Score averaging grew past the batcher's token budget.
**Evidence.** Run 2026-10-04/05 on `interfaze-ai/lev` (HF 7bdc748: the weights and calibration of f8ef711, only README.md changed), bf16, one RTX 5090, this branch rebased on main after #4. The numbers behind every table are in [`data/adr-029-summary.json`](../data/adr-029-summary.json); the committed report CI compares against is [`data/presentation-checks.json`](../data/presentation-checks.json). `bench_ja` is Japanese, outside Lev's target language, so its rows are a reference only; `bench_en` is the one that bears on the decision.
@@ -646,20 +646,23 @@ Calibration of the averaged modes. The shipped Score temperature (2.80) was fitt
On bench_en a refit gains nothing for either averaged mode (held-out NLL within 0.001), and the shipped temperature is if anything a little sharp for `off`. On the Japanese reference the refit goes the other way for the averaged modes. A per-mode temperature is not proposed.
-The rows Score averaging may add (`max_request_rows`, default 32). One `system_one` request of R single-order Score rows; peak CUDA memory on the RTX 5090, GiB (the bf16 weights alone are ~8.1). The benches' rows are at most 509 tokens (en) and 526 (ja):
+The batch budget for averaged requests (`max_batch_tokens`, 16384). The batcher does not split a request: on main, 20 Choice and 40 Score questions with averaging off ran as one forward of 80 rows, up to 35,680 padded tokens. So `prepare` refuses only a request that Score averaging grew, when its padded tokens (rows x widest row) pass the batcher's own budget; the server passes its `max_batch_tokens` to the engine, so the two always agree, and a request without averaged Score rows is never refused (the 80-row request above answers exactly as on main). The value is the batcher's; this run only checks it on the RTX 5090, with the largest averaged request the budget admits at each row length, averaged Score questions topped up with one-row Noul (one question more is refused in every case):
```text
- row tokens rows reserved seconds
- 474 32 14.2 1.6
- 474 64 20.3 3.3
- 474 96 26.4 5.3
- 474 128 32.5 60.0 past the card: Windows spills into system memory
- 824 48 27.5 4.3
- 1124 32 27.1 3.9
- 2124 16 30.1 12.3 spilling
+ row tokens mode rows padded tokens reserved GiB seconds
+ 474 reversed 34 16,116 14.6 1.7
+ 474 cyclic 34 16,116 14.6 1.7
+ 2,124 reversed 7 14,868 14.0 1.5
+ 2,124 cyclic 7 14,868 14.0 1.5
+ 4,124 reversed 3 12,372 13.1 1.4
+ 4,124 cyclic 3 12,372 19.8 2.0
+ 5,423 cyclic 3 16,269 25.8 2.8
+ 8,124 reversed 2 16,248 31.7 18.9 at the card's edge, slow
```
-Memory grows with rows times row length, and faster for long rows, so the limit is set for bench-length requests: 32 rows of ~510 tokens is about the batcher's 16384-token budget (ADR-030). The batcher does not split a request: on main, 20 Choice and 40 Score questions with averaging off ran as one forward of 80 rows, up to 35,680 padded tokens. So the limit refuses only a request that Score averaging grows past it, and never one without averaged Score rows; that 80-row request gets answers identical to main's. A full presentation run with 64-row calls filled the card on bench_en (32 GB in use, 100% busy, no result after 22 minutes) and was stopped; with 32-row calls the six runs peaked at 22.5 GiB reserved, about 50 minutes each.
+Rows of ~5,400 tokens fit with room to spare. At ~8,100-token rows attention outgrows the token count: the admitted request reached the edge of the 32 GB card and took 18.9 s. Averaging is not the cause there: one Choice question on the same state with averaging off, read in two orders as always, needs the same (31.6 GiB reserved, 30.9 s). A row count could not have bounded this either: in a first measurement without any limit, 16 single-order rows of 2,124 tokens already spilled (30.1 GiB reserved, 12.3 s).
+
+The probe's own calls hold at most 32 rows (`--max-rows`, a measurement setting): a full presentation run with 64-row calls filled the card on bench_en (32 GB in use, 100% busy, no result after 22 minutes) and was stopped; with 32-row calls the six runs peaked at 22.5 GiB reserved, about 50 minutes each.
With averaging off, every answer on 20 items of each bench (40 responses each, the probe included) is identical to main at 8ca94b7. Asking the probe's questions together or alone differs by up to |Δp| 0.038 (bench_en) and 0.041 (bench_ja), so the committed report records `max_rows`.
diff --git a/packages/lev/src/lev/batcher.py b/packages/lev/src/lev/batcher.py
index 648d370..0c92080 100644
--- a/packages/lev/src/lev/batcher.py
+++ b/packages/lev/src/lev/batcher.py
@@ -14,7 +14,7 @@
from concurrent.futures import Future
from dataclasses import dataclass
-from .model import DecisionEngine, Prepared
+from .model import MAX_BATCH_TOKENS, DecisionEngine, Prepared
from .types import SystemOneResponse
@@ -42,7 +42,7 @@ def __init__(
self,
engine: DecisionEngine,
max_pending: int = 64,
- max_batch_tokens: int = 16384,
+ max_batch_tokens: int = MAX_BATCH_TOKENS,
timeout: float = 30.0,
):
self.engine = engine
diff --git a/packages/lev/src/lev/cli.py b/packages/lev/src/lev/cli.py
index 4a9fe8c..9e78631 100644
--- a/packages/lev/src/lev/cli.py
+++ b/packages/lev/src/lev/cli.py
@@ -217,11 +217,6 @@ def cmd_serve(args: argparse.Namespace) -> None:
max_pending=args.max_pending,
max_batch_tokens=args.max_batch_tokens,
score_order_average=args.score_order_average,
- **(
- {"max_request_rows": args.max_request_rows}
- if args.max_request_rows is not None
- else {}
- ),
),
host=args.host,
port=args.port,
@@ -259,11 +254,11 @@ def cmd_presentation_checks(args: argparse.Namespace) -> None:
from . import presentation
from .calibrate import CalibrationProfile
- from .model import MAX_REQUEST_ROWS, load
+ from .model import load
states = presentation.read_states(args.states)[: args.limit]
kinds = [k.strip() for k in args.kinds.split(",") if k.strip()]
- max_rows = args.max_rows or MAX_REQUEST_ROWS
+ max_rows = args.max_rows or presentation.MAX_ROWS
settings = {
"n_states": len(states),
"lang": args.lang,
@@ -280,7 +275,6 @@ def cmd_presentation_checks(args: argparse.Namespace) -> None:
cache_dir=args.model_cache,
calibration=args.calibration,
score_order_average=args.score_order_average,
- max_request_rows=max(max_rows, MAX_REQUEST_ROWS),
)
if args.raw:
engine.calibration = CalibrationProfile()
@@ -421,13 +415,8 @@ def main(argv: list[str] | None = None) -> None:
"--max-batch-tokens",
type=int,
default=16384,
- help="padded tokens per batched forward across requests (default 16384)",
- )
- serve_parser.add_argument(
- "--max-request-rows",
- type=int,
- default=None,
- help="refuse (422) a request Score averaging grows past this many batch rows (default 32)",
+ help="padded tokens per batched forward, and the most a request Score averaging grew "
+ "may need (default 16384)",
)
serve_parser.add_argument(
"--score-order-average",
@@ -464,7 +453,7 @@ def main(argv: list[str] | None = None) -> None:
"--max-rows",
type=int,
default=None,
- help="batch rows per call, as the engine counts them (default: max_request_rows)",
+ help="batch rows per probe call, as the engine counts them (default 32)",
)
presentation_parser.add_argument("--slot0-min", type=float, default=-0.20)
presentation_parser.add_argument("--out", default=None, help="write the report JSON here")
diff --git a/packages/lev/src/lev/model.py b/packages/lev/src/lev/model.py
index 3072f74..1255222 100644
--- a/packages/lev/src/lev/model.py
+++ b/packages/lev/src/lev/model.py
@@ -33,8 +33,9 @@
_QUESTIONS = TypeAdapter(dict[str, Question])
-# Bench-length rows (~500 tokens) x 32 is about the batcher's 16384-token budget (ADR-029).
-MAX_REQUEST_ROWS = 32
+# Padded tokens (rows x widest row) per forward: `lev.batcher`'s budget, and the most a
+# request Score averaging grew may need (ADR-029, ADR-030).
+MAX_BATCH_TOKENS = 16384
@dataclass
@@ -53,9 +54,9 @@ class EngineConfig:
# adds the reversed order (2 rows per Score, as Choice), "cyclic" every rotation (K rows).
# Both show the model level orders training never produces (ADR-020, FINDINGS §12).
score_order_average: Literal["off", "reversed", "cyclic"] = "off"
- # Rows a request may need once Score averaging adds rows to it; `prepare` refuses
- # more with a ValueError (HTTP 422). Requests without added rows are never refused.
- max_request_rows: int | None = MAX_REQUEST_ROWS
+ # A request whose Score averaging added rows must fit this many padded tokens; `prepare`
+ # refuses it otherwise (HTTP 422). Requests without added Score rows are never refused.
+ max_batch_tokens: int | None = MAX_BATCH_TOKENS
# Serving can skip split codes; training keeps large sets for Mode B (ADR-028).
skip_multi_token_codes: bool = True
# Forking avoids repeated prefix work but adds a forward and cache copies (ADR-023).
@@ -229,20 +230,6 @@ def question_rows(self, questions: Mapping[str, Question | dict]) -> dict[str, i
def _route_and_tokenise(self, state, questions: dict[str, Question]) -> Prepared:
routes = serving_routes(questions, self.tokenizer, self.config)
- orders = {name: self._orders(q, routes[name]) for name, q in questions.items()}
- rows = sum(len(o) for o in orders.values())
- added = sum(len(o) - 1 for name, o in orders.items() if isinstance(questions[name], Score))
- limit = self.config.max_request_rows
- # The batcher runs an oversized request alone and unsplit; refuse only the rows
- # Score averaging adds, so a request main accepts is never refused for its size.
- if limit is not None and added and rows > limit:
- raise ValueError(
- f"this request needs {rows} batch rows, {added} of them from "
- f"score_order_average={self.config.score_order_average!r}, above "
- f"max_request_rows={limit}; send fewer Score questions per request, "
- "or average fewer orders"
- )
-
unsupported = [
name
for name, r in routes.items()
@@ -257,7 +244,22 @@ def _route_and_tokenise(self, state, questions: dict[str, Question]) -> Prepared
prefix, variants = self._render(state, questions, routes)
prefix_ids = self.tokenizer.encode(prefix, add_special_tokens=True)
- return Prepared(questions, routes, prefix_ids, variants)
+ prepared = Prepared(questions, routes, prefix_ids, variants)
+ added = sum(
+ 1 for v in variants if v.order is not None and isinstance(questions[v.name], Score)
+ )
+ limit = self.config.max_batch_tokens
+ # The batcher runs an oversized request alone and unsplit; refuse only one that
+ # Score averaging grew, so a request main accepts is never refused for its size.
+ if limit is not None and added and prepared.rows * prepared.width > limit:
+ raise ValueError(
+ f"this request needs {prepared.rows} rows x {prepared.width} tokens = "
+ f"{prepared.rows * prepared.width} padded tokens, {added} of the rows from "
+ f"score_order_average={self.config.score_order_average!r}, above "
+ f"max_batch_tokens={limit}; send fewer Score questions or a shorter state, "
+ "or average fewer orders"
+ )
+ return prepared
def answer(self, requests: list[Prepared]) -> list[SystemOneResponse]:
"""Answer prepared requests, in single prefix mode all in one forward.
@@ -561,7 +563,7 @@ def load(
max_label_options: int | None = None,
skip_multi_token_codes: bool = True,
score_order_average: Literal["off", "reversed", "cyclic"] = "off",
- max_request_rows: int | None = MAX_REQUEST_ROWS,
+ max_batch_tokens: int | None = MAX_BATCH_TOKENS,
) -> DecisionEngine:
"""Load a checkpoint -- a release directory, a training output or a Hub id --
into a ready `DecisionEngine`. With no checkpoint, serves `model_id` frozen.
@@ -638,7 +640,7 @@ def load(
max_label_options=max_label_options,
skip_multi_token_codes=skip_multi_token_codes,
score_order_average=score_order_average,
- max_request_rows=max_request_rows,
+ max_batch_tokens=max_batch_tokens,
)
engine = DecisionEngine(model, tokenizer, config, profile, mode_b_head=head)
engine.checkpoint = resolved
diff --git a/packages/lev/src/lev/presentation.py b/packages/lev/src/lev/presentation.py
index 6e3d68b..835002b 100644
--- a/packages/lev/src/lev/presentation.py
+++ b/packages/lev/src/lev/presentation.py
@@ -41,6 +41,8 @@
KEYS = ("■", "●", "◆", "▲", "◎")
SLOT0_MIN = -0.20
FIRST_SLOT_MIN = 0.15
+# Batch rows per probe call: a full bench_en run peaked at 22.5 GiB reserved on a 32 GB card.
+MAX_ROWS = 32
# en follows Laya's presentation_checks.py; ja the same wording in Japanese.
TEXT: dict[str, dict[str, Any]] = {
@@ -125,16 +127,21 @@ def is_identical(qid: str) -> bool:
@contextmanager
-def single_order(engine) -> Iterator[None]:
- """Read every question once, in the order given."""
+def configured(engine, **changes) -> Iterator[None]:
+ """Run the block with `engine.config` changed as given, then restore it."""
config = engine.config
- engine.config = replace(config, order_average=False)
+ engine.config = replace(config, **changes)
try:
yield
finally:
engine.config = config
+def single_order(engine):
+ """Read every question once, in the order given."""
+ return configured(engine, order_average=False)
+
+
def call_groups(engine, questions: dict[str, dict], max_rows: int) -> list[dict[str, dict]]:
"""`questions` split into `system_one` calls of at most `max_rows` batch rows, as the
engine counts them under its current config."""
@@ -161,9 +168,8 @@ def ask(
max_rows: int | None = None,
) -> list[dict[str, Any]]:
"""Every probe answer for every state: the identical questions in one order, the
- permuted ones as the engine is configured. Calls hold at most `max_rows` batch rows,
- the engine's `max_request_rows` by default."""
- max_rows = max_rows or engine.config.max_request_rows
+ permuted ones as the engine is configured. Calls hold at most `max_rows` batch rows."""
+ max_rows = max_rows or MAX_ROWS
questions = probe_questions(lang, kinds)
identical = {q: v for q, v in questions.items() if is_identical(q)}
permuted = {q: v for q, v in questions.items() if not is_identical(q)}
@@ -176,8 +182,11 @@ def ask(
with single_order(engine):
for group in identical_groups:
got.update(engine.system_one(state, group).answers)
- for group in permuted_groups:
- got.update(engine.system_one(state, group).answers)
+ # The probe sizes its own calls; the serving limit would refuse its averaged
+ # calls on long states.
+ with configured(engine, max_batch_tokens=None):
+ for group in permuted_groups:
+ got.update(engine.system_one(state, group).answers)
answers.append(got)
return answers
@@ -283,7 +292,8 @@ def packed_consistency(
with single_order(engine):
alone = engine.system_one(state, {qid: question}).answers[qid]
else:
- alone = engine.system_one(state, {qid: question}).answers[qid]
+ with configured(engine, max_batch_tokens=None):
+ alone = engine.system_one(state, {qid: question}).answers[qid]
for a, b in zip(
slot_probs(packed[qid], question), slot_probs(alone, question), strict=True
):
diff --git a/packages/lev/src/lev/server.py b/packages/lev/src/lev/server.py
index 90c045b..ec9c24a 100644
--- a/packages/lev/src/lev/server.py
+++ b/packages/lev/src/lev/server.py
@@ -13,7 +13,7 @@
from fastapi import Request
from .batcher import Batcher, ClientGone, Overloaded, WorkerStopped
-from .model import MAX_REQUEST_ROWS, load
+from .model import MAX_BATCH_TOKENS, load
from .types import SystemOneRequest, SystemOneResponse
@@ -28,10 +28,9 @@ def create_app(
prompt_style: Literal["plain", "chat"] | None = None,
skip_multi_token_codes: bool = True,
max_pending: int = 64,
- max_batch_tokens: int = 16384,
+ max_batch_tokens: int = MAX_BATCH_TOKENS,
timeout: float = 30.0,
score_order_average: Literal["off", "reversed", "cyclic"] = "off",
- max_request_rows: int | None = MAX_REQUEST_ROWS,
) -> Any:
from fastapi import FastAPI, HTTPException, Response
from starlette.concurrency import run_in_threadpool
@@ -54,7 +53,7 @@ def _load() -> None:
max_label_options=max_label_options,
skip_multi_token_codes=skip_multi_token_codes,
score_order_average=score_order_average,
- max_request_rows=max_request_rows,
+ max_batch_tokens=max_batch_tokens,
)
state["batcher"] = Batcher(
engine, max_pending=max_pending, max_batch_tokens=max_batch_tokens, timeout=timeout
@@ -78,7 +77,7 @@ async def health() -> dict:
"max_label_options": config.max_label_options,
"order_average": config.order_average,
"score_order_average": config.score_order_average,
- "max_request_rows": config.max_request_rows,
+ "max_batch_tokens": config.max_batch_tokens,
"prefix_mode": config.prefix_mode,
"compiled": config.compile,
"prompt_style": config.prompt_style,
diff --git a/packages/lev/tests/test_engine_forward.py b/packages/lev/tests/test_engine_forward.py
index bc679da..b2d9dcf 100644
--- a/packages/lev/tests/test_engine_forward.py
+++ b/packages/lev/tests/test_engine_forward.py
@@ -62,7 +62,7 @@ def encode(self, text, add_special_tokens=True):
def test_without_score_averaging_a_large_request_answers_as_with_no_limit(tiny_model):
- """The row limit never touches a request Score averaging did not grow (ADR-029)."""
+ """The token limit never touches a request Score averaging did not grow (ADR-029)."""
from lev.types import Choice, Score
questions = {
@@ -73,6 +73,8 @@ def test_without_score_averaging_a_large_request_answers_as_with_no_limit(tiny_m
**{f"s{i}": Score(criteria=["low", "mid", "high"]) for i in range(40)},
}
limited = DecisionEngine(tiny_model, TinyTokenizer(), EngineConfig())
- unlimited = DecisionEngine(tiny_model, TinyTokenizer(), EngineConfig(max_request_rows=None))
- assert limited.prepare("state", questions).rows == 80 > limited.config.max_request_rows
- assert limited.system_one("state", questions) == unlimited.system_one("state", questions)
+ unlimited = DecisionEngine(tiny_model, TinyTokenizer(), EngineConfig(max_batch_tokens=None))
+ state = "a long support message " * 16
+ prepared = limited.prepare(state, questions)
+ assert prepared.rows == 80 and prepared.rows * prepared.width > limited.config.max_batch_tokens
+ assert limited.system_one(state, questions) == unlimited.system_one(state, questions)
diff --git a/packages/lev/tests/test_presentation.py b/packages/lev/tests/test_presentation.py
index 7ae072b..9f58e30 100644
--- a/packages/lev/tests/test_presentation.py
+++ b/packages/lev/tests/test_presentation.py
@@ -8,6 +8,7 @@
from __future__ import annotations
+import inspect
import json
import math
from pathlib import Path
@@ -15,11 +16,12 @@
import pytest
from lev import ChoiceAnswer, Score, ScoreAnswer, SystemOneRequest, SystemOneResponse, Usage
-from lev.model import MAX_REQUEST_ROWS, DecisionEngine, EngineConfig, average_orders
+from lev.model import MAX_BATCH_TOKENS, DecisionEngine, EngineConfig, average_orders
from lev.presentation import (
IDENTICAL_KS,
KEYS,
KINDS,
+ MAX_ROWS,
POST_LOAD,
PRE_LOAD,
SLOT0_MIN,
@@ -114,7 +116,7 @@ def system_one(self, state, questions):
return SystemOneResponse(model="fake", answers=answers, usage=Usage(input_tokens=1))
-def run(engine, states, lang="en", kinds=KINDS, max_rows=MAX_REQUEST_ROWS, slot0_min=SLOT0_MIN):
+def run(engine, states, lang="en", kinds=KINDS, max_rows=MAX_ROWS, slot0_min=SLOT0_MIN):
return report(ask(engine, states, lang, kinds, max_rows), lang, kinds, slot0_min)
@@ -376,7 +378,6 @@ def test_it_matches_prepare(self, poor_tokenizer, mode, order_average, noul_read
score_order_average=mode,
order_average=order_average,
noul_readout=noul_readout,
- max_request_rows=None,
),
mode_b_head=object(),
)
@@ -403,48 +404,85 @@ def test_schema_first_is_one_row_per_question(self, rich_tokenizer):
}
-class TestRequestRowLimit:
+LONG = "x" * 6000
+
+
+class TestAveragedTokenLimit:
+ """Only a request Score averaging grew can be refused, and only past the batch budget."""
+
def engine(self, tokenizer, **config) -> DecisionEngine:
return DecisionEngine(model=None, tokenizer=tokenizer, config=EngineConfig(**config))
- def test_a_request_past_the_limit_is_refused_before_any_forward(self, rich_tokenizer):
- questions = {f"q{i}": Score(criteria=["a", "b", "c"]) for i in range(4)}
- engine = self.engine(rich_tokenizer, score_order_average="cyclic", max_request_rows=11)
- with pytest.raises(ValueError, match="needs 12 batch rows, 8 of them from"):
- engine.prepare("s", questions)
- assert self.engine(rich_tokenizer, max_request_rows=11).prepare("s", questions).rows == 4
+ def test_a_long_state_with_few_averaged_rows_is_refused(self, rich_tokenizer):
+ question = {"urgency": Score(criteria=["low", "mid", "high"])}
+ engine = self.engine(rich_tokenizer, score_order_average="cyclic")
+ with pytest.raises(
+ ValueError, match="2 of the rows from score_order_average='cyclic', above"
+ ):
+ engine.prepare(LONG, question)
+ assert engine.prepare("short", question).rows == 3
+ assert self.engine(rich_tokenizer).prepare(LONG, question).rows == 1
def test_without_score_averaging_a_large_request_is_never_refused(self, rich_tokenizer):
"""#4's batcher runs an oversized request alone and unsplit, so main accepts it."""
- prepared = self.engine(rich_tokenizer).prepare("s", BIG)
- assert prepared.rows == 80 > MAX_REQUEST_ROWS
+ prepared = self.engine(rich_tokenizer).prepare("x" * 400, BIG)
+ assert prepared.rows == 80 and prepared.rows * prepared.width > MAX_BATCH_TOKENS
assert [v.order for v in prepared.variants] == [None, [3, 2, 1, 0]] * 20 + [None] * 40
- def test_only_rows_score_averaging_adds_can_refuse(self, rich_tokenizer):
+ def test_only_a_request_score_averaging_grew_can_be_refused(self, rich_tokenizer):
engine = self.engine(rich_tokenizer, score_order_average="cyclic")
- with pytest.raises(ValueError, match="needs 160 batch rows, 80 of them"):
- engine.prepare("s", BIG)
+ with pytest.raises(ValueError, match="80 of the rows from"):
+ engine.prepare("x" * 400, BIG)
choices = {k: q for k, q in BIG.items() if k.startswith("c")}
- assert engine.prepare("s", choices).rows == 40
+ assert engine.prepare("x" * 400, choices).rows == 40
- def test_none_is_no_limit(self, rich_tokenizer):
- questions = {f"q{i}": Score(criteria=["a", "b", "c"]) for i in range(30)}
- engine = self.engine(rich_tokenizer, score_order_average="cyclic", max_request_rows=None)
- assert engine.prepare("s", questions).rows == 90
- assert EngineConfig().max_request_rows == MAX_REQUEST_ROWS
+ def test_none_is_no_limit_and_the_default_is_the_batchers(self, rich_tokenizer):
+ from lev.batcher import Batcher
- def test_the_server_answers_422(self, monkeypatch, rich_tokenizer):
+ question = {"urgency": Score(criteria=["low", "mid", "high"])}
+ engine = self.engine(rich_tokenizer, score_order_average="cyclic", max_batch_tokens=None)
+ assert engine.prepare(LONG, question).rows == 3
+ default = inspect.signature(Batcher).parameters["max_batch_tokens"].default
+ assert EngineConfig().max_batch_tokens == MAX_BATCH_TOKENS == default == 16384
+
+ def serve(self, monkeypatch, tokenizer, **create):
pytest.importorskip("fastapi")
from fastapi.testclient import TestClient
from lev import server
- engine = self.engine(rich_tokenizer, score_order_average="cyclic", max_request_rows=5)
- monkeypatch.setattr(server, "load", lambda *a, **k: engine)
+ seen = {}
+
+ def load(*a, **kwargs):
+ seen["engine"] = self.engine(
+ tokenizer, score_order_average="cyclic", max_batch_tokens=kwargs["max_batch_tokens"]
+ )
+ return seen["engine"]
+
+ def batcher(engine, **kwargs):
+ seen["batcher"] = kwargs["max_batch_tokens"]
+ return real_batcher(engine, **kwargs)
+
+ real_batcher = server.Batcher
+ monkeypatch.setattr(server, "load", load)
+ monkeypatch.setattr(server, "Batcher", batcher)
+ return TestClient(server.create_app(**create)), seen
+
+ def test_the_server_gives_its_budget_to_the_engine(self, monkeypatch, rich_tokenizer):
+ client, seen = self.serve(monkeypatch, rich_tokenizer, max_batch_tokens=1234)
+ with client:
+ health = client.get("/health").json()
+ assert seen["batcher"] == seen["engine"].config.max_batch_tokens == 1234
+ assert health["max_batch_tokens"] == 1234
+
+ def test_the_server_answers_422(self, monkeypatch, rich_tokenizer):
+ client, _ = self.serve(monkeypatch, rich_tokenizer, max_batch_tokens=500)
questions = {f"q{i}": {"type": "score", "criteria": ["a", "b", "c"]} for i in range(2)}
- with TestClient(server.create_app()) as client:
- response = client.post("/v1/systemone", json={"state": "s", "questions": questions})
+ with client:
+ response = client.post(
+ "/v1/systemone", json={"state": "x" * 200, "questions": questions}
+ )
assert response.status_code == 422
- assert "max_request_rows=5" in response.json()["detail"]
+ assert "max_batch_tokens=500" in response.json()["detail"]
class TestCommittedReport:
@@ -455,7 +493,7 @@ def test_it_parses_and_carries_every_metric(self):
path = Path(__file__).resolve().parents[3] / "data" / "presentation-checks.json"
committed = json.loads(path.read_text("utf-8"))
assert committed["score_order_average"] == "off" and committed["n_states"] == 290
- assert committed["max_rows"] == MAX_REQUEST_ROWS
+ assert committed["max_rows"] == MAX_ROWS
for kind in KINDS:
assert committed["checks"][kind]["identical"]["read"] == "single order"
for check in ("identical", "first_slot"):
@@ -501,7 +539,7 @@ def test_matching_settings_are_compared_metric_by_metric(self, monkeypatch, tmp_
engine = FakeEngine(slot0=-1.0)
path = Path(self.committed(monkeypatch, tmp_path, engine))
saved = json.loads(path.read_text("utf-8"))
- assert saved["score_order_average"] == "off" and saved["max_rows"] == MAX_REQUEST_ROWS
+ assert saved["score_order_average"] == "off" and saved["max_rows"] == MAX_ROWS
saved["checks"]["score"]["identical"]["metric"] += 0.5
path.write_text(json.dumps(saved), "utf-8")
with pytest.raises(SystemExit) as exc:
From 771db93b6f57b6b916c97573e824a241ff9babdc Mon Sep 17 00:00:00 2001
From: Abhinavexists
Date: Tue, 6 Oct 2026 15:59:56 +0530
Subject: [PATCH 11/39] perf(model): merge the lora adapter when loading
---
packages/lev/src/lev/model.py | 7 +++++++
1 file changed, 7 insertions(+)
diff --git a/packages/lev/src/lev/model.py b/packages/lev/src/lev/model.py
index 1255222..b0a2c09 100644
--- a/packages/lev/src/lev/model.py
+++ b/packages/lev/src/lev/model.py
@@ -564,6 +564,7 @@ def load(
skip_multi_token_codes: bool = True,
score_order_average: Literal["off", "reversed", "cyclic"] = "off",
max_batch_tokens: int | None = MAX_BATCH_TOKENS,
+ merge_adapter: bool = True,
) -> DecisionEngine:
"""Load a checkpoint -- a release directory, a training output or a Hub id --
into a ready `DecisionEngine`. With no checkpoint, serves `model_id` frozen.
@@ -614,7 +615,13 @@ def load(
if resolved:
from peft import PeftModel
+ # Merged by default: the forward is launch-bound (ADR-023), so the per-layer
+ # lora_A/lora_B matmuls cost launches rather than arithmetic, and graph
+ # capture needs the weights fixed. `merge_adapter=False` keeps the adapter
+ # separate, which is what lets its scale be swept without retraining.
model = PeftModel.from_pretrained(model, str(resolved))
+ if merge_adapter:
+ model = model.merge_and_unload()
head = _load_head(resolved / MODE_B_HEAD, model)
model = model.eval()
From f1752fb2d2311249ee0d7b7de283afc7703f73cc Mon Sep 17 00:00:00 2001
From: Abhinavexists
Date: Tue, 6 Oct 2026 15:59:56 +0530
Subject: [PATCH 12/39] feat(modal): sweep the adapter scale on the rating
subsets
---
modal/app.py | 56 ++++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 56 insertions(+)
diff --git a/modal/app.py b/modal/app.py
index bf375cc..380634d 100644
--- a/modal/app.py
+++ b/modal/app.py
@@ -462,6 +462,62 @@ def check_release(name: str = "4b") -> dict:
return {"health": health, "response": served}
+@app.function(gpu="H100", volumes=VOLUMES, secrets=SECRETS, timeout=2 * 60 * 60)
+def diagnose_ordinal(
+ name: str = "lev",
+ scales: str = "0,0.25,0.5,0.75,1.0",
+ subsets: str = "summeval-consistency,summeval-relevance,helpsteer2",
+) -> dict:
+ """Score the rating subsets at several adapter scales, keeping the adapter unmerged.
+
+ summeval-consistency is 0.271 against the frozen backbone's 0.826, so the
+ regression may be recoverable by scaling the adapter rather than retraining
+ (WiSE-FT, arXiv 2109.01903, which for a LoRA is just the adapter scale).
+ Scale 0 must reproduce the frozen backbone; that is the run's own control.
+ Also reports the argmax histogram, which levbench aggregates away.
+ """
+ import collections
+ import json
+
+ from lev import load
+ from lev.data.s1bench import load_eval_subset
+ from peft.tuners.lora import LoraLayer
+
+ engine = load(f"{RELEASES_DIR}/{name}", cache_dir=MODELS_DIR, merge_adapter=False)
+ layers = [m for m in engine.model.modules() if isinstance(m, LoraLayer)]
+ if not layers:
+ raise RuntimeError("no LoraLayer found; the adapter was merged or never applied")
+ adapters = sorted({a for m in layers for a in m.scaling})
+ print(f"{len(layers)} lora layers, adapters {adapters}")
+
+ data = {s: load_eval_subset(s) for s in subsets.split(",")}
+ report: dict = {"layers": len(layers), "subsets": {}}
+ for scale in [float(x) for x in scales.split(",")]:
+ for m in layers:
+ for a in m.scaling:
+ m.set_scale(a, scale)
+ for subset, (question, items) in data.items():
+ hits = 0
+ picks: collections.Counter = collections.Counter()
+ for item in items:
+ answer = engine.system_one(item.state, {"decision": question}).answers["decision"]
+ probs = {int(k): float(v) for k, v in answer.probabilities.items()}
+ pick = max(probs, key=probs.get)
+ picks[pick] += 1
+ hits += pick == item.truth
+ row = report["subsets"].setdefault(subset, {})
+ row[f"{scale:g}"] = {
+ "accuracy": round(hits / len(items), 4),
+ "n": len(items),
+ "argmax": dict(sorted(picks.items())),
+ }
+ acc = hits / len(items)
+ print(f" scale {scale:<5g} {subset:22} {acc:.4f} {dict(sorted(picks.items()))}")
+
+ print(json.dumps(report, indent=2))
+ return report
+
+
@app.function(gpu="H100", volumes=VOLUMES, secrets=SECRETS, timeout=30 * 60)
def diagnose_candidates(model_id: str = "Qwen/Qwen3.5-4B-Base") -> dict:
"""Compare candidate representations alone, batched, reversed and left-padded.
From a7e045e6983db07b614bd4a82526a67fcaef0ddf Mon Sep 17 00:00:00 2001
From: Abhinavexists
Date: Tue, 6 Oct 2026 15:59:56 +0530
Subject: [PATCH 13/39] docs(decisions): correct the torch.compile rejection in
ADR-023
---
docs/DECISIONS.md | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/docs/DECISIONS.md b/docs/DECISIONS.md
index 9b6e762..37d2487 100644
--- a/docs/DECISIONS.md
+++ b/docs/DECISIONS.md
@@ -482,7 +482,7 @@ Both are flat in the number of questions, which says the forward is bound by ker
With `causal_conv1d` present (the image now builds it against `torch 2.14.0+cu130`): fork 140-156 ms, single 65-75 ms -- about 10% each. That the kernel bought so little is the diagnosis confirmed from a second angle: the time is in launching ~200 kernels twice, not in any one of them. What removes launches is CUDA-graph replay, so the engine gains `EngineConfig.compile` (`torch.compile(mode="reduce-overhead", dynamic=True)`) with inputs padded to shape buckets -- width to 32 tokens, rows to a power of two -- so the set of recorded graphs stays small, and a `warmup()` the server runs at startup so no request pays a capture. Padding changes no real row's logits: right padding is masked and rows are independent. The GPU forward is taken under a lock, because graph replay is not thread-safe and the server now accepts concurrent requests; parsing, tokenising and the network overlap around it.
-**Compile: measured, rejected.** On the same container, eager single-forward ran 86–94 ms and the compiled path 103–137 ms, after a 104 s warmup; dynamo hit its recompile limit (8) on the hybrid's linear-attention layers, so much of the forward stayed eager with extra guard overhead, and the Mode B shape then failed with a CUDA-graphs output-buffer overwrite when hidden states from one replay were read after the next. The flag stays (`LEV_SERVE_COMPILE=1`) so the measurement can be repeated on a future transformers/FLA release, and defaults off. Getting decider's 8 ms would mean a static-shape engine built for this model, not `torch.compile` over the stock one.
+**Compile: measured, rejected.** On the same container, eager single-forward ran 86–94 ms and the compiled path 103–137 ms, after a 104 s warmup; `torch.compile` cannot reach this model's dominant cost at all: FLA decorates `chunk_gated_delta_rule` and `fused_gdn_gate` with `@torch.compiler.disable` (`fla/ops/gated_delta_rule/chunk.py`, `gate.py`), so each of the 24 GDN layers is a permanent graph break and stays eager at full launch cost whatever the mode. What is left to capture is the 8 attention layers, the MLPs and the norms, while every partition boundary copies inputs to static buffers -- overhead without the win. Dynamo's `recompile_limit` (8, per code object, so all 24 GDN layers share one budget) compounded it, and the Mode B shape then failed with a CUDA-graphs output-buffer overwrite when hidden states from one replay were read after the next. `dynamic=True` was not the cause: it does not veto graph capture. The flag stays (`LEV_SERVE_COMPILE=1`) so the measurement can be repeated once FLA lifts the decorator ([fla-org RFC #1155](https://github.com/fla-org/flash-linear-attention/issues/1155) tracks graph capture for the GDN paths), and defaults off. Getting decider's 8 ms means manual `torch.cuda.CUDAGraph` capture over the eager forward, one graph per padded length bucket, warmed eagerly first so the Triton autotune choice is fixed before capture -- not `torch.compile` over the stock model.
Also decided here, from the same session's measurements (FINDINGS.md §13): the image builds `causal_conv1d` from a CUDA devel base, since no wheel exists for `torch 2.14.0+cu130` and every log said the reference path was in use; and the serving path was verified correct end to end -- in-distribution accuracy over HTTP matches the collator-path `evaluate` on every source -- so no inference-side error is hiding behind the S1Bench gap.
From 724f6e785b3b5fd9c68c5ac550e3bf02da291f22 Mon Sep 17 00:00:00 2001
From: Abhinavexists
Date: Wed, 7 Oct 2026 00:04:35 +0530
Subject: [PATCH 14/39] refactor: tighten comments, docstrings and naming
across the repo
---
.env.example | 9 +-
CONTRIBUTING.md | 2 +-
Makefile | 81 ++++----
data/adr-029-summary.json | 24 +--
docs/ARCHITECTURE.md | 4 +-
docs/DECISIONS.md | 4 +-
docs/charts/build.py | 21 +-
docs/charts/levbench_log.py | 4 +-
modal/app.py | 117 ++---------
packages/lev/src/lev/__init__.py | 5 -
packages/lev/src/lev/batcher.py | 95 +++++----
packages/lev/src/lev/calibrate.py | 174 ++++++++++++-----
packages/lev/src/lev/cli.py | 17 +-
packages/lev/src/lev/data/build.py | 16 +-
packages/lev/src/lev/data/contamination.py | 25 +--
packages/lev/src/lev/data/export_eval.py | 13 +-
packages/lev/src/lev/data/mixture.py | 44 ++---
packages/lev/src/lev/data/s1bench.py | 30 +--
packages/lev/src/lev/data/sources.py | 130 +++++--------
packages/lev/src/lev/data/splits.py | 18 +-
packages/lev/src/lev/labels.py | 27 +--
packages/lev/src/lev/model.py | 125 ++++--------
packages/lev/src/lev/presentation.py | 59 ++----
packages/lev/src/lev/prompt.py | 44 ++---
packages/lev/src/lev/readout/mode_a.py | 13 +-
packages/lev/src/lev/readout/mode_b.py | 16 +-
packages/lev/src/lev/release.py | 26 +--
packages/lev/src/lev/router.py | 8 +-
packages/lev/src/lev/server.py | 18 +-
packages/lev/src/lev/train/calibration_run.py | 16 +-
packages/lev/src/lev/train/checkpoints.py | 34 +---
packages/lev/src/lev/train/collate.py | 54 +++---
packages/lev/src/lev/train/config.py | 13 +-
packages/lev/src/lev/train/evaluate.py | 12 +-
packages/lev/src/lev/train/loop.py | 59 ++----
packages/lev/src/lev/train/progress.py | 10 +-
packages/lev/src/lev/types.py | 10 +-
packages/lev/tests/conftest.py | 12 +-
packages/lev/tests/test_batcher.py | 12 +-
packages/lev/tests/test_cache_fork.py | 5 +-
packages/lev/tests/test_calibration.py | 20 +-
packages/lev/tests/test_collate.py | 15 +-
packages/lev/tests/test_contamination.py | 5 +-
packages/lev/tests/test_core.py | 41 ++--
packages/lev/tests/test_data_pipeline.py | 95 ++++-----
packages/lev/tests/test_engine_forward.py | 1 +
packages/lev/tests/test_eval_export.py | 17 +-
packages/lev/tests/test_evaluate.py | 4 +-
packages/lev/tests/test_loss.py | 11 +-
packages/lev/tests/test_packaging.py | 11 +-
packages/lev/tests/test_presentation.py | 40 ++--
packages/lev/tests/test_progress.py | 19 +-
packages/lev/tests/test_release.py | 6 +-
packages/lev/tests/test_s1bench.py | 12 +-
packages/lev/tests/test_training_loop.py | 36 ++--
packages/levbench/src/levbench/__init__.py | 6 +-
packages/levbench/src/levbench/batching.py | 20 +-
packages/levbench/src/levbench/cli.py | 9 +-
.../levbench/src/levbench/confidence_id.py | 38 ++--
packages/levbench/src/levbench/metrics.py | 20 +-
packages/levbench/src/levbench/pricing.py | 11 +-
packages/levbench/src/levbench/runner.py | 40 ++--
.../levbench/src/levbench/snake/__init__.py | 7 +-
packages/levbench/src/levbench/snake/game.py | 10 +-
.../levbench/src/levbench/snake/policy.py | 18 +-
packages/levbench/src/levbench/snake/run.py | 38 ++--
packages/levbench/src/levbench/snake/ui.py | 8 +-
packages/levbench/src/levbench/tasks.py | 13 +-
packages/levbench/tests/fake_transport.py | 29 ++-
packages/levbench/tests/test_offline.py | 181 ++++++++----------
packages/levbench/tests/test_snake.py | 95 +++++----
pyproject.toml | 2 +-
scripts/publish_hf.sh | 24 ++-
73 files changed, 904 insertions(+), 1404 deletions(-)
diff --git a/.env.example b/.env.example
index 4a8a40f..abf1c78 100644
--- a/.env.example
+++ b/.env.example
@@ -1,11 +1,6 @@
-# Benchmarking against the hosted Jev API.
TYPESAFE_API_KEY=
-
-# The LLM baseline `levbench compare` measures against.
ANTHROPIC_API_KEY=
-# Only if you front a local /v1/systemone server with auth. Your real
-# TYPESAFE_API_KEY is deliberately never sent to a --base-url host.
-# Left commented on purpose: `VAR=` sets the variable to the empty string,
-# which is not the same as leaving it unset.
+# Optional local-server auth; TYPESAFE_API_KEY is never sent to --base-url hosts.
+# Leave commented to keep unset; VAR= assigns an empty string.
# LEVBENCH_LOCAL_API_KEY=
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
index d598c1c..996f384 100644
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@@ -6,7 +6,7 @@
make check # ruff + the full test suite
```
-Both must be clean. Tests run without a GPU, network or API keys. A basic `uv sync` skips tests that need optional model dependencies; use `uv sync --extra train` to include the CPU tensor, cache and training-loop tests.
+Both must be clean. Tests run without a GPU, network or API keys. use `uv sync --extra train` to include the CPU tensor, cache and training-loop tests.
## Finding the code
diff --git a/Makefile b/Makefile
index e3d49b6..808d89a 100644
--- a/Makefile
+++ b/Makefile
@@ -1,4 +1,15 @@
-# `make help` lists every target. Variables are overridable: `make train PRESET=9b`.
+# setup: Install workspace | setup-train: + training deps
+# setup-serve: + serving deps | setup-modal: + Modal client
+
+# test: Run tests | lint: Check code | fmt: Format code
+# check: Lint + test | clean: Remove caches
+
+# smoke-local: CPU smoke test | smoke: Modal H100 smoke test
+# train: Full training run | calibrate: Fit temperatures
+
+# serve: Dev server | deploy: Stable endpoint
+# release: Package checkpoint | weights: Pull weights | publish: Push to HF
+
.DEFAULT_GOAL := help
PRESET ?= 4b-instruct
@@ -15,111 +26,101 @@ URL ?= http://localhost:8000
FRESH ?=
RESUME ?=
-help: ## Show this help
- @awk 'BEGIN{FS=":.*?## "} \
- /^##@/ {printf "\n\033[1m%s\033[0m\n", substr($$0, 5)} \
- /^[a-zA-Z0-9_-]+:.*?## / {printf " \033[36m%-12s\033[0m %s\n", $$1, $$2}' $(MAKEFILE_LIST)
-
-##@ Setup
+help:
+ @awk '/^#/ {sub(/^# ?/, ""); print; next} /^$$/ {print; next} {exit}' $(MAKEFILE_LIST)
.PHONY: setup setup-train setup-serve setup-modal
-setup: ## Install the workspace
+setup:
uv sync
-setup-train: ## Install with the training extras (torch, transformers, peft)
+setup-train:
uv sync --extra train
-setup-serve: ## Install with the serving extras (adds fastapi, uvicorn)
+setup-serve:
uv sync --extra serve
-setup-modal: ## Install with the Modal client
+setup-modal:
uv sync --extra modal
-##@ Develop
-
.PHONY: test lint fmt check clean
-test: ## Run every test
+test:
uv run pytest
-lint: ## Check formatting and lint rules
+lint:
uv run ruff check .
uv run ruff format --check .
-fmt: ## Apply formatting and autofixes
+fmt:
uv run ruff check --fix .
uv run ruff format .
-check: lint test ## Lint then test — run before pushing
+check: lint test
-clean: ## Remove caches and build artefacts
+clean:
find . -type d -name __pycache__ -prune -exec rm -rf {} +
rm -rf .pytest_cache .ruff_cache dist build
-##@ Data
.PHONY: plan check-data data eval-set s1bench
-plan: ## Print the H100 training budget without spending it
+plan:
uv run lev plan --preset $(PRESET)
-check-data: ## Contamination guard over a source list (SOURCES=path)
+check-data:
uv run lev check-data $(SOURCES)
-data: ## Build the training mixture locally (LIMIT=rows per source)
+data:
uv run lev data build --out $(OUT) --limit-per-source $(LIMIT)
-eval-set: ## Export the held-out split as levbench task files
+eval-set:
uv run lev data eval --data $(OUT) --out $(EVAL)
-s1bench: ## Export the S1Bench eval subsets as levbench task files
+s1bench:
uv run lev s1bench export --out $(S1)
-##@ Train
.PHONY: smoke-local smoke train calibrate
-smoke-local: ## Train 0.8B on CPU for a few steps — proves the path without Modal
+smoke-local:
uv run lev data build --out /tmp/lev-mix --limit-per-source 2000 --n-examples 4000
uv run lev train --preset smoke --data /tmp/lev-mix \
--output-dir /tmp/lev-smoke --max-steps $(STEPS)
-smoke: ## Modal: exercise the whole training path on 0.8B (~5 min of H100)
+smoke:
modal run modal/app.py::smoke
-train: ## Modal: the real run; resumes the newest checkpoint (FRESH=1 to start over, RESUME=path)
+train:
modal run modal/app.py::train --preset $(PRESET) $(if $(FRESH),--fresh,) $(if $(RESUME),--resume $(RESUME),)
-calibrate: ## Modal: fit per-bucket temperatures after training
+calibrate:
modal run modal/app.py::calibrate --preset $(PRESET)
-##@ Serve and release
.PHONY: serve deploy release weights publish
-serve: ## Modal: dev server on an H100 (ephemeral; any `modal run` on this app steals its URL)
+serve:
LEV_SERVE_PRESET=$(PRESET) modal serve modal/app.py
-deploy: ## Modal: deploy /v1/systemone with a stable URL (PRESET selects the checkpoint)
+deploy:
LEV_SERVE_PRESET=$(PRESET) modal deploy modal/app.py
-release: ## Modal: package the newest PRESET checkpoint into a release dir on the volume
+release:
modal run modal/app.py::export_checkpoint --preset $(PRESET) $(if $(NAME),--name $(NAME),)
-weights: ## Pull a packaged release to weights/RELEASE (RELEASE=name, default PRESET)
+weights:
mkdir -p weights
modal volume get --force lev-checkpoints releases/$(RELEASE) weights/
-publish: ## Upload weights/RELEASE to the Hub (REPO=org/name; needs HF_TOKEN)
+publish:
uv run lev release publish weights/$(RELEASE) --repo $(REPO)
-##@ Benchmark
.PHONY: bench bench-local sweep snake
-bench: s1bench ## Benchmark Jev on the S1Bench subsets (needs TYPESAFE_API_KEY)
+bench: s1bench
uv run levbench eval --backend jev --tasks $(S1)
-bench-local: s1bench ## Benchmark a /v1/systemone server on the S1Bench subsets (URL=server)
+bench-local: s1bench
uv run levbench eval --backend lev --base-url $(URL) --tasks $(S1)
-sweep: ## Measure shared-state batching economics
+sweep:
uv run levbench sweep --backend jev
-snake: ## The decision model plays Snake over /v1/systemone (URL=server; --record in artifacts/)
+snake:
uv run levbench snake --backend lev --base-url $(URL) --record artifacts/snake/run.jsonl
diff --git a/data/adr-029-summary.json b/data/adr-029-summary.json
index 3348833..da41b22 100644
--- a/data/adr-029-summary.json
+++ b/data/adr-029-summary.json
@@ -403,14 +403,14 @@
"rps": 0.152,
"ece15": 0.1296,
"ece15_equal_mass": 0.1683,
- "nll": 0.9348
+ "negative_log_likelihood": 0.9348
},
"held_out_refit": {
"acc": 0.5903,
"rps": 0.1466,
"ece15": 0.108,
"ece15_equal_mass": 0.134,
- "nll": 0.9087
+ "negative_log_likelihood": 0.9087
}
},
"reversed": {
@@ -423,14 +423,14 @@
"rps": 0.1264,
"ece15": 0.1108,
"ece15_equal_mass": 0.1453,
- "nll": 0.8388
+ "negative_log_likelihood": 0.8388
},
"held_out_refit": {
"acc": 0.6736,
"rps": 0.1263,
"ece15": 0.1193,
"ece15_equal_mass": 0.1668,
- "nll": 0.8381
+ "negative_log_likelihood": 0.8381
}
},
"cyclic": {
@@ -443,14 +443,14 @@
"rps": 0.1238,
"ece15": 0.0775,
"ece15_equal_mass": 0.1387,
- "nll": 0.8279
+ "negative_log_likelihood": 0.8279
},
"held_out_refit": {
"acc": 0.6597,
"rps": 0.1238,
"ece15": 0.0765,
"ece15_equal_mass": 0.1382,
- "nll": 0.8284
+ "negative_log_likelihood": 0.8284
}
}
},
@@ -466,14 +466,14 @@
"rps": 0.1321,
"ece15": 0.0896,
"ece15_equal_mass": 0.1275,
- "nll": 0.7807
+ "negative_log_likelihood": 0.7807
},
"held_out_refit": {
"acc": 0.6577,
"rps": 0.13,
"ece15": 0.0772,
"ece15_equal_mass": 0.0914,
- "nll": 0.7792
+ "negative_log_likelihood": 0.7792
}
},
"reversed": {
@@ -486,14 +486,14 @@
"rps": 0.1015,
"ece15": 0.0888,
"ece15_equal_mass": 0.1219,
- "nll": 0.641
+ "negative_log_likelihood": 0.641
},
"held_out_refit": {
"acc": 0.7584,
"rps": 0.1008,
"ece15": 0.0611,
"ece15_equal_mass": 0.1142,
- "nll": 0.6267
+ "negative_log_likelihood": 0.6267
}
},
"cyclic": {
@@ -506,14 +506,14 @@
"rps": 0.1013,
"ece15": 0.0681,
"ece15_equal_mass": 0.0959,
- "nll": 0.6436
+ "negative_log_likelihood": 0.6436
},
"held_out_refit": {
"acc": 0.7517,
"rps": 0.1006,
"ece15": 0.0865,
"ece15_equal_mass": 0.0909,
- "nll": 0.6286
+ "negative_log_likelihood": 0.6286
}
}
}
diff --git a/docs/ARCHITECTURE.md b/docs/ARCHITECTURE.md
index 770e9bf..9772bdc 100644
--- a/docs/ARCHITECTURE.md
+++ b/docs/ARCHITECTURE.md
@@ -107,7 +107,7 @@ This is our plan's §14 ("encode the question, don't build per-question heads")
### 3.6 Training objectives — from `NanoJev` and `decider`
- NanoJev: CE / Brier / paired-proper-reward variants, with exact gradient checks.
-- decider: cross-entropy fine-tune with **random layout per example** and **abstain augmentation**; evaluation reports accuracy / NLL / Brier / ECE / AURC / selective accuracy per task; v10 adds **RL with a proper-score belief reward** — the open analogue of RLCD.
+- decider: cross-entropy fine-tune with **random layout per example** and **abstain augmentation**; evaluation reports accuracy / negative_log_likelihood / Brier / ECE / AURC / selective accuracy per task; v10 adds **RL with a proper-score belief reward** — the open analogue of RLCD.
Supervised proper-scoring first, RL only after. That vindicates the plan's §17 ordering.
@@ -237,7 +237,7 @@ Three splits, not two. Temperature is fitted on a split disjoint from **both** t
1. Train with a proper scoring rule (cross-entropy + Brier).
2. **Abstain augmentation** (decider): examples where the answer is not determinable from the state, teaching the model to spread mass rather than guess confidently.
3. Post-hoc temperature fit on the calibration split.
-4. Report accuracy, NLL, Brier and **ECE** per source, with and without the temperature.
+4. Report accuracy, negative_log_likelihood, Brier and **ECE** per source, with and without the temperature.
Evidence this is worth the effort: the untuned Qwen3-8B (`qwen3-8b-full`) sits at **ECE 0.4252**; reflex, Qwen3.5-4B plus one fitted scalar, reaches **0.0849**. Jev is **0.0764**. **A single scalar is most of the gap.**
diff --git a/docs/DECISIONS.md b/docs/DECISIONS.md
index 37d2487..bc4610e 100644
--- a/docs/DECISIONS.md
+++ b/docs/DECISIONS.md
@@ -634,7 +634,7 @@ Accuracy, RPS, Choice and Noul are identical, to the last digit, to the runs bef
Calibration of the averaged modes. The shipped Score temperature (2.80) was fitted on single-order rows. Refit per mode on half of each bench (stratified by the gold level, seed 0), judged on the other half:
```text
- T refit held-out NLL held-out ECE held-out RPS
+ T refit held-out negative_log_likelihood held-out ECE held-out RPS
shipped refit shipped refit shipped refit
en (144) off 3.96 0.935 0.909 0.130 0.108 0.152 0.147
reversed 3.16 0.839 0.838 0.111 0.119 0.126 0.126
@@ -644,7 +644,7 @@ Calibration of the averaged modes. The shipped Score temperature (2.80) was fitt
cyclic 2.40 0.644 0.629 0.068 0.086 0.101 0.101
```
-On bench_en a refit gains nothing for either averaged mode (held-out NLL within 0.001), and the shipped temperature is if anything a little sharp for `off`. On the Japanese reference the refit goes the other way for the averaged modes. A per-mode temperature is not proposed.
+On bench_en a refit gains nothing for either averaged mode (held-out negative_log_likelihood within 0.001), and the shipped temperature is if anything a little sharp for `off`. On the Japanese reference the refit goes the other way for the averaged modes. A per-mode temperature is not proposed.
The batch budget for averaged requests (`max_batch_tokens`, 16384). The batcher does not split a request: on main, 20 Choice and 40 Score questions with averaging off ran as one forward of 80 rows, up to 35,680 padded tokens. So `prepare` refuses only a request that Score averaging grew, when its padded tokens (rows x widest row) pass the batcher's own budget; the server passes its `max_batch_tokens` to the engine, so the two always agree, and a request without averaged Score rows is never refused (the 80-row request above answers exactly as on main). The value is the batcher's; this run only checks it on the RTX 5090, with the largest averaged request the budget admits at each row length, averaged Score questions topped up with one-row Noul (one question more is refused in every case):
diff --git a/docs/charts/build.py b/docs/charts/build.py
index 53cdc1a..bee494b 100644
--- a/docs/charts/build.py
+++ b/docs/charts/build.py
@@ -1,16 +1,5 @@
# ruff: noqa: E501 -- SVG and HTML markup, one element per line
-"""Render docs/charts/lev-vs-jev.html from measured logs and recorded history.
-
- uv run python docs/charts/build.py
-
-Inputs, all checked in:
- logs/jev-s1bench-*.txt, logs/lev-s1bench-*.txt both on the 13 S1Bench task files, same laptop
- logs/pre-nimble/ earlier six-subset definitions, the scale of history.json
- history.json lev iterations and speed measurements, with FINDINGS sections
- ../../data/s1bench-snapshot.json the public S1Bench board: every other model
-
-lev blue and Jev orange (CVD-checked in both modes); every other model is context gray.
-"""
+"""Render docs/charts/lev-vs-jev.html from checked-in logs, history, and the S1Bench snapshot."""
from __future__ import annotations
@@ -25,7 +14,6 @@
HERE = Path(__file__).parent
SNAPSHOT = HERE.parent.parent / "data" / "s1bench-snapshot.json"
-# The six subsets every model on the board completed; the leaderboard is over these.
SUBSETS = ["aegis2", "boolq", "helpsteer2", "massive-en-US", "paws", "vitaminc-dev"]
ALL_SUBSETS = list(EVAL_SUBSETS)
W = 760
@@ -142,7 +130,6 @@ def grouped_bars(cats, series, vmax, fmt, ticks, label_w=118) -> str:
def scatter(
points, xr, yr, xfmt, yfmt, xticks, yticks, xlabel, ylabel, xlog=False, hline=None
) -> str:
- """Points: dicts with x, y, name, var, and label (direct-label it)."""
h, left, right, top, bottom = 380, 58, 24, 14, 44
pw, ph = W - left - right, h - top - bottom
tx = math.log10 if xlog else (lambda v: v)
@@ -169,7 +156,7 @@ def scatter(
f''
f'{esc(hline["name"])}'
)
- for p in sorted(points, key=lambda p: p["var"] != CTX): # context underneath
+ for p in sorted(points, key=lambda p: p["var"] != CTX):
cx, cy = px(p["x"]), py(p["y"])
r = 6 if p["var"] != CTX else 4.5
tip = f"{p['name']}|{xfmt(p['x'])} · {yfmt(p['y'])}"
@@ -294,8 +281,6 @@ def majority_rate(subset: str) -> float:
def completed_targets(board: dict) -> list[dict]:
- """Board models that completed all six subsets; a model stopped after one
- subset reports a one-subset macro and is not comparable."""
return [
t
for t in board["targets"]
@@ -316,7 +301,6 @@ def leaderboard_row(name: str, value: float, var: str, size: str, ece: float, bo
def leaderboard_rows(done: list[dict], data: Inputs) -> list[dict]:
- """Board models plus lev and Jev as measured here, best first."""
rows = []
for t in done:
is_jev = t["target"] == "jev"
@@ -526,7 +510,6 @@ def size_section(points: list[dict], jev_board: dict) -> str:
def per_subset_series(data: Inputs, value) -> list[dict]:
- """lev and Jev as grouped-bar series, `value(scores)` per subset."""
return [
{"name": "lev", "var": LEV, "values": [value(data.lev[s]) for s in ALL_SUBSETS]},
{"name": "Jev", "var": JEV, "values": [value(data.jev[s]) for s in ALL_SUBSETS]},
diff --git a/docs/charts/levbench_log.py b/docs/charts/levbench_log.py
index 9fa5780..36a53aa 100644
--- a/docs/charts/levbench_log.py
+++ b/docs/charts/levbench_log.py
@@ -20,8 +20,6 @@
def parse(text: str) -> dict:
- """`{"subsets": {subset: {n, accuracy, log_loss, brier, ece, p50_s, ...}}}`, plus
- `served_by` when the log names the serving model."""
out: dict = {"subsets": {}}
marks = list(_HEADER.finditer(text))
for i, mark in enumerate(marks):
@@ -29,7 +27,7 @@ def parse(text: str) -> dict:
scores = _SCORES.search(body)
if not scores:
continue
- row = {"n": int(mark.group(2))}
+ row: dict[str, float | int] = {"n": int(mark.group(2))}
row.update(
zip(("accuracy", "log_loss", "brier", "ece"), map(float, scores.groups()), strict=True)
)
diff --git a/modal/app.py b/modal/app.py
index 380634d..3586ba5 100644
--- a/modal/app.py
+++ b/modal/app.py
@@ -1,8 +1,6 @@
-"""Build data, train, calibrate, evaluate and serve lev on Modal. See docs/SETUP.md.
+"""Build, train, calibrate, evaluate, and serve lev on Modal (SETUP.md).
-Use `modal deploy` for a stable URL: other ephemeral runs can take over `modal serve`'s
-`-dev` label.
-"""
+Use modal deploy for a stable URL; modal serve uses a shared development label."""
from __future__ import annotations
@@ -25,14 +23,10 @@
APP_NAME = "lev"
-# From this file, not the cwd: `modal run` may start anywhere, and a relative path mounts nothing.
REPO_ROOT = Path(__file__).resolve().parent.parent
-# Pinned for reproducibility. `transformers` must be 5.x: the prefix-cache fork
-# calls `reorder_cache` on a hybrid cache, which 4.x cannot do.
+# Pin dependencies; hybrid-cache reordering requires Transformers 5.x.
image = (
- # Devel base for the `nvcc` that `causal-conv1d` builds with (served compute 81 -> 69 ms,
- # ADR-023). The CUDA major must match torch's build (2.14.0+cu130).
modal.Image.from_registry("nvidia/cuda:13.0.3-devel-ubuntu24.04", add_python="3.12")
.pip_install(
"torch==2.14.0",
@@ -45,14 +39,10 @@
"fastapi==0.141.1",
"huggingface-hub==1.32.0",
)
- # Speed, not correctness: if the conv build breaks on a pin change, delete it.
.pip_install("flash-linear-attention", "ninja", "packaging")
- # The devel image has nvcc but no host C++ compiler, which torch's extension
- # builder requires ("clang++ 0.0.0"). Arch 9.0 only: this image runs on H100s.
.apt_install("build-essential")
.env({"CC": "gcc", "CXX": "g++", "TORCH_CUDA_ARCH_LIST": "9.0", "MAX_JOBS": "8"})
- # Length-bucketed batches vary widely in shape; without this the caching
- # allocator fragments (the OOM at step 6,075 had 29 GiB reserved-but-free).
+ # Expandable segments reduce fragmentation from varying batch shapes.
.env({"PYTORCH_CUDA_ALLOC_CONF": "expandable_segments:True"})
.pip_install("causal-conv1d", extra_options="--no-build-isolation")
.add_local_dir(
@@ -78,17 +68,11 @@
DATA_DIR: datasets_vol,
}
-# The preset `serve` exposes when `LEV_SERVE_PRESET` is unset. Not a function
-# argument: Modal requires `@modal.asgi_app` functions to take none.
+# Use an environment preset because Modal ASGI functions accept no arguments.
SERVE_PRESET = "4b"
def _hf_secrets() -> list:
- """The Secret named by `LEV_HF_SECRET`, else the local `HF_TOKEN`, else none.
-
- Never an unconditional `Secret.from_name`: it fails at function creation when the
- Secret is missing, and the backbones are public.
- """
named = os.environ.get("LEV_HF_SECRET")
if named:
return [modal.Secret.from_name(named, required_keys=[])]
@@ -96,13 +80,6 @@ def _hf_secrets() -> list:
return [modal.Secret.from_dict({"HF_TOKEN": token})] if token else []
-# Deploy knobs (docs/SETUP.md), read where `modal deploy` runs: decorator arguments are
-# fixed at import, so these cannot travel as Secrets.
-# LEV_SERVE_CONCURRENCY per-container inputs, also the server's max_pending (32; ADR-030)
-# LEV_SERVE_WARM containers kept running (0); one removes the 20-55 s cold start
-# LEV_SERVE_REGION Modal region near the client (unset); the measured 280 ms RTT is distance
-# LEV_SERVE_SCALEDOWN idle seconds before a container stops (300)
-# LEV_SERVE_MAX container ceiling (unset: Modal's limit); pin it for benchmark sweeps
SERVE_CONCURRENCY = int(os.environ.get("LEV_SERVE_CONCURRENCY", "32"))
SERVE_WARM = int(os.environ.get("LEV_SERVE_WARM", "0"))
SERVE_REGION = os.environ.get("LEV_SERVE_REGION")
@@ -119,7 +96,6 @@ def _prompt_style(value: str | None) -> Literal["plain", "chat"] | None:
def _serve_overrides() -> list:
- """The set `LEV_SERVE_*` overrides as a Secret: local env does not reach the container."""
overrides: dict[str, str | None] = {
key: value
for key in (
@@ -141,10 +117,6 @@ def _serve_overrides() -> list:
@app.function(volumes={MODELS_DIR: models}, secrets=SECRETS, timeout=60 * 60)
def download(model_id: str = "Qwen/Qwen3.5-4B-Base") -> str:
- """Pre-fetch a checkpoint into the models Volume. Idempotent.
-
- `cache_dir`, not `local_dir`: `from_pretrained(..., cache_dir=)` expects the HF cache layout.
- """
from huggingface_hub import snapshot_download
path = snapshot_download(
@@ -161,7 +133,7 @@ def download(model_id: str = "Qwen/Qwen3.5-4B-Base") -> str:
gpu="H100",
volumes=VOLUMES,
secrets=SECRETS,
- timeout=24 * 60 * 60, # the released run took 7.8 h; a truncated run costs everything
+ timeout=24 * 60 * 60, # Allow the full training run to finish.
)
def train(
preset: str = "4b",
@@ -170,11 +142,6 @@ def train(
max_steps: int | None = None,
fresh: bool = False,
) -> dict:
- """Fine-tune on one H100. See `lev.train.config.PRESETS`.
-
- Resumes from the newest checkpoint in the preset's directory unless
- `--fresh`; `--resume ` names one explicitly.
- """
from lev.train.config import PRESETS
from lev.train.loop import run_training
@@ -204,10 +171,6 @@ def train(
@app.function(volumes={DATA_DIR: datasets_vol}, secrets=SECRETS, timeout=4 * 60 * 60)
def build_data(limit_per_source: int = 20_000, n_examples: int = 200_000) -> dict:
- """Download every source and write the three splits to the data volume. CPU only.
-
- Run it once, before `train`.
- """
from lev.data.build import build_dataset
# The HF download cache lives on the volume, so rebuilds reuse the corpora.
@@ -223,10 +186,6 @@ def build_data(limit_per_source: int = 20_000, n_examples: int = 200_000) -> dic
@app.function(gpu="H100", volumes=VOLUMES, secrets=SECRETS, timeout=2 * 60 * 60)
def smoke(steps: int = 40) -> dict:
- """Exercise every part of `train` but its duration, on the 0.8B preset. Run this first.
-
- Builds a small mixture if the data volume is empty.
- """
if not (Path(DATA_DIR) / "train.jsonl").is_file():
# Builds on the GPU, acceptable only for this small mixture (a couple of minutes).
print("data volume is empty; building a small mixture first")
@@ -247,21 +206,16 @@ def smoke(steps: int = 40) -> dict:
@app.function(gpu="H100", volumes=VOLUMES, secrets=SECRETS, timeout=60 * 60)
def calibrate(preset: str = "4b", split: str = "calibration", method: str = "transfer") -> dict:
- """Fit per-bucket temperatures on a split that is neither train nor test.
-
- Separate from `train` so re-fitting needs no fine-tune.
- """
from lev.train.calibration_run import fit_profile
from lev.train.config import PRESETS
config = replace(PRESETS[preset], output_dir=f"{CKPT_DIR}/{preset}")
return fit_profile(
- # `fit_profile` resolves the newest checkpoint inside this directory.
checkpoint_dir=config.output_dir,
data_dir=DATA_DIR,
split=split,
on_complete=lambda: checkpoints.commit(),
- # Otherwise the head is built at the 4B hidden size whatever the preset.
+ # Use the preset's hidden size when building the head.
config=config,
method=method,
)
@@ -273,9 +227,6 @@ def evaluate(
split: str = "test",
limit_per_source: int | None = None,
) -> dict:
- """Score the newest checkpoint on a held-out split, with and without the
- fitted temperature.
- """
from lev.calibrate import CalibrationProfile
from lev.train.checkpoints import CALIBRATION
from lev.train.config import PRESETS
@@ -316,14 +267,6 @@ def evaluate(
@modal.concurrent(max_inputs=SERVE_CONCURRENCY)
@modal.asgi_app()
def serve():
- """Serve `/v1/systemone`, wire-compatible with the TypeSafe API.
-
- levbench eval --backend lev --base-url \
- --tasks data/eval
-
- With no trained checkpoint for the preset it serves the base backbone
- uncalibrated, with a warning, rather than refusing to start.
- """
from lev.server import create_app
from lev.train.checkpoints import ADAPTER_WEIGHTS, latest_checkpoint
from lev.train.config import PRESETS
@@ -335,13 +278,12 @@ def serve():
# Off unless asked: measured slower than eager on this model (ADR-023).
compile = os.environ.get("LEV_SERVE_COMPILE", "0") in ("1", "true", "yes")
- # Default None: Mode A up to the tokenizer limit (ADR-025). Set to force a
- # lower cap, e.g. 26 to reproduce the ADR-020 policy.
+ # Default to the tokenizer limit; allow an explicit lower label cap.
cap_env = os.environ.get("LEV_SERVE_MAX_LABEL_OPTIONS")
max_label_options = int(cap_env) if cap_env else None
prompt_style = _prompt_style(os.environ.get("LEV_SERVE_PROMPT")) or config.prompt_style
skip_codes = os.environ.get("LEV_SERVE_SKIP_CODES", "1") not in ("0", "false", "no")
- # A frozen model is a different backbone from the adapter's, so no checkpoint.
+ # A frozen-model override must bypass the adapter checkpoint.
frozen = os.environ.get("LEV_SERVE_MODEL")
if frozen:
print(f"serving {frozen} frozen: no adapter, binary Noul, raw softmax")
@@ -379,11 +321,6 @@ def main(preset: str = "4b", dry_run: bool = True):
@app.function(volumes={CKPT_DIR: checkpoints}, secrets=SECRETS, timeout=30 * 60)
def export_checkpoint(preset: str = "4b", name: str | None = None) -> dict:
- """Package the newest checkpoint of a preset into `/checkpoints/releases/`.
-
- No GPU: it copies files (adapter, head, tokenizer, calibration, manifest)
- for `make weights` to pull and `lev release publish` to upload.
- """
from lev.release import build_release
from lev.train.config import PRESETS
@@ -406,11 +343,6 @@ def export_checkpoint(preset: str = "4b", name: str | None = None) -> dict:
@app.function(gpu="H100", volumes=VOLUMES, secrets=SECRETS, timeout=30 * 60)
def check_release(name: str = "4b") -> dict:
- """Load a packaged release the way a user will -- `lev.load` on the flat
- directory, then the HTTP server over it -- and answer one request of every
- question type through each. Raises if the release loads without its
- calibration or head, or if the two paths disagree.
- """
import json
from fastapi.testclient import TestClient
@@ -468,14 +400,7 @@ def diagnose_ordinal(
scales: str = "0,0.25,0.5,0.75,1.0",
subsets: str = "summeval-consistency,summeval-relevance,helpsteer2",
) -> dict:
- """Score the rating subsets at several adapter scales, keeping the adapter unmerged.
-
- summeval-consistency is 0.271 against the frozen backbone's 0.826, so the
- regression may be recoverable by scaling the adapter rather than retraining
- (WiSE-FT, arXiv 2109.01903, which for a LoRA is just the adapter scale).
- Scale 0 must reproduce the frozen backbone; that is the run's own control.
- Also reports the argmax histogram, which levbench aggregates away.
- """
+ """Sweep adapter scales on rating tasks; use scale zero as the frozen-backbone control."""
import collections
import json
@@ -520,14 +445,11 @@ def diagnose_ordinal(
@app.function(gpu="H100", volumes=VOLUMES, secrets=SECRETS, timeout=30 * 60)
def diagnose_candidates(model_id: str = "Qwen/Qwen3.5-4B-Base") -> dict:
- """Compare candidate representations alone, batched, reversed and left-padded.
-
- Tests whether Mode B's padded batch leaks across rows. Result: bit-identical (FINDINGS §12).
- """
+ """Compare candidate embeddings alone, batched, reversed, and left-padded."""
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
- tok = AutoTokenizer.from_pretrained(model_id, cache_dir=MODELS_DIR)
+ tok: Any = AutoTokenizer.from_pretrained(model_id, cache_dir=MODELS_DIR)
# `from_pretrained` is typed as returning the class, not an instance.
loaded: Any = AutoModelForCausalLM.from_pretrained(
model_id, dtype=torch.bfloat16, cache_dir=MODELS_DIR
@@ -618,10 +540,6 @@ def gap(a, b):
def _probabilities(answer: Answer) -> dict[Any, float]:
- """An answer's probability map; a Noul without one stands in its single probability.
-
- Choice keys are strings and Score keys ints, so compare a map only with the same answer's.
- """
from lev.types import NoulAnswer # noqa: PLC0415
if isinstance(answer, NoulAnswer):
@@ -631,10 +549,7 @@ def _probabilities(answer: Answer) -> dict[Any, float]:
@app.function(gpu="H100", volumes=VOLUMES, secrets=SECRETS, timeout=30 * 60)
def profile_engine(preset: str = "4b", rounds: int = 20) -> dict:
- """Time each engine stage of a `/v1/systemone` call in the container, CUDA-synchronised.
-
- Both prefix modes and the compiled path; medians over `rounds` after warmup. Excludes network.
- """
+ """Time CUDA-synchronized engine stages after warmup, excluding network time."""
import statistics
import time
@@ -724,8 +639,7 @@ def wrapper(*a, **k):
worst = max(worst, *(abs(pa[k] - pb[k]) for k in pa))
report["fork_vs_single_max_prob_diff"] = round(worst, 5)
print(f"fork vs single: max |dp| over all answers = {worst:.5f}", flush=True)
- # Fork wins when the forward is FLOP-bound, single when launch-bound; then the
- # compiled single path, with its warmup reported separately.
+ # Compare forked, single, and compiled forwards; report warmup separately.
variants: list[tuple[Literal["fork", "single"], bool]] = [
("fork", False),
("single", False),
@@ -795,13 +709,10 @@ def wrapper(*a, **k):
@app.function(secrets=SECRETS, timeout=10 * 60)
def env_info() -> dict:
- """What the image actually runs: torch, its CUDA build, and which kernels import."""
import platform
import torch
- # Plain strings: `torch.__version__` is a TorchVersion, which the local
- # `modal run` process cannot unpickle without torch installed.
info = {
"python": platform.python_version(),
"torch": str(torch.__version__),
diff --git a/packages/lev/src/lev/__init__.py b/packages/lev/src/lev/__init__.py
index 89836fd..4973c8b 100644
--- a/packages/lev/src/lev/__init__.py
+++ b/packages/lev/src/lev/__init__.py
@@ -1,8 +1,3 @@
-"""Typed, calibrated decisions without token generation.
-
-The core imports without Torch; loading a model requires the train extra.
-"""
-
from .calibrate import CalibrationProfile
from .model import DecisionEngine, load
from .prompt import Layout
diff --git a/packages/lev/src/lev/batcher.py b/packages/lev/src/lev/batcher.py
index 0c92080..5c2c207 100644
--- a/packages/lev/src/lev/batcher.py
+++ b/packages/lev/src/lev/batcher.py
@@ -13,21 +13,26 @@
from collections.abc import Awaitable, Callable
from concurrent.futures import Future
from dataclasses import dataclass
+from typing import Protocol
-from .model import MAX_BATCH_TOKENS, DecisionEngine, Prepared
+from .model import MAX_BATCH_TOKENS, Prepared
from .types import SystemOneResponse
+class AnswerEngine(Protocol):
+ def answer(self, requests: list[Prepared]) -> list[SystemOneResponse]: ...
+
+
class Overloaded(Exception):
- """`max_pending` requests are already in flight."""
+ pass
class ClientGone(Exception):
- """The client disconnected before its answer was ready."""
+ pass
class WorkerStopped(RuntimeError):
- """The GPU worker thread has exited; the server needs a restart."""
+ pass
@dataclass
@@ -40,25 +45,27 @@ class _Job:
class Batcher:
def __init__(
self,
- engine: DecisionEngine,
+ engine: AnswerEngine,
max_pending: int = 64,
max_batch_tokens: int = MAX_BATCH_TOKENS,
timeout: float = 30.0,
):
self.engine = engine
self.max_pending = max_pending
- # Padded tokens per forward (rows x widest row); bounds activation memory.
- # A request larger than the budget still runs, alone.
- self.max_batch_tokens = max_batch_tokens
+ self.max_batch_tokens = max_batch_tokens # Bounds activation memory per forward pass.
self.timeout = timeout
- # A request counts until the worker finishes or skips it, not until its
- # client leaves: abandoned work still occupies the GPU.
- self._pending = 0
+
+ self._pending = 0 # Counts work until the worker finishes, even if the client leaves.
self._pending_lock = threading.Lock()
self._queue: queue.SimpleQueue[_Job] = queue.SimpleQueue()
self._held: _Job | None = None
self._batch: list[_Job] = []
- self._worker = threading.Thread(target=self._run, name="lev-gpu", daemon=True)
+
+ self._worker = threading.Thread(
+ target=self._run,
+ name="lev-gpu",
+ daemon=True,
+ )
self._worker.start()
@property
@@ -71,38 +78,47 @@ def alive(self) -> bool:
return self._worker.is_alive()
async def submit(
- self, prepared: Prepared, disconnected: Callable[[], Awaitable] | None = None
+ self,
+ prepared: Prepared,
+ disconnected: Callable[[], Awaitable] | None = None,
) -> SystemOneResponse:
- """Queue one request and wait for its answer. `disconnected` makes an
- awaitable that completes if the client leaves; it is called only once
- the request is admitted. If it raises instead, disconnects go undetected
- and the answer is awaited alone."""
+
if not self.alive:
raise WorkerStopped("the GPU worker has stopped; restart the server")
+
with self._pending_lock:
if self._pending >= self.max_pending:
raise Overloaded(f"{self._pending} requests in flight; retry shortly")
self._pending += 1
+
job = _Job(prepared, Future(), time.monotonic() + self.timeout)
job.future.add_done_callback(self._release)
self._queue.put(job)
- # Cancelling this wrapper cancels `job.future` unless the worker has
- # already started it.
- result = asyncio.wrap_future(job.future)
+
+ result = asyncio.wrap_future(job.future) # Cancelling may cancel the queued job.
+
if disconnected is None:
return await result
+
watcher = asyncio.ensure_future(disconnected())
+
try:
- await asyncio.wait({result, watcher}, return_when=asyncio.FIRST_COMPLETED)
+ await asyncio.wait(
+ {result, watcher},
+ return_when=asyncio.FIRST_COMPLETED,
+ )
+
if not result.done() and watcher.exception() is not None:
return await result
finally:
if watcher.done() and not watcher.cancelled():
- watcher.exception() # retrieved, so asyncio does not log it as lost
+ watcher.exception() # Mark the exception as retrieved.
watcher.cancel()
+
if not result.done():
result.cancel()
raise ClientGone
+
return result.result()
def _release(self, _future: Future) -> None:
@@ -113,10 +129,11 @@ def _run(self) -> None:
try:
while True:
self._batch = []
+
try:
self._fill(self._batch)
self._compute(self._batch)
- except Exception as error: # noqa: BLE001 -- the worker must outlive any one batch
+ except Exception as error: # noqa: BLE001 -- worker must outlive one batch
traceback.print_exc()
_fail([job.future for job in self._batch], error)
finally:
@@ -125,34 +142,40 @@ def _run(self) -> None:
def _compute(self, batch: list[_Job]) -> None:
try:
responses = self.engine.answer([job.prepared for job in batch])
- except Exception as error: # noqa: BLE001 -- a failed forward fails requests, not the worker
+ except Exception as error: # noqa: BLE001 -- failed forward should not kill worker
if len(batch) == 1:
batch[0].future.set_exception(error)
return
- # A batch can fail where its requests alone would not (out of
- # memory), and one bad request must not fail the others.
+
+ # A batch may fail where its requests alone would not.
for job in batch:
self._compute([job])
return
+
for job, response in zip(batch, responses, strict=True):
job.future.set_result(response)
def _fill(self, batch: list[_Job]) -> None:
- # Marked running only on joining, so a request held for the next batch
- # can still be cancelled or time out.
+ # Held jobs remain cancellable until they actually join a batch.
rows = width = 0
+
while (job := self._take(block=not batch)) is not None:
- grown_rows, grown_width = rows + job.prepared.rows, max(width, job.prepared.width)
+ grown_rows = rows + job.prepared.rows
+ grown_width = max(width, job.prepared.width)
+
if batch and grown_rows * grown_width > self.max_batch_tokens:
self._held = job
return
+
if not job.future.set_running_or_notify_cancel():
- continue # the client left between `_take` and here
+ continue # Client disconnected while queued.
+
batch.append(job)
rows, width = grown_rows, grown_width
def _take(self, block: bool) -> _Job | None:
- """The next request still worth computing, not yet marked running."""
+ """Return the next request worth computing."""
+
while True:
if self._held is not None:
job, self._held = self._held, None
@@ -161,23 +184,27 @@ def _take(self, block: bool) -> _Job | None:
job = self._queue.get(block=block)
except queue.Empty:
return None
+
if job.future.cancelled():
- continue # the client left while it waited
+ continue
+
if time.monotonic() > job.deadline:
if job.future.set_running_or_notify_cancel():
job.future.set_exception(
TimeoutError(f"queued longer than the {self.timeout:.0f}s deadline")
)
continue
+
return job
def _fail_queued(self) -> None:
- """On worker exit, fail the running batch and everything still waiting,
- rather than leave them hanging."""
+
stopped = WorkerStopped("the GPU worker has stopped; restart the server")
_fail([job.future for job in self._batch], stopped)
+
while (job := self._held or self._next_queued()) is not None:
self._held = None
+
if job.future.set_running_or_notify_cancel():
job.future.set_exception(stopped)
diff --git a/packages/lev/src/lev/calibrate.py b/packages/lev/src/lev/calibrate.py
index 9bf42aa..3d8074a 100644
--- a/packages/lev/src/lev/calibrate.py
+++ b/packages/lev/src/lev/calibrate.py
@@ -1,11 +1,11 @@
-"""Temperature scaling per question type, readout mode and Choice option band (ADR-028)."""
+"""Temperature scaling by question type, readout mode, and Choice option band (ADR-028)."""
from __future__ import annotations
import json
import math
from collections import Counter
-from collections.abc import Sequence
+from collections.abc import Mapping, Sequence
from dataclasses import dataclass, field
from pathlib import Path
@@ -15,28 +15,34 @@
def softmax(logits: Logits, temperature: float = 1.0) -> list[float]:
if temperature <= 0:
raise ValueError(f"temperature must be positive, got {temperature}")
+
scaled = [x / temperature for x in logits]
top = max(scaled)
exps = [math.exp(x - top) for x in scaled]
total = sum(exps)
+
return [e / total for e in exps]
-def nll(
+def negative_log_likelihood(
samples: Sequence[tuple[Logits, int]],
temperature: float,
weights: Sequence[float] | None = None,
) -> float:
- """Mean (optionally weighted) negative log-likelihood of the true class."""
+
if not samples:
return 0.0
+
total = 0.0
weight_sum = 0.0
+
for i, (logits, truth) in enumerate(samples):
- w = 1.0 if weights is None else weights[i]
- p = softmax(logits, temperature)[truth]
- total -= w * math.log(max(p, 1e-15))
- weight_sum += w
+ weight = 1.0 if weights is None else weights[i]
+ probability = softmax(logits, temperature)[truth]
+
+ total -= weight * math.log(max(probability, 1e-15))
+ weight_sum += weight
+
return total / weight_sum
@@ -47,22 +53,29 @@ def fit_temperature(
tol: float = 1e-4,
weights: Sequence[float] | None = None,
) -> float:
- """Minimise NLL over temperature by ternary search (NLL is unimodal in it).
- Returns 1.0 when no temperature beats the identity, e.g. for all-equal logits.
- """
if not samples:
return 1.0
+
while hi - lo > tol:
m1 = lo + (hi - lo) / 3
m2 = hi - (hi - lo) / 3
- if nll(samples, m1, weights) < nll(samples, m2, weights):
+
+ if negative_log_likelihood(samples, m1, weights) < negative_log_likelihood(
+ samples, m2, weights
+ ):
hi = m2
else:
lo = m1
+
fitted = (lo + hi) / 2
- if nll(samples, fitted, weights) >= nll(samples, 1.0, weights) - 1e-12:
+
+ if (
+ negative_log_likelihood(samples, fitted, weights)
+ >= negative_log_likelihood(samples, 1.0, weights) - 1e-12
+ ):
return 1.0
+
return fitted
@@ -70,131 +83,182 @@ def fit_temperature(
def family_weights(families: Sequence[str]) -> list[float]:
- """Each family's rows share a total weight of 1."""
+ """Give each family equal total weight."""
+
counts = Counter(families)
- return [1.0 / counts[f] for f in families]
+ return [1.0 / counts[family] for family in families]
def _fit(samples, families, method: str) -> float:
return fit_temperature(
- samples, weights=family_weights(families) if method == "family" else None
+ samples,
+ weights=family_weights(families) if method == "family" else None,
)
def leave_one_family_out_ece(
- samples: Sequence[tuple[Logits, int]], families: Sequence[str], method: str
+ samples: Sequence[tuple[Logits, int]],
+ families: Sequence[str],
+ method: str,
) -> float:
- """Mean over families of the ECE on that family, at the temperature `method`
- fits on every other family. Families weigh equally, as a new task would."""
+ """Measure transfer ECE by holding out each family once."""
+
eces = []
+
for held_out in sorted(set(families)):
- train = [(s, f) for s, f in zip(samples, families, strict=True) if f != held_out]
- test = [s for s, f in zip(samples, families, strict=True) if f == held_out]
- t = _fit([s for s, _ in train], [f for _, f in train], method)
- probs = [softmax(logits, t) for logits, _ in test]
- eces.append(expected_calibration_error(probs, [y for _, y in test]))
+ train = [
+ (sample, family)
+ for sample, family in zip(samples, families, strict=True)
+ if family != held_out
+ ]
+ test = [
+ sample for sample, family in zip(samples, families, strict=True) if family == held_out
+ ]
+
+ temperature = _fit(
+ [sample for sample, _ in train],
+ [family for _, family in train],
+ method,
+ )
+ probs = [softmax(logits, temperature) for logits, _ in test]
+ eces.append(expected_calibration_error(probs, [truth for _, truth in test]))
+
return sum(eces) / len(eces)
def fit_for_transfer(
- buckets: dict[str, Sequence[tuple[Logits, int, str]]],
+ buckets: Mapping[str, Sequence[tuple[Logits, int, str]]],
split_name: str,
min_samples: int = 50,
) -> tuple[CalibrationProfile, dict[str, dict]]:
- """Per bucket, keep whichever of the row and family fits transfers better.
+ """Choose the row- or family-weighted fit that transfers better per bucket."""
- Buckets from fewer than `MIN_FAMILIES_FOR_TRANSFER` families keep the row fit.
- Returns the profile and a per-bucket report of both fits and the choice.
- """
_refuse_held_out(split_name)
+
profile = CalibrationProfile(fitted_on=f"{split_name} (transfer-selected)")
report: dict[str, dict] = {}
+
for bucket, rows in buckets.items():
if len(rows) < min_samples:
continue
- samples = [(logits, y) for logits, y, _ in rows]
- families = [f for _, _, f in rows]
- entry: dict = {"n": len(rows), "families": len(set(families))}
+
+ samples = [(logits, truth) for logits, truth, _ in rows]
+ families = [family for _, _, family in rows]
+
+ entry: dict = {
+ "n": len(rows),
+ "families": len(set(families)),
+ }
entry["t_rows"] = _fit(samples, families, "rows")
+
if entry["families"] >= MIN_FAMILIES_FOR_TRANSFER:
entry["t_family"] = _fit(samples, families, "family")
entry["lofo_ece_rows"] = leave_one_family_out_ece(samples, families, "rows")
- entry["lofo_ece_family"] = leave_one_family_out_ece(samples, families, "family")
+ entry["lofo_ece_family"] = leave_one_family_out_ece(
+ samples,
+ families,
+ "family",
+ )
entry["chosen"] = (
"family" if entry["lofo_ece_family"] < entry["lofo_ece_rows"] else "rows"
)
else:
entry["chosen"] = "rows"
+
profile.temperatures[bucket] = entry[f"t_{entry['chosen']}"]
profile.n_samples[bucket] = len(rows)
report[bucket] = entry
+
return profile, report
def expected_calibration_error(
- probs: Sequence[Sequence[float]], truths: Sequence[int], n_bins: int = 10
+ probs: Sequence[Sequence[float]],
+ truths: Sequence[int],
+ n_bins: int = 10,
) -> float:
- """Sample-weighted mean absolute gap between top probability and accuracy."""
+
if not probs:
return 0.0
bins: list[list[tuple[float, bool]]] = [[] for _ in range(n_bins)]
+
for distribution, truth in zip(probs, truths, strict=True):
confidence = max(distribution)
predicted = max(range(len(distribution)), key=distribution.__getitem__)
- # The top bin is closed: a confidence of exactly 1.0 would otherwise
- # index one past the end.
- bin_index = min(int(confidence * n_bins), n_bins - 1)
+
+ bin_index = min(int(confidence * n_bins), n_bins - 1) # Keep 1.0 in the top bin.
bins[bin_index].append((confidence, predicted == truth))
total_weighted_gap = 0.0
+
for members in bins:
if not members:
continue
- mean_confidence = sum(c for c, _ in members) / len(members)
+
+ mean_confidence = sum(confidence for confidence, _ in members) / len(members)
accuracy = sum(hit for _, hit in members) / len(members)
+
total_weighted_gap += len(members) * abs(mean_confidence - accuracy)
+
return total_weighted_gap / len(probs)
-# Softmax spreads mass differently as option count grows, so Choice fits use bands.
+# Choice calibration is banded because option count changes softmax behaviour.
CHOICE_BANDS = ((8, "small"), (26, "mid"))
def option_band(n_options: int | None) -> str | None:
if n_options is None:
return None
+
for upper, name in CHOICE_BANDS:
if n_options <= upper:
return name
+
return "large"
@dataclass
class CalibrationProfile:
- """Fitted temperatures keyed by `"{question_type}:{mode}"`, and for Choice
- by `"choice:{mode}:{band}"` as well."""
-
temperatures: dict[str, float] = field(default_factory=dict)
fitted_on: str = ""
n_samples: dict[str, int] = field(default_factory=dict)
@staticmethod
- def key(question_type: str, mode: str, n_options: int | None = None) -> str:
+ def key(
+ question_type: str,
+ mode: str,
+ n_options: int | None = None,
+ ) -> str:
band = option_band(n_options) if question_type == "choice" else None
return f"{question_type}:{mode}" + (f":{band}" if band else "")
- def temperature(self, question_type: str, mode: str, n_options: int | None = None) -> float:
- # Banded, then unbanded (profiles fitted before bands), then 1.0: an
- # unfitted bucket gets raw softmax rather than a borrowed scalar.
+ def temperature(
+ self,
+ question_type: str,
+ mode: str,
+ n_options: int | None = None,
+ ) -> float:
banded = self.key(question_type, mode, n_options)
plain = self.key(question_type, mode)
- return self.temperatures.get(banded, self.temperatures.get(plain, 1.0))
+
+ return self.temperatures.get(
+ banded,
+ self.temperatures.get(plain, 1.0),
+ )
def apply(
- self, logits: Logits, question_type: str, mode: str, n_options: int | None = None
+ self,
+ logits: Logits,
+ question_type: str,
+ mode: str,
+ n_options: int | None = None,
) -> list[float]:
- return softmax(logits, self.temperature(question_type, mode, n_options))
+ return softmax(
+ logits,
+ self.temperature(question_type, mode, n_options),
+ )
def save(self, path: str | Path) -> None:
Path(path).write_text(
@@ -211,6 +275,7 @@ def save(self, path: str | Path) -> None:
@classmethod
def load(cls, path: str | Path) -> CalibrationProfile:
payload = json.loads(Path(path).read_text())
+
return cls(
temperatures=payload["temperatures"],
fitted_on=payload.get("fitted_on", ""),
@@ -219,19 +284,22 @@ def load(cls, path: str | Path) -> CalibrationProfile:
def fit(
- buckets: dict[str, Sequence[tuple[Logits, int]]],
+ buckets: Mapping[str, Sequence[tuple[Logits, int]]],
split_name: str,
min_samples: int = 50,
) -> CalibrationProfile:
- """Fit one temperature per bucket; reject test, eval and holdout splits."""
+ """Fit one temperature per sufficiently large bucket."""
+
_refuse_held_out(split_name)
profile = CalibrationProfile(fitted_on=split_name)
+
for bucket, samples in buckets.items():
if len(samples) < min_samples:
- # Leave it unfitted (identity) rather than fit a scalar on noise.
continue
+
profile.temperatures[bucket] = fit_temperature(samples)
profile.n_samples[bucket] = len(samples)
+
return profile
diff --git a/packages/lev/src/lev/cli.py b/packages/lev/src/lev/cli.py
index 9e78631..0cd616f 100644
--- a/packages/lev/src/lev/cli.py
+++ b/packages/lev/src/lev/cli.py
@@ -10,7 +10,6 @@
def _measure_tokens(config, data_dir: str, sample: int = 1500) -> dict:
- """Length distribution of real rendered prompts; the budget rests on it (ADR-016)."""
import statistics
from transformers import AutoTokenizer
@@ -21,8 +20,9 @@ def _measure_tokens(config, data_dir: str, sample: int = 1500) -> dict:
from .train.collate import RouteCache, render
tokenizer = AutoTokenizer.from_pretrained(config.model_id)
+ if tokenizer is None:
+ raise RuntimeError(f"could not load tokenizer for {config.model_id!r}")
rows = load_split(data_dir, Split.TRAIN)[:sample]
- # The preset's own prompt style: `chat` adds a system turn to every prompt.
style = Style(config.prompt_style)
routes = RouteCache(tokenizer, config.max_label_options, style)
lengths = sorted(
@@ -92,8 +92,6 @@ def cmd_check_data(args: argparse.Namespace) -> None:
def _preset_names() -> list[str]:
- """Lazy, like every CLI import, so `lev plan` and `lev check-data` run
- without the train extra."""
from .train.config import PRESETS
return sorted(PRESETS)
@@ -224,7 +222,6 @@ def cmd_serve(args: argparse.Namespace) -> None:
def _cuda():
- """`torch.cuda` when a GPU is in use, else None; torch is imported only here."""
try:
import torch
except ImportError:
@@ -233,13 +230,14 @@ def _cuda():
def _snapshot_revision(checkpoint) -> str | None:
- """The Hub commit a checkpoint was loaded from: the `snapshots/` directory name."""
parts = Path(checkpoint).parts if checkpoint else ()
return parts[parts.index("snapshots") + 1] if "snapshots" in parts[:-1] else None
def _weights_dtype(engine) -> str | None:
model = getattr(engine, "model", None)
+ if model is None:
+ return None
try:
return str(next(model.parameters()).dtype)
except (AttributeError, StopIteration, TypeError):
@@ -247,9 +245,7 @@ def _weights_dtype(engine) -> str | None:
def cmd_presentation_checks(args: argparse.Namespace) -> None:
- """Label-free presentation checks (ADR-029). With `--compare`, exits 2 when the run's
- settings differ from the report's, and 1 when a metric drifts beyond `--tolerance`;
- `--enforce-gate` also fails a check below its gate."""
+ """Run presentation checks; exit 2 for settings mismatches and 1 for failed checks."""
import time
from . import presentation
@@ -331,7 +327,6 @@ def cmd_presentation_checks(args: argparse.Namespace) -> None:
def _refuse_other_settings(saved: dict | None, settings: dict, keys) -> None:
- """Metrics move with the order mode, the batching and the weights; compare like with like."""
from .presentation import settings_mismatch
differ = settings_mismatch(saved, settings, keys) if saved is not None else []
@@ -359,7 +354,7 @@ def main(argv: list[str] | None = None) -> None:
route_parser = sub.add_parser("route", help="show the readout mode per question")
route_parser.add_argument("request", help="path to a /v1/systemone request JSON")
route_parser.add_argument("--model", default="Qwen/Qwen3.5-4B-Base")
- # Defaults mirror `EngineConfig`, so the preview matches what a server does.
+ # Keep route-preview defaults aligned with EngineConfig.
route_parser.add_argument(
"--max-label-options",
type=int,
diff --git a/packages/lev/src/lev/data/build.py b/packages/lev/src/lev/data/build.py
index 0503770..2612786 100644
--- a/packages/lev/src/lev/data/build.py
+++ b/packages/lev/src/lev/data/build.py
@@ -1,7 +1,4 @@
-"""Write train, calibration and test JSONL files and a manifest for offline training.
-
-Source rows are split before sampling or augmentation, so no row leaks across splits (ADR-014).
-"""
+"""Write offline training splits and a manifest; split before augmentation (ADR-014)."""
from __future__ import annotations
@@ -77,10 +74,7 @@ def read_jsonl(path: Path) -> list[Example]:
def verify_round_trip(path: Path, sample: int = 1000) -> int:
- """Check sampled rows of a written split read back exactly; return the number checked.
-
- Order-sensitive, since `target` indexes option order (ADR-024). Raises `ValueError` on drift.
- """
+ """Check sampled rows round-trip exactly, including option order; return the count."""
rows = read_jsonl(path)
step = max(1, len(rows) // sample)
with path.open(encoding="utf-8") as handle:
@@ -89,7 +83,7 @@ def verify_round_trip(path: Path, sample: int = 1000) -> int:
continue
written = json.loads(line)
reread = to_json(rows[i])
- # As text: dict equality ignores key order, which a merged question loses.
+ # Compare serialized text because dict equality ignores option order.
if json.dumps(reread, ensure_ascii=False) != json.dumps(written, ensure_ascii=False):
raise ValueError(
f"{path} row {i} did not survive the round trip: written "
@@ -156,8 +150,7 @@ def build_dataset(
# A different stream per split, so layouts do not repeat in lockstep.
seed=seed + split_index,
adjacent=adjacent,
- # Test keeps canonical questions (one per eval file); calibration varies
- # option sets, not wording, so each option-count band has rows (ADR-026).
+ # Keep test questions canonical; vary calibration option sets to cover count bands.
augment=augment if split is not Split.TEST else {},
**held_out_overrides,
)
@@ -171,7 +164,6 @@ def build_dataset(
unique_train = sum(len(rows) for rows in by_source[Split.TRAIN].values())
manifest = {
"sources": {name: REGISTRY[name].hf_id for name in weights},
- # Sampling with replacement can expose each unique row more than once per epoch.
"unique_train_rows": unique_train,
"oversample_ratio": round(n_examples / max(1, unique_train), 2),
"weights": weights,
diff --git a/packages/lev/src/lev/data/contamination.py b/packages/lev/src/lev/data/contamination.py
index 83a2466..f49236a 100644
--- a/packages/lev/src/lev/data/contamination.py
+++ b/packages/lev/src/lev/data/contamination.py
@@ -1,7 +1,4 @@
-"""Reject training sources that resolve to any of the 13 S1Bench subsets.
-
-`assert_eval_only` is the inverse gate, for the S1Bench evaluation loader.
-"""
+"""Block S1Bench training sources and restrict evaluation to S1Bench subsets."""
from __future__ import annotations
@@ -27,8 +24,7 @@
}
)
-# Aliases and parent datasets that resolve to a blocked subset. A mixture that lists
-# `google-research-datasets/paws` is contaminated even though the string differs.
+# Block aliases and parent datasets, not just exact subset names.
_ALIASES: dict[str, str] = {
"vitaminc": "vitaminc-dev",
"tals/vitaminc": "vitaminc-dev",
@@ -38,8 +34,7 @@
"mteb/amazon_massive_scenario": "massive-en-US",
"setfit/amazon_massive_intent_en-us": "massive-en-US",
"setfit/amazon_massive_scenario_en-us": "massive-en-US",
- # Different rows, same 64-intent schema MASSIVE inherited via SLURP: it would
- # make massive-en-US a seen taxonomy (ADR-020).
+ # SLURP shares MASSIVE's intent taxonomy (ADR-020).
"hwu64": "massive-en-US",
"deeppavlov/hwu64": "massive-en-US",
"super_glue/boolq": "boolq",
@@ -71,7 +66,7 @@
class ContaminationError(RuntimeError):
- """Raised when a training mixture touches an evaluation subset."""
+ pass
def normalise(name: str) -> str:
@@ -83,8 +78,7 @@ def normalise(name: str) -> str:
_BLOCKED_BY_NORMALISED: dict[str, str] = {normalise(s): s for s in BLOCKED_SUBSETS}
_ALIASES_BY_NORMALISED: dict[str, str] = {normalise(a): t for a, t in _ALIASES.items()}
-# Bare aliases match as segment sets, so an unlisted re-host (`someorg/squad_v2_dedup`)
-# is caught (ADR-020); org-qualified aliases stay exact-match only.
+# Match bare aliases by segments to catch re-hosts; qualified aliases stay exact.
_ALIAS_SEGMENTS: tuple[tuple[frozenset[str], str], ...] = tuple(
(frozenset(alias.split("-")), target)
for alias, target in _ALIASES_BY_NORMALISED.items()
@@ -107,8 +101,7 @@ def resolve(name: str) -> str | None:
for normalised_subset, subset in _BLOCKED_BY_NORMALISED.items():
if normalised_subset in segments:
return subset
- # Likewise for a bare alias: every segment of `squad-v2` present, in any order.
- # Conservative on purpose: a false block costs one dataset, a miss the result.
+ # Conservative segment matching prevents renamed evaluation data from entering training.
for alias_segments, subset in _ALIAS_SEGMENTS:
if alias_segments <= segments:
return subset
@@ -116,7 +109,7 @@ def resolve(name: str) -> str | None:
def check_mixture(dataset_names: Iterable[str]) -> dict[str, str]:
- """Return `{offending name: blocked subset}` for everything that collides."""
+ """Return the blocked subset for each offending source."""
hits: dict[str, str] = {}
for name in dataset_names:
if (subset := resolve(name)) is not None:
@@ -125,7 +118,7 @@ def check_mixture(dataset_names: Iterable[str]) -> dict[str, str]:
def assert_eval_only(dataset_name: str) -> str:
- """Return the blocked subset `dataset_name` resolves to; raise `ContaminationError` if none."""
+ """Resolve an evaluation subset or raise ContaminationError."""
subset = resolve(dataset_name)
if subset is None:
raise ContaminationError(
@@ -138,7 +131,7 @@ def assert_eval_only(dataset_name: str) -> str:
def assert_clean(dataset_names: Iterable[str]) -> None:
- """Raise `ContaminationError` if any dataset resolves to a blocked evaluation subset."""
+ """Reject sources that resolve to blocked evaluation subsets."""
if hits := check_mixture(dataset_names):
listed = "\n".join(f" {src!r} -> blocked subset {dst!r}" for src, dst in hits.items())
raise ContaminationError(
diff --git a/packages/lev/src/lev/data/export_eval.py b/packages/lev/src/lev/data/export_eval.py
index e824107..a8fc3af 100644
--- a/packages/lev/src/lev/data/export_eval.py
+++ b/packages/lev/src/lev/data/export_eval.py
@@ -1,7 +1,4 @@
-"""Write the held-out test split as levbench task files, one per source.
-
-levbench does not import `lev` (ADR-010). A combined file would ask every question of every item.
-"""
+"""Export held-out rows as levbench task files, one question per source."""
from __future__ import annotations
@@ -13,13 +10,12 @@
from .build import _group_by_source, load_split
from .splits import Split
-# Below this an accuracy is not a measurement: at n=24 the 95% interval is
-# about +/-16 points.
+# Require enough rows for a useful accuracy estimate.
MIN_USEFUL_ITEMS = 100
def truth_for(question: Question, target: int) -> bool | int | str:
- """The label levbench compares: a Choice option string, a Score level index, a Noul bool."""
+ """Convert a target to a Choice key, Score index, or Noul bool."""
if isinstance(question, Noul):
# The target is a rating index; collapse it as the readout does.
return noul_probability({target: 1.0}) >= 0.5
@@ -48,8 +44,7 @@ def export(
if limit_per_source:
rows = rows[:limit_per_source]
question = rows[0].question
- # A task file has one question for all items, so QA sources with per-row
- # options (race, sciq, ...) are skipped; `evaluate_split` still scores them.
+ # Skip per-row option sets: each task file has one shared question.
payloads = {json.dumps(question_payload(r.question), sort_keys=True) for r in rows}
if len(payloads) != 1:
variable.append(source)
diff --git a/packages/lev/src/lev/data/mixture.py b/packages/lev/src/lev/data/mixture.py
index cebc1ba..a985fda 100644
--- a/packages/lev/src/lev/data/mixture.py
+++ b/packages/lev/src/lev/data/mixture.py
@@ -1,12 +1,9 @@
-"""Sample typed training examples with layout, abstention and question variation.
-
-Loaders are injected so the mixture can be built and tested offline.
-"""
+"""Sample typed examples with layout, abstention, and question variation."""
from __future__ import annotations
import random
-from collections.abc import Callable, Iterable, Iterator
+from collections.abc import Callable, Iterable, Iterator, Mapping, Sequence
from dataclasses import dataclass, field
from ..labels import NOUL_RATING_TOKENS
@@ -25,23 +22,18 @@ class Example:
layout: Layout
source: str
abstain: bool = False
- # Abstain examples only: no gold index, so the loss reads this uniform target
- # instead of `target`.
+ # Abstain rows use a uniform soft target instead of a gold index.
soft_target: list[float] | None = None
@dataclass(frozen=True)
class Augment:
- """Per-source variation: train varies wording and options, calibration options only (ADR-026).
-
- `negations` flip the Noul target so "yes" is not always "good"; `min_options` floors subsets.
- """
+ """Per-source wording and option variation; negations flip Noul targets."""
paraphrases: tuple[str, ...] = ()
negations: tuple[str, ...] = ()
min_options: int = 2
- # Fraction of Choice rows exempt from subsampling; they may still be shuffled.
- # For a large taxonomy this is the candidate-path head's training data.
+ # Keep full option sets on this share of rows to train the candidate head.
keep_full_fraction: float = 0.0
@@ -52,8 +44,7 @@ class MixtureSpec:
schema_first_fraction: float = 0.5
abstain_fraction: float = 0.1
seed: int = 17
- # source -> sources whose states must not be used as abstain donors,
- # because they would in fact answer the question. See `sources.ADJACENT`.
+ # Exclude sources that could still answer the question from abstain donors.
adjacent: dict[str, frozenset[str]] = field(default_factory=dict)
# Variation per source; empty means canonical questions (ADR-020).
augment: dict[str, Augment] = field(default_factory=dict)
@@ -64,7 +55,6 @@ class MixtureSpec:
description_dropout: float = 0.3
def validate(self) -> None:
- # Before any example loads: a contaminated run silently looks better.
assert_clean(self.sources.keys())
if not self.sources:
raise ValueError("mixture has no sources")
@@ -76,12 +66,8 @@ def validate(self) -> None:
Loader = Callable[[], Iterable[Example]]
-def build_mixture(spec: MixtureSpec, loaders: dict[str, Loader]) -> Iterator[Example]:
- """Yield `spec.n_examples`, respecting weights, layout split and abstain rate.
-
- Raises `ContaminationError` for a blocked source, `KeyError` for one with no loader, and
- `ValueError` for no sources, weights not summing to 1, or a source that yields nothing.
- """
+def build_mixture(spec: MixtureSpec, loaders: Mapping[str, Loader]) -> Iterator[Example]:
+ """Sample examples from validated sources using the configured weights and augmentations."""
spec.validate()
missing = set(spec.sources) - set(loaders)
if missing:
@@ -107,16 +93,14 @@ def build_mixture(spec: MixtureSpec, loaders: dict[str, Loader]) -> Iterator[Exa
state = example.state
question, target = example.question, example.target
if (augment := spec.augment.get(source)) is not None:
- # Negation flips the target, so it must not touch an abstain row,
- # whose supervision is uniform regardless of polarity.
+ # Abstain supervision is uniform, so do not flip its target.
question, target = _vary(
question, target, augment, spec, rng, allow_negation=not abstain
)
soft_target = None
if abstain:
- # Swap in an unrelated state and supervise a uniform distribution:
- # with no evidence that is the calibrated answer (ADR-012).
+ # Pair unrelated evidence with uniform supervision (ADR-012).
state = _donor_state(pools, source, spec.adjacent, rng)
n_candidates = candidate_count(question)
soft_target = [1.0 / n_candidates] * n_candidates
@@ -142,8 +126,7 @@ def _vary(
rng: random.Random,
allow_negation: bool,
) -> tuple[Question, int]:
- # Preserves the gold answer: negation mirrors a Noul rating, subsampling keeps and
- # re-indexes the gold option, and Score levels are never reordered.
+ # Preserve gold targets through negation and option subsets; never reorder Score levels.
instructions = question.instructions
can_negate = isinstance(question, Noul) and augment.negations and allow_negation
if can_negate and rng.random() < spec.negate_fraction:
@@ -177,7 +160,7 @@ def _vary(
return Choice(instructions=instructions, criteria=dict(options)), _index_of(options, gold[0])
-def _index_of(options: list[tuple[str, object]], key: str) -> int:
+def _index_of(options: Sequence[tuple[str, object]], key: str) -> int:
for i, (k, _) in enumerate(options):
if k == key:
return i
@@ -190,8 +173,7 @@ def _donor_state(
adjacent: dict[str, frozenset[str]],
rng: random.Random,
) -> str | dict | list:
- # Another source is not enough: a rotten_tomatoes state answers imdb's question, so
- # `adjacent` sources are excluded too; the fallbacks let a narrow mixture still yield.
+ # Exclude adjacent sources when possible; fall back for narrow mixtures.
excluded = {source} | set(adjacent.get(source, ()))
eligible = (
[name for name in pools if name not in excluded]
diff --git a/packages/lev/src/lev/data/s1bench.py b/packages/lev/src/lev/data/s1bench.py
index d97e00b..e81ee79 100644
--- a/packages/lev/src/lev/data/s1bench.py
+++ b/packages/lev/src/lev/data/s1bench.py
@@ -1,9 +1,4 @@
-"""Export the 13 S1Bench subsets as levbench task files, for evaluation only.
-
-Must not import training data builders or produce an `Example` (checked by `test_s1bench.py`).
-Ids, questions and labels come from Nimble's pinned manifests in `s1bench_subsets/`;
-conversion details and the multinli metadata exception are in docs/FINDINGS.md §17.
-"""
+"""Export S1Bench evaluation data from pinned manifests; keep it outside training (FINDINGS §17)."""
from __future__ import annotations
@@ -17,19 +12,19 @@
from pathlib import Path
from typing import TYPE_CHECKING, cast
-if TYPE_CHECKING: # `datasets` is heavy and only imported where it is used.
+if TYPE_CHECKING:
from datasets import Dataset
from ..types import Choice, Noul, Question, Score, question_payload
from .contamination import assert_eval_only
SUBSET_DIR = Path(__file__).with_name("s1bench_subsets")
-QUESTION_NAME = "decision" # S1Bench asks every record one question under this name
+QUESTION_NAME = "decision"
@dataclass(frozen=True)
class EvalItem:
- """One S1Bench record; `truth` is a Choice option string, Score level index or Noul bool."""
+ """An evaluation record with a Choice key, Score index, or Noul bool as truth."""
id: str
state: dict | str
@@ -38,11 +33,7 @@ class EvalItem:
@dataclass(frozen=True)
class EvalSubset:
- """One S1Bench subset and Jev's score on it.
-
- `jev_published` is Jev 1.13's figure; `jev_measured` is the board's `s1-fast` run in
- `data/s1bench-snapshot.json`, which covered only six subsets.
- """
+ """A subset with published Jev 1.13 and measured s1-fast scores."""
name: str
hf_id: str
@@ -100,14 +91,13 @@ def _download(repo: str, filename: str, revision: str | None = None) -> str:
def _parquet(repo: str, filename: str, revision: str | None = None) -> Dataset:
- from datasets import load_dataset # heavy, and only needed to export
+ from datasets import load_dataset
- # `split=` narrows load_dataset's return union to the one Dataset the annotation names.
return load_dataset("parquet", data_files=_download(repo, filename, revision), split="train")
def _rows(dataset: Dataset) -> Iterator[dict]:
- # Upstream leaves `Dataset.__iter__` unannotated, so checkers reject column lookups.
+ # Upstream row iteration lacks type annotations.
return cast("Iterator[dict]", iter(dataset))
@@ -264,11 +254,7 @@ def _read_pubmedqa() -> Iterator[EvalItem]:
def load_eval_subset(name: str) -> tuple[Question, list[EvalItem]]:
- """One S1Bench subset as its question plus exactly its records, sorted by id.
-
- Raises `ContaminationError` for anything that is not S1Bench evaluation data,
- and `ValueError` if the upstream data no longer yields the pinned ids and labels.
- """
+ """Load a S1Bench subset, validating its records against pinned ids and labels."""
subset = get_subset(name)
spec = definition(subset.name)
by_id = {item.id: item for item in _READERS[subset.name]()}
diff --git a/packages/lev/src/lev/data/sources.py b/packages/lev/src/lev/data/sources.py
index 4a56415..25b0606 100644
--- a/packages/lev/src/lev/data/sources.py
+++ b/packages/lev/src/lev/data/sources.py
@@ -1,7 +1,4 @@
-"""Training dataset registry and readers for Choice, Score and Noul examples.
-
-Tests use fake loaders, so they do not verify that upstream datasets remain available.
-"""
+"""Training dataset registry and readers for Choice, Score, and Noul examples."""
from __future__ import annotations
@@ -17,8 +14,7 @@
from .contamination import assert_clean
from .mixture import Augment, Example
-# Large taxonomies whose label names are read at load time, so the registry
-# cannot count them.
+# Taxonomies with label counts determined at load time.
_LARGE_OPTION_SETS = frozenset({"banking77", "clinc_oos"})
Primitive = Literal["choice", "score", "noul"]
@@ -27,11 +23,13 @@
MODE_B_SHARE = 0.25
-# What a per-row reader returns: the state to show, the option texts if the row
-# carries its own (None means the source's fixed label set), and the gold index.
+# Reader output: state, per-row options (None uses fixed labels), and gold index.
Row = tuple[str | dict, list[str] | None, int]
+ChoiceRow = tuple[str | dict, list[str], int]
+PairRow = tuple[dict, None, int]
+Rows = list[Row] | list[ChoiceRow] | list[PairRow]
# Readers may yield several examples from one row, or None to skip it.
-Reader = Callable[[dict, random.Random], Row | list[Row] | None]
+Reader = Callable[[dict, random.Random], Row | Rows | None]
@dataclass(frozen=True)
@@ -61,7 +59,7 @@ def is_mode_b(self) -> bool:
return bool(self.label_names) and len(self.label_names) > LABEL_OPTION_CAP
-def _read_race(row: dict, rng: random.Random) -> Row | None:
+def _read_race(row: dict, rng: random.Random) -> ChoiceRow | None:
options = [str(o) for o in row["options"]]
index = ord(str(row["answer"]).strip().upper()[:1] or "?") - ord("A")
if len(options) < 2 or not 0 <= index < len(options):
@@ -69,13 +67,12 @@ def _read_race(row: dict, rng: random.Random) -> Row | None:
return {"passage": row["article"], "question": row["question"]}, options, index
-def _read_keyed_choices(question_field: str, context_field: str | None = None) -> Reader:
- """Read the `choices={"label", "text"}` + `answerKey` shape of commonsense_qa, openbookqa, ARC.
-
- ARC keys some rows 1-4 rather than A-D, so the key is matched against the labels, not A-Z.
- """
+def _read_keyed_choices(
+ question_field: str, context_field: str | None = None
+) -> Callable[[dict, random.Random], ChoiceRow | None]:
+ """Match answerKey to choice labels, including numeric ARC labels."""
- def reader(row: dict, rng: random.Random) -> Row | None:
+ def reader(row: dict, rng: random.Random) -> ChoiceRow | None:
labels = [str(x) for x in row["choices"]["label"]]
texts = [str(x) for x in row["choices"]["text"]]
key = str(row["answerKey"]).strip()
@@ -92,7 +89,7 @@ def reader(row: dict, rng: random.Random) -> Row | None:
return reader
-def _read_sciq(row: dict, rng: random.Random) -> Row | None:
+def _read_sciq(row: dict, rng: random.Random) -> ChoiceRow | None:
gold = row["correct_answer"]
options = [gold, row["distractor1"], row["distractor2"], row["distractor3"]]
options = [o for o in options if o and str(o).strip()]
@@ -122,10 +119,7 @@ def _read_nli(row: dict, rng: random.Random) -> Row | None:
def swap_two_words(text: str, rng: random.Random) -> str | None:
- """Exchange two distinct content words: full lexical overlap, changed meaning.
-
- None when there are fewer than two candidates or 8 draws find no distinct pair.
- """
+ """Swap distinct content words, or return None if no pair is found in eight draws."""
words = text.split()
slots = [i for i, w in enumerate(words) if len(w) > 3 and w.isalpha()]
if len(slots) < 2:
@@ -139,17 +133,15 @@ def swap_two_words(text: str, rng: random.Random) -> str | None:
return None
-def _read_pair(first: str, second: str, adversarial: float = 0.0) -> Reader:
- """Read labelled sentence pairs, optionally adding word-swapped negatives.
+def _read_pair(
+ first: str, second: str, adversarial: float = 0.0
+) -> Callable[[dict, random.Random], list[PairRow]]:
+ """Read sentence pairs, optionally adding word-swapped negatives to positive rows."""
- `adversarial` is the fraction of positive pairs that also yield a negative,
- reducing the usefulness of lexical overlap as a shortcut.
- """
-
- def reader(row: dict, rng: random.Random) -> list[Row] | None:
+ def reader(row: dict, rng: random.Random) -> list[PairRow]:
a, b = row[first], row[second]
label = int(row["label"])
- out: list[Row] = [({"sentence1": a, "sentence2": b}, None, label)]
+ out: list[PairRow] = [({"sentence1": a, "sentence2": b}, None, label)]
adversary = label == 1 and adversarial and rng.random() < adversarial
if adversary and (swapped := swap_two_words(b, rng)) is not None:
out.append(({"sentence1": a, "sentence2": swapped}, None, 0))
@@ -167,15 +159,14 @@ def reader(row: dict, rng: random.Random) -> list[Row] | None:
def _read_nli_fever(row: dict, rng: random.Random) -> Row | None:
- # `premise` is the claim and `hypothesis` the evidence; the integer label's
- # order differs from FEVER's own, so the string label is used.
+ # Use FEVER's string label; its integer order differs.
label = row.get("fever_gold_label")
if label not in FEVER_LABELS:
return None
return {"claim": row["premise"], "evidence": row["hypothesis"]}, None, FEVER_LABELS.index(label)
-def _read_parade(row: dict, rng: random.Random) -> Row | None:
+def _read_parade(row: dict, rng: random.Random) -> PairRow:
return (
{"sentence1": row["Definition1"], "sentence2": row["Definition2"]},
None,
@@ -183,19 +174,20 @@ def _read_parade(row: dict, rng: random.Random) -> Row | None:
)
-def _read_strategyqa(row: dict, rng: random.Random) -> Row | None:
+def _read_strategyqa(row: dict, rng: random.Random) -> PairRow:
return {"question": row["question"], "facts": row["facts"]}, None, int(bool(row["answer"]))
-def yes_no_from_choices(reader: Reader) -> Reader:
+def yes_no_from_choices(reader: Reader) -> Callable[[dict, random.Random], list[PairRow] | None]:
"""Turn each multiple-choice row into two Noul rows: gold and wrong proposals."""
- def wrapped(row: dict, rng: random.Random) -> list[Row] | None:
+ def wrapped(row: dict, rng: random.Random) -> list[PairRow] | None:
produced = reader(row, rng)
if produced is None:
return None
- out: list[Row] = []
- for state, options, target in produced if isinstance(produced, list) else [produced]:
+ out: list[PairRow] = []
+ rows = produced if isinstance(produced, list) else [produced]
+ for state, options, target in rows:
if not options or len(options) < 2:
continue
base = state if isinstance(state, dict) else {"question": state}
@@ -207,7 +199,7 @@ def wrapped(row: dict, rng: random.Random) -> list[Row] | None:
return wrapped
-def _read_toxic_chat(row: dict, rng: random.Random) -> Row | None:
+def _read_toxic_chat(row: dict, rng: random.Random) -> Row:
return row["user_input"], None, int(row["toxicity"])
@@ -219,9 +211,7 @@ def _read_toxigen(row: dict, rng: random.Random) -> Row | None:
return row["text"], None, int(float(score) >= 3.0)
-def _read_beavertails(row: dict, rng: random.Random) -> Row | None:
- # `is_safe` True -> yes; the registry's negation keeps polarity from being
- # constant.
+def _read_beavertails(row: dict, rng: random.Random) -> PairRow:
return {"prompt": row["prompt"], "response": row["response"]}, None, int(bool(row["is_safe"]))
@@ -234,8 +224,8 @@ def _read_beavertails(row: dict, rng: random.Random) -> Row | None:
)
-def _read_ultrafeedback(row: dict, rng: random.Random) -> list[Row] | None:
- out: list[Row] = []
+def _read_ultrafeedback(row: dict, rng: random.Random) -> list[PairRow] | None:
+ out: list[PairRow] = []
for completion in row.get("completions") or []:
rating = ((completion.get("annotations") or {}).get("helpfulness") or {}).get("Rating")
if rating is None:
@@ -273,7 +263,6 @@ def _read_snips(row: dict, rng: random.Random) -> Row | None:
REGISTRY: dict[str, SourceSpec] = {
- # Choice: small, well-separated option sets.
"ag_news": SourceSpec(
name="ag_news",
hf_id="fancyzhx/ag_news",
@@ -307,8 +296,7 @@ def _read_snips(row: dict, rng: random.Random) -> Row | None:
"What type of entity is being described?",
),
),
- # Choice with the question varying per row: the state alone cannot identify
- # the answer set, so the model has to read the question.
+ # Per-row questions require reading the options as well as the state.
"race": SourceSpec(
name="race",
hf_id="ehovy/race",
@@ -355,8 +343,6 @@ def _read_snips(row: dict, rng: random.Random) -> Row | None:
paraphrases=("Choose the correct answer.", "What is the right answer to this question?"),
reader=_read_keyed_choices("question"),
),
- # Natural language inference: the same three options every time, but the
- # answer depends on a relation between two fields, not on either alone.
"snli": SourceSpec(
name="snli",
hf_id="stanfordnlp/snli",
@@ -406,7 +392,7 @@ def _read_snips(row: dict, rng: random.Random) -> Row | None:
label_names=SNIPS_INTENTS,
reader=_read_snips,
),
- # Large taxonomies: their full sets train the Mode B head.
+ # Full taxonomies train the Mode B head.
"banking77": SourceSpec(
name="banking77",
hf_id="legacy-datasets/banking77",
@@ -467,8 +453,7 @@ def _read_snips(row: dict, rng: random.Random) -> Row | None:
label_names=HELPFULNESS_LEVELS,
reader=_read_ultrafeedback,
),
- # Noul: a single yes/no proposition. Every source carries a negation, so the
- # mixture asks both polarities and "yes" is not always the good outcome.
+ # Each Noul source includes a negation to vary answer polarity.
"imdb": SourceSpec(
name="imdb",
hf_id="stanfordnlp/imdb",
@@ -485,7 +470,6 @@ def _read_snips(row: dict, rng: random.Random) -> Row | None:
paraphrases=("Did the critic like it?", "Is the sentiment favourable?"),
negations=("Is this review negative?", "Did the critic pan it?"),
),
- # Paraphrase, with word-swapped negatives so overlap is not the answer.
"mrpc": SourceSpec(
name="mrpc",
hf_id="SetFit/mrpc",
@@ -516,7 +500,6 @@ def _read_snips(row: dict, rng: random.Random) -> Row | None:
label_names=YES_NO,
reader=_read_parade,
),
- # Yes/no on a factual axis: a proposed answer to a grounded question.
"strategyqa": SourceSpec(
name="strategyqa",
hf_id="ChilleD/StrategyQA",
@@ -594,7 +577,7 @@ def _read_snips(row: dict, rng: random.Random) -> Row | None:
),
}
-# Checked once, at import. A contaminated id must never reach a loader.
+# Reject contaminated source ids before loading.
assert_clean(REGISTRY.keys())
assert_clean([spec.hf_id for spec in REGISTRY.values()])
@@ -602,8 +585,7 @@ def _read_snips(row: dict, rng: random.Random) -> Row | None:
name for name, spec in REGISTRY.items() if spec.is_mode_b
)
-# Sources whose states can answer each other's questions, so none donates an abstain
-# state to another (ADR-012); e.g. clinc_oos contains banking intents.
+# Adjacent sources may answer each other's questions; exclude them as abstain donors.
ADJACENT: tuple[frozenset[str], ...] = (
frozenset({"imdb", "rotten_tomatoes", "sst5", "yelp_review_full", "emotion"}),
frozenset({"banking77", "clinc_oos", "snips"}),
@@ -628,11 +610,7 @@ def _read_snips(row: dict, rng: random.Random) -> Row | None:
def family_of(source: str) -> str:
- """The task family a source belongs to: its `ADJACENT` group, else itself.
-
- Holding out mrpc while qqp stays in is not a new task, so calibration
- transfer is measured over families, not sources (ADR-028).
- """
+ """Return the adjacent-source family used for calibration transfer (ADR-028)."""
for group in ADJACENT:
if source in group:
return "+".join(sorted(group))
@@ -640,7 +618,7 @@ def family_of(source: str) -> str:
def adjacency_map() -> dict[str, frozenset[str]]:
- """source -> the sources whose states must not be used to make it unanswerable."""
+ """Map sources to those excluded as abstain donors."""
return {
name: frozenset().union(*(group for group in ADJACENT if name in group)) - {name}
for name in REGISTRY
@@ -679,7 +657,6 @@ def build_question(spec: SourceSpec, names: list[str]) -> Question:
return Noul(instructions=spec.instructions)
options = [humanise(n) for n in names]
if spec.primitive == "score":
- # `list` is invariant, so the element type has to be the one Score declares.
levels: list[JSONContent] = [*options]
return Score(instructions=spec.instructions, criteria=levels)
return Choice(
@@ -696,13 +673,9 @@ def load_source(
seed: int = 17,
load_dataset: Callable | None = None,
) -> Iterator[Example]:
- """Yield examples, shuffling before applying a row limit.
-
- A head slice can contain only one class in a label-sorted corpus.
- `load_dataset` is injectable so tests run offline.
- """
+ """Yield examples through an injectable loader, shuffling before applying the limit."""
if load_dataset is None:
- from datasets import load_dataset # noqa: PLC0415
+ from datasets import load_dataset
extra = {"revision": spec.revision} if spec.revision else {}
dataset = load_dataset(
@@ -711,8 +684,7 @@ def load_source(
if limit is not None and limit < len(dataset):
dataset = dataset.shuffle(seed=seed).select(range(limit))
- # A fixed label set comes from the ClassLabel or the spec; a reader that
- # supplies options per row has none.
+ # Fixed labels come from ClassLabel or the spec; per-row readers supply their own.
names = _label_names(spec, dataset) if spec.reader is None or spec.label_names else None
fixed_question = build_question(spec, names) if names is not None else None
is_noul = spec.primitive == "noul"
@@ -755,8 +727,7 @@ def emit(state, question: Question, target: int, n_labels: int, i: int) -> Examp
raise ValueError(f"{spec.name}: reader gave no options and spec has no labels")
yield emit(state, fixed_question, target, len(names), i)
else:
- # Duplicate options would collapse in the criteria map and shift
- # the gold index.
+ # Duplicate options collapse map keys and corrupt the gold index.
if len(set(options)) != len(options):
continue
question = Choice(
@@ -776,12 +747,7 @@ def _blank(state) -> bool:
def augmentation_map() -> dict[str, Augment]:
- """Per-source augmentation for the train and calibration mixtures, read off the registry.
-
- Large taxonomies keep their full option set half the time (the Mode B head's
- data) and otherwise may be cut to `LARGE_SET_MIN_OPTIONS` or more, so Mode A
- also learns large lettered sets. Everything else may be cut to two (ADR-026).
- """
+ """Configure option variation; large taxonomies retain full sets half the time (ADR-026)."""
return {
name: Augment(
paraphrases=spec.paraphrases,
@@ -794,11 +760,7 @@ def augmentation_map() -> dict[str, Augment]:
def default_weights() -> dict[str, float]:
- """Sampling weight per source: `MODE_B_SHARE` to Mode B, the rest by primitive.
-
- The remainder splits evenly across the three primitives, then within each,
- so no readout is starved by having few sources.
- """
+ """Reserve MODE_B_SHARE for large taxonomies; balance the remainder across primitives."""
mode_b_sources = sorted(MODE_B_SOURCES)
sources_by_primitive: dict[str, list[str]] = {}
for name, spec in REGISTRY.items():
diff --git a/packages/lev/src/lev/data/splits.py b/packages/lev/src/lev/data/splits.py
index ce5fdf6..2d09c2a 100644
--- a/packages/lev/src/lev/data/splits.py
+++ b/packages/lev/src/lev/data/splits.py
@@ -1,7 +1,4 @@
-"""Deterministic, disjoint train, calibration and test splits (ADR-014).
-
-The key includes row position: the same input order reproduces a split, reordering changes it.
-"""
+"""Deterministic splits keyed by source and row position; input order matters (ADR-014)."""
from __future__ import annotations
@@ -24,8 +21,6 @@ class Split(StrEnum):
# Changing this re-splits every row, making models before and after incomparable.
SPLIT_SALT = "lev-split-v1"
-# A temperature needs far fewer rows than a model, and test only has to separate
-# two runs.
DEFAULT_FRACTIONS: dict[Split, float] = {
Split.TRAIN: 0.80,
Split.CALIBRATION: 0.10,
@@ -38,7 +33,7 @@ def row_key(source: str, index: int, text: str) -> str:
def hash_position(key: str, salt: str = SPLIT_SALT) -> float:
- """Map a key to a uniform float in [0, 1), stable across processes, unlike salted `hash()`."""
+ """Map a key to a process-stable float in [0, 1)."""
digest = hashlib.blake2b(f"{salt}|{key}".encode(), digest_size=8).digest()
return int.from_bytes(digest, "big") / float(1 << 64)
@@ -89,19 +84,14 @@ def split_examples(
def check_coverage(splits: dict[Split, list[Example]], strict: bool = True) -> SplitReport:
- """Run two label-coverage checks (ADR-014); with `strict`, raise `ValueError` on either.
-
- Every observed label must appear in train, and every label the question offers must be
- observed at all, which catches a truncated or label-sorted corpus the first check passes.
- """
+ """Check observed labels appear in train and offered labels appear in the data."""
seen: dict[str, set[int]] = defaultdict(set)
in_train: dict[str, set[int]] = defaultdict(set)
offered: dict[str, int] = {}
for split, items in splits.items():
for example in items:
seen[example.source].add(example.target)
- # A Noul offers nine rating levels but its data supplies only the two
- # ends, so "every offered label must be observed" does not apply.
+ # Binary Noul data only covers the rating scale's endpoints.
if example.question.type != "noul":
offered.setdefault(example.source, candidate_count(example.question))
if split is Split.TRAIN:
diff --git a/packages/lev/src/lev/labels.py b/packages/lev/src/lev/labels.py
index e1b4ee0..8439649 100644
--- a/packages/lev/src/lev/labels.py
+++ b/packages/lev/src/lev/labels.py
@@ -4,17 +4,18 @@
from itertools import islice, product
from string import ascii_uppercase
+from typing import Protocol
-# Above this the source registry trains a label set's full taxonomy on the Mode B
-# head; routing follows the tokenizer, not this cap (ADR-025/026).
+
+class TokenEncoder(Protocol):
+ def encode(self, text: str, *, add_special_tokens: bool = True) -> list[int]: ...
+
+
+# Training taxonomy cap; serving routes by tokenizer support (ADR-025/026).
LABEL_OPTION_CAP = len(ascii_uppercase)
def label_codes(n: int) -> list[str]:
- """First `n` codes: A..Z, then AA..ZZ, then AAA..ZZZ.
-
- Raises for n > 18,278 (past ZZZ).
- """
if n < 1:
raise ValueError("need at least one code")
codes = list(islice(_all_codes(), n))
@@ -26,19 +27,13 @@ def label_codes(n: int) -> list[str]:
def single_token_codes(
tokenizer, n: int, prefix: str = " ", skip_multi_token: bool = False
) -> list[str] | None:
- """Return `n` single-token codes, or None to request Mode B.
-
- The prefix must match the answer boundary: a space in plain prompts,
- no space in chat prompts. By default, all first `n` codes must pass.
- Serving may skip split codes; training retains contiguous codes (ADR-028).
- """
+ """Find n single-token codes using the prompt boundary prefix; optionally skip split codes."""
if not skip_multi_token:
try:
codes = label_codes(n)
except ValueError:
return None
- verified = [c for c in codes if _is_single_token(tokenizer, prefix + c)]
- return verified[:n] if len(verified) >= n else None
+ return codes if all(_is_single_token(tokenizer, prefix + c) for c in codes) else None
picked: list[str] = []
for code in _all_codes():
@@ -50,7 +45,6 @@ def single_token_codes(
def _all_codes():
- """A..Z, AA..ZZ, AAA..ZZZ, lazily -- the same order as `label_codes`."""
yield from ascii_uppercase
for width in (2, 3):
for t in product(ascii_uppercase, repeat=width):
@@ -61,8 +55,7 @@ def _is_single_token(tokenizer, text: str) -> bool:
return len(tokenizer.encode(text, add_special_tokens=False)) == 1
-# A rating scale, not yes/no, so a Noul answer carries a calibratable
-# distribution (docs/ARCHITECTURE.md §3.4).
+# Rating probabilities allow Noul calibration (ARCHITECTURE §3.4).
NOUL_RATING_TOKENS = [str(i) for i in range(9)]
diff --git a/packages/lev/src/lev/model.py b/packages/lev/src/lev/model.py
index b0a2c09..00a7eaa 100644
--- a/packages/lev/src/lev/model.py
+++ b/packages/lev/src/lev/model.py
@@ -33,8 +33,7 @@
_QUESTIONS = TypeAdapter(dict[str, Question])
-# Padded tokens (rows x widest row) per forward: `lev.batcher`'s budget, and the most a
-# request Score averaging grew may need (ADR-029, ADR-030).
+# Padded-token budget shared by batching and Score-order averaging.
MAX_BATCH_TOKENS = 16384
@@ -50,19 +49,15 @@ class EngineConfig:
noul_readout: Literal["rating", "binary"] = "rating"
# A second option order reduces letter-position bias; costs one extra batch row.
order_average: bool = True
- # Score levels keep their low-to-high order unless this opts in (ADR-029): "reversed"
- # adds the reversed order (2 rows per Score, as Choice), "cyclic" every rotation (K rows).
- # Both show the model level orders training never produces (ADR-020, FINDINGS §12).
+ # Score averaging adds reversed (2 rows) or cyclic (K rows) orders absent from training.
score_order_average: Literal["off", "reversed", "cyclic"] = "off"
- # A request whose Score averaging added rows must fit this many padded tokens; `prepare`
- # refuses it otherwise (HTTP 422). Requests without added Score rows are never refused.
+ # Reject over-budget requests only when Score averaging added rows.
max_batch_tokens: int | None = MAX_BATCH_TOKENS
# Serving can skip split codes; training keeps large sets for Mode B (ADR-028).
skip_multi_token_codes: bool = True
# Forking avoids repeated prefix work but adds a forward and cache copies (ADR-023).
prefix_mode: Literal["fork", "single"] = "single"
- # Off because it measured slower on this backbone (ADR-023). When enabled,
- # inputs are padded to shape buckets and warmup captures graphs at startup.
+ # Opt-in compilation uses shape buckets; measured slower than eager (ADR-023).
compile: bool = False
pad_to: int = 32
@@ -108,11 +103,7 @@ def want_hidden(self) -> bool:
def average_orders(prob_rows: list[list[float]], orders: list[list[int] | None]) -> list[float]:
- """Map each rendered-order distribution back to canonical order and average.
-
- `probs[j]` belongs to canonical slot `order[j]`; averaging probabilities, not
- logits, keeps the result a distribution.
- """
+ """Restore canonical candidate order and average probabilities, not logits."""
if len(prob_rows) != len(orders):
raise ValueError("one order per probability row")
k = len(prob_rows[0])
@@ -138,8 +129,7 @@ def serving_routes(
class DecisionEngine:
- """Answers typed questions about a state in one forward pass; `answer` puts
- several prepared requests in the same forward."""
+ """Answer typed questions, batching prepared requests into shared forwards."""
def __init__(
self,
@@ -157,17 +147,15 @@ def __init__(
self.checkpoint: Path | None = None
self._candidate_cache: dict[tuple[str, ...], Any] = {}
self._device = getattr(model, "device", None)
- # One forward at a time: CUDA-graph replay is not thread-safe. The server
- # batches concurrent requests into one forward instead (`lev.batcher`).
+ # Serialize forwards because CUDA-graph replay is not thread-safe.
self._lock = threading.Lock()
- # A fast tokenizer's padding and truncation are shared state: concurrent
- # calls raise "Already borrowed" or encode with another call's truncation.
+ # Serialize tokenizer access to protect shared padding and truncation settings.
self._tokenizer_lock = threading.RLock()
- # The decoder runs alone and `lm_head` projects only the scored
- # positions: full (rows, width, vocab) logits are GBs at a 248k vocabulary.
base = model.get_base_model() if hasattr(model, "get_base_model") else model
- self._decoder = base.get_decoder() if model is not None else None
- self._lm_head = base.get_output_embeddings() if model is not None else None
+ # Held apart so the head runs on scored positions only (`_project`), instead of
+ # allocating logits for the whole sequence. `Any`: the wrappers are dynamically typed.
+ self._decoder: Any = base.get_decoder() if model is not None else None
+ self._lm_head: Any = base.get_output_embeddings() if model is not None else None
self._eager_decoder = self._decoder
if self.config.compile:
import torch
@@ -185,10 +173,7 @@ def warmup(
(2, 512),
),
) -> float:
- """Run one forward per shape bucket so compile and graph capture happen
- before the first request. Returns seconds spent. Falls back to eager,
- with a warning, if the compiled path fails.
- """
+ """Warm compiled shape buckets; return elapsed seconds and fall back to eager on failure."""
import torch
started = time.perf_counter()
@@ -208,21 +193,17 @@ def warmup(
return time.perf_counter() - started
def system_one(self, state, questions: Mapping[str, Question | dict]) -> SystemOneResponse:
- """Answer every question about `state` in one forward pass. Questions may
- be `Noul`/`Choice`/`Score` objects or the same shapes as plain dicts."""
+ """Answer typed or JSON-shaped questions about a state."""
return self.answer([self.prepare(state, questions)])[0]
def prepare(self, state, questions: Mapping[str, Question | dict]) -> Prepared:
- """Validate, route and tokenise one request. Raises `ValueError` or
- `TypeError` for a malformed one, and `RuntimeError` for one that needs
- Mode B with no head loaded, before any GPU work."""
+ """Validate and tokenize a request, rejecting malformed input or a missing Mode B head."""
questions = _QUESTIONS.validate_python(questions)
with self._tokenizer_lock:
return self._route_and_tokenise(state, questions)
def question_rows(self, questions: Mapping[str, Question | dict]) -> dict[str, int]:
- """Batch rows each question adds to a request under this engine's config:
- the orders `prepare` renders it in. Depends on routing, not on the state."""
+ """Count rendered orders per question under the current routing configuration."""
questions = _QUESTIONS.validate_python(questions)
with self._tokenizer_lock:
routes = serving_routes(questions, self.tokenizer, self.config)
@@ -249,8 +230,7 @@ def _route_and_tokenise(self, state, questions: dict[str, Question]) -> Prepared
1 for v in variants if v.order is not None and isinstance(questions[v.name], Score)
)
limit = self.config.max_batch_tokens
- # The batcher runs an oversized request alone and unsplit; refuse only one that
- # Score averaging grew, so a request main accepts is never refused for its size.
+ # Oversized requests run alone; reject only growth from Score averaging.
if limit is not None and added and prepared.rows * prepared.width > limit:
raise ValueError(
f"this request needs {prepared.rows} rows x {prepared.width} tokens = "
@@ -262,8 +242,7 @@ def _route_and_tokenise(self, state, questions: dict[str, Question]) -> Prepared
return prepared
def answer(self, requests: list[Prepared]) -> list[SystemOneResponse]:
- """Answer prepared requests, in single prefix mode all in one forward.
- Rows are independent, so batching requests changes no row's inputs."""
+ """Answer prepared requests; single-prefix mode batches them into one forward."""
logits, hidden = self._forward(requests)
responses, start = [], 0
for request in requests:
@@ -310,12 +289,7 @@ def _respond(self, request: Prepared, logits, hidden) -> SystemOneResponse:
)
def _orders(self, question: Question, route: Route) -> list[list[int] | None]:
- """Candidate orders to render; None means the original order. Only lettered
- Choice and binary Noul in state-first layout are reversed (ordered scales
- keep training's low-to-high order; schema-first shares options), unless
- `score_order_average` opts a Score in: "reversed" adds one row, "cyclic"
- K-1, both in orders training never shows (ADR-029). `order_average=False`
- reads every question once."""
+ """Choose candidate orders; keep scales canonical unless Score averaging is enabled."""
if (
not self.config.order_average
or route.mode is not Mode.LABEL_TOKEN
@@ -361,8 +335,7 @@ def _render(
for name, question in questions.items()
for order in self._orders(question, routes[name])
]
- # Every variant shares one prefix by construction; a mismatch would make
- # the cache wrong, not just slow.
+ # Variants must share a prefix for cache reuse to be valid.
prefixes = {r.prefix for _, _, r in rendered}
if len(prefixes) != 1:
raise AssertionError(f"layout produced {len(prefixes)} prefixes, expected 1")
@@ -372,8 +345,7 @@ def _render(
]
def _forward(self, requests: list[Prepared]):
- """Logits `(N, V)` and, for Mode B, hidden states `(N, H)` at every
- variant's last token, rows in request order."""
+ """Return last-token logits (N, V) and optional hidden states (N, H) in request order."""
import torch
want_hidden = any(r.want_hidden for r in requests)
@@ -386,13 +358,13 @@ def _forward(self, requests: list[Prepared]):
for r in requests
]
logits = torch.cat([logits for logits, _ in outs])
- hidden = torch.cat([hidden for _, hidden in outs]) if want_hidden else None
+ hidden = (
+ torch.cat([hidden for _, hidden in outs if hidden is not None]) if want_hidden else None
+ )
return logits, hidden
def _forward_single(self, rows: list[list[int]], want_hidden: bool):
- """One right-padded batch of prefix+suffix rows. Under `compile`, rows pad to
- a power of two and width to `pad_to`, so few graphs are recorded; padding
- changes no real row, since right padding is masked and rows are independent."""
+ """Run right-padded prefix/suffix rows, using shape buckets when compiled."""
import torch
device = self._device
@@ -465,6 +437,7 @@ def _label_token_scores(self, logits, route: Route) -> list[float]:
self.tokenizer, prefix=label_prefix(Style(self.config.prompt_style))
)
with self._tokenizer_lock:
+ assert route.codes is not None, "Mode A requires label codes"
candidate_ids = readout.candidate_ids(route.codes)
return logits[candidate_ids].float().tolist()
@@ -486,9 +459,7 @@ def _candidate_path_scores(self, hidden, question: Question) -> list[float]:
return scores[0].float().tolist()
def _candidate_reprs(self, texts: list[str]):
- """One hidden vector per candidate string, cached per candidate set. The cache
- is unbounded: fine for fixed schemas (151 candidates ~1.5 MB), not for
- arbitrary caller-supplied option sets."""
+ """Cache candidate embeddings without a size bound; intended for fixed schemas."""
import torch
key = tuple(texts)
@@ -522,7 +493,7 @@ def _to_answer(self, question: Question, route: Route, probs: list[float]):
if isinstance(question, Choice):
distribution = dict(zip(question.criteria, probs, strict=True))
return ChoiceAnswer(
- choice=max(distribution, key=distribution.get),
+ choice=max(distribution, key=distribution.__getitem__),
probabilities=distribution,
confidence=confidence,
)
@@ -538,8 +509,7 @@ def _to_answer(self, question: Question, route: Route, probs: list[float]):
if isinstance(question, Noul):
if route.reason == BINARY_NOUL:
- # Two lettered options, yes first; no rating distribution exists,
- # so `probabilities` stays absent.
+ # Binary Noul has no rating distribution.
return NoulAnswer(noul=probs[0], probabilities=None, confidence=confidence)
by_rating = dict(enumerate(probs))
return NoulAnswer(
@@ -566,24 +536,16 @@ def load(
max_batch_tokens: int | None = MAX_BATCH_TOKENS,
merge_adapter: bool = True,
) -> DecisionEngine:
- """Load a checkpoint -- a release directory, a training output or a Hub id --
- into a ready `DecisionEngine`. With no checkpoint, serves `model_id` frozen.
+ """Load an engine from a release, training output, or Hub id; otherwise use the frozen base.
- engine = lev.load("interfaze-ai/lev")
- engine.system_one(state, {"urgent": {"type": "noul", "instructions": "..."}})
-
- A release's `lev_release.json` names the base model and prompt style its
- weights were trained under and overrides `model_id` and `prompt_style`. The
- Mode B head and `calibration.json` beside the adapter are picked up; a
- missing calibration is warned about, not hidden.
- """
+ Release metadata overrides the base model and prompt style. Load the checkpoint's
+ tokenizer, head, and calibration when present; warn if calibration is missing."""
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from .train.checkpoints import CALIBRATION, resolve_checkpoint
- # A checkpoint holds a LoRA adapter, not a full model: load the base from
- # `model_id`, then apply the adapter.
+ # Load the base model before applying the checkpoint's adapter.
resolved = resolve_checkpoint(checkpoint, cache_dir) if checkpoint else None
manifest_file = resolved / "lev_release.json" if resolved else None
if manifest_file and manifest_file.is_file():
@@ -596,8 +558,7 @@ def load(
)
model_id = manifest["base_model"]
- # The checkpoint's own tokenizer when saved: Mode A reads specific token ids,
- # so a mismatched tokenizer gives plausible wrong answers, not an error.
+ # Use the saved tokenizer to preserve Mode A label ids.
tok_source = resolved if resolved and (resolved / "tokenizer.json").is_file() else model_id
tokenizer = AutoTokenizer.from_pretrained(str(tok_source), cache_dir=cache_dir)
@@ -615,18 +576,14 @@ def load(
if resolved:
from peft import PeftModel
- # Merged by default: the forward is launch-bound (ADR-023), so the per-layer
- # lora_A/lora_B matmuls cost launches rather than arithmetic, and graph
- # capture needs the weights fixed. `merge_adapter=False` keeps the adapter
- # separate, which is what lets its scale be swept without retraining.
+ # Merge to reduce launches and fix graph weights; keep separate for adapter-scale sweeps.
model = PeftModel.from_pretrained(model, str(resolved))
if merge_adapter:
model = model.merge_and_unload()
head = _load_head(resolved / MODE_B_HEAD, model)
model = model.eval()
- # Default to the profile sitting beside the weights it was fitted for:
- # inside a flat release directory, or one level up in a training tree.
+ # Use calibration beside release weights or above training checkpoints.
if calibration is None and resolved:
candidates = [resolved / CALIBRATION, resolved.parent / CALIBRATION]
calibration = next((c for c in candidates if c.is_file()), candidates[-1])
@@ -639,8 +596,7 @@ def load(
config = EngineConfig(
model_id=model_id,
- # A stock checkpoint pins the 0-8 rating scale at one end regardless of
- # content (ADR-007), so untrained serving reads Noul as two options.
+ # Use binary Noul for untrained backbones with unreliable rating scales (ADR-007).
noul_readout=noul_readout or ("rating" if resolved else "binary"),
compile=compile,
prompt_style=prompt_style or "plain",
@@ -679,12 +635,6 @@ def _bucket_type(question: Question, route: Route) -> str:
def _fork(cache, n: int, device=None):
- """Copy a batch-1 prefix cache and expand it to `n` rows.
-
- `reorder_cache` supports both full- and linear-attention layers;
- `batch_repeat_interleave` does not. Copy first because reordering mutates
- the cache and the original must stay reusable.
- """
import torch
forked = copy.deepcopy(cache)
@@ -694,7 +644,6 @@ def _fork(cache, n: int, device=None):
def _gini(probs: list[float]) -> float:
- """Normalised Gini concentration: uniform -> 0, point mass -> 1."""
k = len(probs)
if k <= 1:
return 1.0
diff --git a/packages/lev/src/lev/presentation.py b/packages/lev/src/lev/presentation.py
index 835002b..34b9020 100644
--- a/packages/lev/src/lev/presentation.py
+++ b/packages/lev/src/lev/presentation.py
@@ -1,37 +1,15 @@
-"""Presentation checks: does an answer move with where an option is shown?
-
-The design of NandhaKishorM/laya#259, over `system_one` (ADR-029). Per question type
-(Score and Choice), on any list of states, with no labels:
-
-- **identical**: the position bias of one read, before any order averaging. Every
- option carries the same text, so options differ only by position and code. Metric:
- slot 0's log-probability minus the mean over the slots, averaged over states and
- configurations; 0 is no position effect. It is read with `order_average=False`,
- because averaged orders cancel it by construction (cyclic) or hide the middle slots
- (reversed): it measures the bias averaging has to remove, not what is left after it.
- Choice keys must be unique, so they are shapes with no order (`KEYS`), and every
- assignment of keys to slots is read, so a key's own pull cancels.
-- **first_slot**: three real options in all 3! = 6 orders per state, read as the engine
- is configured. An order-free readout picks slot 0 in exactly 1/3 of the decisions; a
- tie gives each of its m slots 1/m. `consistent` is the share of states whose six orders
- all give one answer. In the chat style a Score line names its level by its place in
- the request (`(level i of K)`), so a permuted listing renumbers the levels too.
-- **packed**: the largest probability difference between the probe's answers and each
- question asked alone, so what is measured is the presentation and not the packing.
-
-Probabilities are whatever `system_one` returns: calibrated, unless the engine carries
-an empty `CalibrationProfile`. Within one question, calibrated log-probabilities are the
-raw scores divided by the bucket's temperature, so the identical metric shrinks by 1/T
-and the first-slot rate does not change. The proposed gate, -0.20, is Laya's, set on
-raw scores.
-"""
+"""Label-free checks for option-position bias and packing consistency (ADR-029).
+
+Identical-option probes use one order; real-option probes use configured averaging.
+Scores use returned probabilities: calibration scales the identical metric by 1/T.
+The -0.20 gate comes from Laya's raw-score checks (NandhaKishorM/laya#259)."""
from __future__ import annotations
import itertools
import json
import math
-from collections.abc import Iterator, Mapping, Sequence
+from collections.abc import Generator, Mapping, Sequence
from contextlib import contextmanager
from dataclasses import replace
from typing import Any
@@ -41,7 +19,7 @@
KEYS = ("■", "●", "◆", "▲", "◎")
SLOT0_MIN = -0.20
FIRST_SLOT_MIN = 0.15
-# Batch rows per probe call: a full bench_en run peaked at 22.5 GiB reserved on a 32 GB card.
+# Limit probe rows to bound GPU memory use.
MAX_ROWS = 32
# en follows Laya's presentation_checks.py; ja the same wording in Japanese.
@@ -80,7 +58,7 @@
},
}
-# Settings a report records. The first group is known before the model loads.
+# Check pre-load settings before spending time loading weights.
PRE_LOAD = ("n_states", "lang", "raw", "kinds", "score_order_average", "max_rows")
POST_LOAD = ("checkpoint_revision", "dtype")
SETTINGS = PRE_LOAD + POST_LOAD
@@ -101,8 +79,7 @@ def identical_questions(kind: str, text: str, k: int, instructions: str) -> dict
def probe_questions(lang: str = "en", kinds: Sequence[str] = KINDS) -> dict[str, dict]:
- """Every probe question for one state, keyed `kind|identical|text|k|keys` or
- `kind|perm|012`."""
+ """Build identical-option and permutation probes with structured question ids."""
questions: dict[str, dict] = {}
for kind in kinds:
spec = TEXT[lang][kind]
@@ -127,7 +104,7 @@ def is_identical(qid: str) -> bool:
@contextmanager
-def configured(engine, **changes) -> Iterator[None]:
+def configured(engine, **changes) -> Generator[None, None, None]:
"""Run the block with `engine.config` changed as given, then restore it."""
config = engine.config
engine.config = replace(config, **changes)
@@ -138,13 +115,11 @@ def configured(engine, **changes) -> Iterator[None]:
def single_order(engine):
- """Read every question once, in the order given."""
return configured(engine, order_average=False)
def call_groups(engine, questions: dict[str, dict], max_rows: int) -> list[dict[str, dict]]:
- """`questions` split into `system_one` calls of at most `max_rows` batch rows, as the
- engine counts them under its current config."""
+ """Group questions into calls within the engine's rendered-row budget."""
rows = engine.question_rows(questions)
groups: list[dict[str, dict]] = []
current: dict[str, dict] = {}
@@ -167,8 +142,7 @@ def ask(
kinds: Sequence[str] = KINDS,
max_rows: int | None = None,
) -> list[dict[str, Any]]:
- """Every probe answer for every state: the identical questions in one order, the
- permuted ones as the engine is configured. Calls hold at most `max_rows` batch rows."""
+ """Run single-order identical probes and configured permutation probes within max_rows."""
max_rows = max_rows or MAX_ROWS
questions = probe_questions(lang, kinds)
identical = {q: v for q, v in questions.items() if is_identical(q)}
@@ -182,8 +156,7 @@ def ask(
with single_order(engine):
for group in identical_groups:
got.update(engine.system_one(state, group).answers)
- # The probe sizes its own calls; the serving limit would refuse its averaged
- # calls on long states.
+ # Probes enforce their own row budget instead of the serving token limit.
with configured(engine, max_batch_tokens=None):
for group in permuted_groups:
got.update(engine.system_one(state, group).answers)
@@ -282,8 +255,7 @@ def packed_consistency(
lang: str = "en",
kinds: Sequence[str] = KINDS,
) -> float:
- """Largest |p| difference between `ask`'s answers for `states` and each question asked
- alone, under the same order setting."""
+ """Return the largest probability difference between packed and individual probes."""
questions = probe_questions(lang, kinds)
worst = 0.0
for state, packed in zip(states, answers, strict=False):
@@ -302,8 +274,6 @@ def packed_consistency(
def settings_mismatch(saved: dict, run: dict, keys: Sequence[str] = SETTINGS) -> list[str]:
- """Every setting among `keys` that `saved` recorded and this run does not share, as
- ` (report: X, run: Y)`. A setting the report never recorded is not checked."""
return [
f"{key} (report: {saved[key]!r}, run: {run.get(key)!r})"
for key in keys
@@ -343,6 +313,5 @@ def to_markdown(report: dict) -> str:
def read_states(path: str) -> list[Any]:
- """`state` from each line of a JSONL file (bench_en.jsonl, bench_ja.jsonl or your own)."""
with open(path, encoding="utf-8") as fh:
return [json.loads(line)["state"] for line in fh if line.strip()]
diff --git a/packages/lev/src/lev/prompt.py b/packages/lev/src/lev/prompt.py
index 4b65878..6e03c0e 100644
--- a/packages/lev/src/lev/prompt.py
+++ b/packages/lev/src/lev/prompt.py
@@ -3,6 +3,7 @@
from __future__ import annotations
import json
+from collections.abc import Mapping
from dataclasses import dataclass
from enum import StrEnum
@@ -18,20 +19,14 @@ class Layout(StrEnum):
class Style(StrEnum):
- """Prompt format recorded with the weights and shared by training and serving.
-
- Plain uses Context/Question/Options/Answer sections. Chat uses ChatML with
- a system prompt and an empty think block (ADR-027).
- """
+ """Plain or ChatML prompts; the format must match training (ADR-027)."""
PLAIN = "plain"
CHAT = "chat"
def label_prefix(style: Style) -> str:
- """The character before a label token as the model emits it: a space after
- `Answer:`, nothing at the start of an assistant turn. Verifying the wrong
- variant passes while the scored token differs."""
+ """Use a space after plain Answer: and no prefix at the start of a chat answer."""
return " " if style is Style.PLAIN else ""
@@ -50,11 +45,7 @@ def label_prefix(style: Style) -> str:
@dataclass(frozen=True)
class Rendered:
- """A prompt split at the point the cache is taken.
-
- `prefix` is computed once and reused; `suffix` is the varying part. The scored
- position is always the final token of `suffix`.
- """
+ """Reusable prefix and varying suffix; score the suffix's final token."""
prefix: str
suffix: str
@@ -85,19 +76,13 @@ def render_question(
order: list[int] | None = None,
style: Style = Style.PLAIN,
) -> str:
- """Render one question. `codes` are Mode A label codes; None means Mode B.
-
- `order` indexes the question's own candidates; codes are positional, so two
- orders averaged cancel first-letter bias. A Noul with two codes is the binary
- readout for an untrained checkpoint, which cannot rate 0-8 (ADR-007).
- """
+ """Render positional label codes or Mode B text; two Noul codes select binary readout."""
if style is Style.CHAT:
return _render_question_chat(name, question, codes, order)
lines = [f"Question: {question.instructions or name}"]
if isinstance(question, Choice):
- # Mode B does not list options: the head embeds each candidate's text,
- # and listing 151 intents would cost ~700 tokens per prompt.
+ # Mode B embeds candidates separately, avoiding long option lists in prompts.
if codes:
lines.append("Options:")
items = _ordered(list(question.criteria.items()), order)
@@ -134,8 +119,7 @@ def render_question(
def _render_question_chat(
name: str, question: Question, codes: list[str] | None, order: list[int] | None
) -> str:
- """reflex's headed layout: `# Criterion`, `# Options` with `A. text` lines,
- and an explicit ask. Codes are positional, as in the plain style."""
+ """Render ChatML criterion and option sections with positional codes."""
lines = [f"# Criterion\n{question.instructions or name}\n"]
if isinstance(question, Choice):
@@ -181,8 +165,7 @@ def _render_question_chat(
def render_content(value: JSONContent) -> str:
- """A `JSONContent` field as prompt text. Objects are dumped with sorted keys
- so an identical value always renders identically, as `render_state` does."""
+ """Render JSON content deterministically, sorting object keys."""
return value if isinstance(value, str) else json.dumps(value, sort_keys=True)
@@ -214,16 +197,13 @@ def build(
if layout is Layout.STATE_FIRST:
return Rendered(prefix=state_block, suffix=f"{question_block}\n{ANSWER_CUE}")
- # Schema-first: the catalogue is the reusable prefix and must not depend on the
- # state; `cached_schema` passes the whole catalogue, not just this question.
+ # The reusable schema prefix must contain the full catalogue and exclude state.
prefix = (cached_schema if cached_schema is not None else question_block) + "\n\n"
return Rendered(prefix=prefix, suffix=f"{state_block}{ANSWER_CUE}")
def _build_chat(state, name, question, codes, layout, cached_schema, order) -> Rendered:
- """ChatML: system turn, then a user turn holding evidence and criterion in
- either order, then the opened assistant turn the label token is read from.
- The scored position is the final `\\n\\n` after the empty think block."""
+ """Render ChatML evidence and criterion; score the boundary after the empty think block."""
head = f"<|im_start|>system\n{CHAT_SYSTEM_PROMPT}<|im_end|>\n<|im_start|>user\n"
evidence = f"# Evidence\n{render_state(state)}\n\n"
criterion = render_question(name, question, codes, order, Style.CHAT)
@@ -234,8 +214,8 @@ def _build_chat(state, name, question, codes, layout, cached_schema, order) -> R
def schema_block(
- questions: dict[str, Question],
- codes: dict[str, list[str] | None],
+ questions: Mapping[str, Question],
+ codes: Mapping[str, list[str] | None],
style: Style = Style.PLAIN,
) -> str:
"""The full question catalogue, for schema-first caching across states."""
diff --git a/packages/lev/src/lev/readout/mode_a.py b/packages/lev/src/lev/readout/mode_a.py
index a8affb6..69e2d35 100644
--- a/packages/lev/src/lev/readout/mode_a.py
+++ b/packages/lev/src/lev/readout/mode_a.py
@@ -4,21 +4,18 @@
from dataclasses import dataclass
+from ..labels import TokenEncoder
+
@dataclass
class LabelTokenReadout:
- """Maps label codes to the single token ids Mode A reads.
-
- `tokenizer` must be the one the router verified against; another tokenizer
- silently changes which ids are read.
- """
+ """Read label ids using the same tokenizer that verified the codes."""
- tokenizer: object
+ tokenizer: TokenEncoder
prefix: str = " "
def candidate_ids(self, codes: list[str]) -> list[int]:
- """Token id for each label code as the model emits it at the answer boundary:
- after `Answer:` in the plain style, opening the assistant turn in chat."""
+ """Encode each label code at the configured answer boundary."""
ids = []
for code in codes:
encoded = self.tokenizer.encode(self.prefix + code, add_special_tokens=False)
diff --git a/packages/lev/src/lev/readout/mode_b.py b/packages/lev/src/lev/readout/mode_b.py
index 87de06f..54410ef 100644
--- a/packages/lev/src/lev/readout/mode_b.py
+++ b/packages/lev/src/lev/readout/mode_b.py
@@ -13,8 +13,7 @@ def __init__(self, hidden_size: int = 2560, proj_dim: int = 512, n_heads: int =
super().__init__()
self.question_proj = nn.Linear(hidden_size, proj_dim, bias=False)
self.candidate_proj = nn.Linear(hidden_size, proj_dim, bias=False)
- # Candidates attend to each other: "is this the best of these", not "is
- # this good in isolation".
+ # Compare candidates jointly through set attention.
self.set_attention = nn.MultiheadAttention(proj_dim, n_heads, batch_first=True, dropout=0.0)
self.norm = nn.LayerNorm(proj_dim)
self.score = nn.Linear(proj_dim, 1, bias=False)
@@ -25,16 +24,9 @@ def forward(
candidate_reprs: torch.Tensor,
candidate_mask: torch.Tensor | None = None,
) -> torch.Tensor:
- """
- Args:
- question_repr: `(batch, hidden)`.
- candidate_reprs: `(batch, n_candidates, hidden)`.
- candidate_mask: `(batch, n_candidates)`, True where padded.
-
- Returns:
- `(batch, n_candidates)` raw scores. Masked slots are -inf so they
- vanish under softmax regardless of the temperature applied later.
- """
+ """Score candidates (B, K); input shapes are (B, H) and (B, K, H).
+
+ candidate_mask is (B, K), True for padding; masked scores are -inf."""
q = self.question_proj(question_repr).unsqueeze(1)
c = self.candidate_proj(candidate_reprs)
diff --git a/packages/lev/src/lev/release.py b/packages/lev/src/lev/release.py
index 3e7adcd..e57d53e 100644
--- a/packages/lev/src/lev/release.py
+++ b/packages/lev/src/lev/release.py
@@ -1,7 +1,4 @@
-"""Package adapters, head, tokenizer and calibration into a flat Hub release.
-
-The manifest pins the base model and prompt format required to load it.
-"""
+"""Package adapter, head, tokenizer, and calibration with a base-model and prompt manifest."""
from __future__ import annotations
@@ -26,20 +23,14 @@ def build_release(
metrics: dict | None = None,
prompt_style: str = "plain",
) -> dict:
- """Copy a `step-N` checkpoint into a self-describing release directory.
-
- `calibration` defaults to the `calibration.json` beside the checkpoint's
- parent, where `calibrate` writes it. An uncalibrated release is recorded as
- such and serves raw softmax.
- """
+ """Package a checkpoint; default to parent calibration, otherwise serve raw softmax."""
step_dir = resolve_checkpoint(checkpoint)
target = Path(out)
target.mkdir(parents=True, exist_ok=True)
copied: list[str] = []
for file in sorted(step_dir.iterdir()):
- # Training state is for resuming, not serving; PEFT's README is replaced
- # by the model card.
+ # Exclude resume state and replace PEFT's README with the release card.
if file.is_file() and file.name not in (TRAINING_STATE, MODEL_CARD):
shutil.copy2(file, target / file.name)
copied.append(file.name)
@@ -59,8 +50,7 @@ def build_release(
"lora_rank": adapter_config.get("r"),
"mode_b_head": MODE_B_HEAD in copied,
"calibrated": calibrated,
- # `lev.load` reads `prompt_style` (and `base_model`); `noul_readout`
- # records the readout the temperatures were fitted on.
+ # Record the model, prompt format, and readout required by these weights and temperatures.
"noul_readout": "rating",
"prompt_style": prompt_style,
"files": copied,
@@ -73,7 +63,7 @@ def build_release(
def model_card(manifest: dict) -> str:
- """A Hub model card: front matter the Hub indexes, then what a user needs."""
+ """Build Hub metadata and usage instructions for a release."""
metrics = manifest.get("metrics") or {}
metric_lines = "\n".join(f"| {k} | {v} |" for k, v in metrics.items()) or "| — | not recorded |"
calibration = (
@@ -137,11 +127,7 @@ def model_card(manifest: dict) -> str:
def publish(release_dir: str | Path, repo_id: str, *, private: bool = False) -> str:
- """Upload a release directory to the Hub. Returns the repo URL.
-
- Needs `HF_TOKEN` or `hf auth login`. Creates the repo if it does not exist;
- re-running uploads only the changed files.
- """
+ """Upload a release using HF authentication; create the repo if needed and return its URL."""
from huggingface_hub import HfApi
folder = Path(release_dir)
diff --git a/packages/lev/src/lev/router.py b/packages/lev/src/lev/router.py
index f23dcaf..35222d7 100644
--- a/packages/lev/src/lev/router.py
+++ b/packages/lev/src/lev/router.py
@@ -44,13 +44,7 @@ def route(
label_prefix: str = " ",
skip_multi_token: bool = False,
) -> Route:
- """Pick a mode for one question.
-
- `max_label_options` caps Mode A below the tokenizer limit (decider measured it
- losing 5-24 points on 50-219 options); None uses Mode A whenever it fits.
- `noul_binary` reads a Noul as two lettered options: a stock checkpoint pins the
- 0-8 scale at one end regardless of content (ADR-007).
- """
+ """Choose a readout using tokenizer support, an optional label cap, and binary Noul mode."""
n = candidate_count(question)
if max_label_options is not None and n > max_label_options:
diff --git a/packages/lev/src/lev/server.py b/packages/lev/src/lev/server.py
index ec9c24a..08236f3 100644
--- a/packages/lev/src/lev/server.py
+++ b/packages/lev/src/lev/server.py
@@ -1,12 +1,9 @@
-"""FastAPI endpoints for typed decisions and model health.
-
-Status codes: 422 malformed, 529 still loading, 503 past `max_pending` in flight
-or worker stopped, 504 queued past `timeout`.
-"""
+"""Serve typed decisions and health; return 422, 529, 503, or 504 for request failures."""
from __future__ import annotations
import asyncio
+from contextlib import asynccontextmanager
from typing import Any, Literal
# Module level: FastAPI resolves the endpoint's string annotations here.
@@ -37,11 +34,10 @@ def create_app(
from . import __version__
- app = FastAPI(title="lev", version=__version__)
state: dict[str, Any] = {"engine": None, "batcher": None}
- @app.on_event("startup")
- def _load() -> None:
+ @asynccontextmanager
+ async def lifespan(app):
engine = load(
checkpoint_dir,
model_id=model_id,
@@ -59,6 +55,9 @@ def _load() -> None:
engine, max_pending=max_pending, max_batch_tokens=max_batch_tokens, timeout=timeout
)
state["engine"] = engine
+ yield
+
+ app = FastAPI(title="lev", version=__version__, lifespan=lifespan)
# Async, so a saturated thread pool cannot starve it.
@app.get("/health")
@@ -112,8 +111,7 @@ async def system_one(body: SystemOneRequest, request: Request) -> Any:
async def _disconnected(request: Request) -> None:
- """Return when the client hangs up. With the body read, the next ASGI message is
- the disconnect; any other means none is reported, so wait forever, not spin."""
+ """Wait for an ASGI disconnect; if none is reported, wait without polling."""
if (await request.receive())["type"] == "http.disconnect":
return
await asyncio.Event().wait()
diff --git a/packages/lev/src/lev/train/calibration_run.py b/packages/lev/src/lev/train/calibration_run.py
index 40bd39a..1eb3fc1 100644
--- a/packages/lev/src/lev/train/calibration_run.py
+++ b/packages/lev/src/lev/train/calibration_run.py
@@ -19,14 +19,9 @@ def fit_profile(
config=None,
method: str = "transfer",
) -> dict:
- """Collect raw logits on `split`, fit one temperature per bucket.
+ """Fit bucket temperatures on calibration data; optionally select by family transfer.
- `method="transfer"` fits each bucket both by rows and by task family and
- keeps whichever has the lower leave-one-family-out ECE (ADR-028), writing
- the comparison to `calibration.report.json`; a bucket with too few families
- keeps the row fit. `method="rows"` fits pooled rows only, ignoring family.
- An existing profile is kept as `calibration.previous.json`.
- """
+ Save transfer comparisons and preserve the previous profile before replacing it."""
rows = collect_logits(checkpoint_dir, data_dir, split, config=config)
out = Path(checkpoint_dir) / CALIBRATION
if out.is_file():
@@ -73,12 +68,7 @@ def collect_logits(
batch_size: int = 16,
limit: int | None = None,
) -> dict[str, list[tuple[list[float], int, str]]]:
- """Raw candidate logits per `CalibrationProfile.key` bucket, from the trained
- model under `no_grad`, before any temperature.
-
- Abstain rows are skipped: a temperature is fitted against a gold index. Each
- sample carries its task family (`sources.family_of`) for the transfer fit.
- """
+ """Collect raw logits and task families by calibration bucket, skipping abstain rows."""
import torch
from ..data.build import load_split
diff --git a/packages/lev/src/lev/train/checkpoints.py b/packages/lev/src/lev/train/checkpoints.py
index 3735149..6325b77 100644
--- a/packages/lev/src/lev/train/checkpoints.py
+++ b/packages/lev/src/lev/train/checkpoints.py
@@ -11,10 +11,7 @@
def latest_checkpoint(output: str | Path) -> Path | None:
- """The newest `step-N` under `output` that holds adapter weights, or None.
-
- By step number, not mtime: a resumed run rewrites older directories.
- """
+ """Return the highest numbered step with adapter weights, regardless of mtime."""
source = Path(output)
steps = sorted(
(d for d in source.glob("step-*") if (d / ADAPTER_WEIGHTS).is_file()),
@@ -24,11 +21,7 @@ def latest_checkpoint(output: str | Path) -> Path | None:
def fetch_checkpoint(spec: str | Path, cache_dir: str | None = None) -> Path:
- """A local path as-is, else a Hub id (`hf://org/name` or `org/name`) downloaded.
-
- A missing path shaped `a/b` (not starting `/` or `.`) is tried on the Hub, so a typo like
- `checkpoints/lev-instuct` raises `FileNotFoundError` naming both readings.
- """
+ """Use an existing local path or download a Hub id; report ambiguous missing paths clearly."""
local = Path(spec)
if local.exists():
return local
@@ -47,8 +40,7 @@ def fetch_checkpoint(spec: str | Path, cache_dir: str | None = None) -> Path:
def resolve_checkpoint(path: str | Path, cache_dir: str | None = None) -> Path:
- """Accept a `step-N` directory, a parent holding several (newest step wins), a flat
- release directory, or a Hub id for one."""
+ """Resolve a step directory, training output, flat release, or Hub id."""
source = fetch_checkpoint(path, cache_dir)
if (source / ADAPTER_WEIGHTS).is_file():
return source
@@ -62,27 +54,18 @@ def resolve_checkpoint(path: str | Path, cache_dir: str | None = None) -> Path:
def load_training_state(path: str | Path) -> dict | None:
- """The optimiser, schedule and position saved beside the weights, if any.
-
- None for a weights-only checkpoint (including those from before ADR-021);
- the caller then starts the optimiser and schedule fresh.
- """
+ """Load resume state, or return None for a weights-only checkpoint."""
import torch
file = resolve_checkpoint(path) / TRAINING_STATE
if not file.is_file():
return None
- # Tensors, dicts, tuples and ints only, so `weights_only` loads it without arbitrary
- # unpickling: safe for a state downloaded from a Hub id.
+ # Use weights_only to avoid arbitrary unpickling of downloaded resume state.
return torch.load(file, map_location="cpu", weights_only=True)
def load_checkpoint(model, head, path: str | Path) -> None:
- """Restore adapter and head weights into existing parameter tensors.
-
- Keeping the tensors preserves optimiser references. A missing or unexpected
- head is an error, since dropping it would discard Mode B training.
- """
+ """Restore weights in place to preserve optimizer references; reject head mismatches."""
import torch
from peft import set_peft_model_state_dict
from safetensors.torch import load_file
@@ -117,10 +100,7 @@ def save_checkpoint(
on_checkpoint=None,
state: dict | None = None,
) -> Path:
- """Write adapter, head and tokenizer, then optional training state.
-
- Not atomic: an interrupted save may be incomplete (ADR-021).
- """
+ """Write weights, tokenizer, and optional resume state; interrupted saves may be incomplete."""
import torch
path = output / f"step-{step}"
diff --git a/packages/lev/src/lev/train/collate.py b/packages/lev/src/lev/train/collate.py
index a442cbc..cf103bd 100644
--- a/packages/lev/src/lev/train/collate.py
+++ b/packages/lev/src/lev/train/collate.py
@@ -1,13 +1,14 @@
-"""Batch and collate examples by readout mode.
-
-Rows are right-padded: `last_positions` points to each final real token.
-"""
+"""Collate by readout mode with right padding and final-real-token positions."""
from __future__ import annotations
import random
from collections.abc import Iterable, Iterator
from dataclasses import dataclass
+from typing import TYPE_CHECKING
+
+if TYPE_CHECKING:
+ from torch import Tensor
from ..data.mixture import Example
from ..prompt import Style, candidate_texts, label_prefix
@@ -27,8 +28,7 @@ def __init__(self, tokenizer, max_label_options: int | None = None, style: Style
self._routes: dict[tuple[str, str, int], Route] = {}
def route_for(self, example: Example) -> Route:
- # Keyed on the option count: a source has many questions (subsampled option sets,
- # per-row QA choices), and the route depends only on the count.
+ # Cache by option count; routing is independent of option wording.
key = (example.source, example.name, candidate_count(example.question))
if key not in self._routes:
self._routes[key] = route(
@@ -42,24 +42,23 @@ def route_for(self, example: Example) -> Route:
@dataclass
class Batch:
- """One homogeneous training batch. Tensors are torch, typed loosely to keep
- this module importable without torch."""
+ """A training batch with one readout mode."""
mode: Mode
- input_ids: object # (B, T)
- attention_mask: object # (B, T)
- last_positions: object # (B,)
- candidate_mask: object # (B, Kmax) True where padded
- targets: object # (B,) gold index, IGNORE_INDEX on abstain rows
- soft_targets: object | None # (B, Kmax) or None if no abstain rows
- ordinal: object # (B,) True for the ordered types: Score and Noul
+ input_ids: Tensor # (B, T)
+ attention_mask: Tensor # (B, T)
+ last_positions: Tensor # (B,)
+ candidate_mask: Tensor # (B, Kmax) True where padded
+ targets: Tensor # (B,) gold index, IGNORE_INDEX on abstain rows
+ soft_targets: Tensor | None # (B, Kmax) or None if no abstain rows
+ ordinal: Tensor # (B,) True for the ordered types: Score and Noul
# Mode A only: the label token id per candidate slot.
- candidate_token_ids: object | None = None # (B, Kmax)
+ candidate_token_ids: Tensor | None = None # (B, Kmax)
# Mode B only: a shared pool of encoded candidate strings, indexed per row.
- candidate_input_ids: object | None = None # (C, Tc)
- candidate_attention_mask: object | None = None # (C, Tc)
- candidate_last_positions: object | None = None # (C,)
- candidate_index: object | None = None # (B, Kmax) -> row in C, 0 where padded
+ candidate_input_ids: Tensor | None = None # (C, Tc)
+ candidate_attention_mask: Tensor | None = None # (C, Tc)
+ candidate_last_positions: Tensor | None = None # (C,)
+ candidate_index: Tensor | None = None # (B, Kmax) -> row in C, 0 where padded
@property
def size(self) -> int:
@@ -79,11 +78,7 @@ def render(example: Example, codes: list[str] | None, style: Style = Style.PLAIN
class ModeBatcher:
- """Group by readout mode, sort within length windows, then shuffle batches.
-
- Windowed sorting limits padding without globally ordering rows by length
- or source. No examples are dropped.
- """
+ """Group by mode, sort within length windows, and shuffle batches without dropping examples."""
def __init__(
self,
@@ -131,13 +126,12 @@ def __call__(self, examples: Iterable[Example], epoch: int = 0) -> Iterator[list
for i in range(0, len(chunk), self.batch_size)
)
- # Seeded on the epoch: differs between epochs, reproduces on a re-run.
random.Random(self.seed + epoch).shuffle(batches)
yield from (b for b in batches if b)
class DecisionCollator:
- """Examples -> Batch. All rows must share a mode; `ModeBatcher` guarantees it."""
+ """Collate rows sharing one readout mode."""
def __init__(
self,
@@ -187,8 +181,7 @@ def __call__(self, examples: list[Example]) -> Batch:
)
soft_targets[i, : n_candidates[i]] = torch.tensor(e.soft_target)
- # Noul is ordinal too ("0 = certainly no, 8 = certainly yes"): without
- # this, rating 4 against a truth of 8 costs as much as rating 0.
+ # Treat Noul ratings as ordered so distant errors cost more.
ordinal = torch.tensor(
[e.question.type in ("score", "noul") for e in examples], dtype=torch.bool
)
@@ -219,8 +212,7 @@ def _encode(self, prompts: list[str]) -> dict:
max_length=self.max_seq_len,
add_special_tokens=False,
)
- # The last real token, from the mask rather than len(text): truncation
- # may have shortened the row.
+ # Derive final token positions from the mask to account for truncation.
out["last_positions"] = out["attention_mask"].sum(dim=1).long() - 1
if (out["last_positions"] < 0).any():
raise ValueError("a prompt encoded to zero tokens")
diff --git a/packages/lev/src/lev/train/config.py b/packages/lev/src/lev/train/config.py
index 64690b4..2b41c8a 100644
--- a/packages/lev/src/lev/train/config.py
+++ b/packages/lev/src/lev/train/config.py
@@ -1,21 +1,16 @@
-"""Training presets and approximate H100 compute and memory budgets.
-
-Estimate assumptions are documented in docs/ARCHITECTURE.md §5.
-"""
+"""Training presets and approximate H100 budgets (ARCHITECTURE §5)."""
from __future__ import annotations
from dataclasses import dataclass
from typing import Literal
-# Measured on the released 4b-instruct run (2.46e18 FLOPs in 7.8 h, padding and overhead
-# included); reads high for the plain-prompt 4B runs (4h50, 5h53). docs/TRAINING.md.
+# Effective throughput measured on the 4b-instruct run, including overhead (TRAINING.md).
H100_EFFECTIVE_FLOPS = 8.75e13
H100_VRAM_GB = 80.0
-# ~6*N FLOPs/token forward+backward, plus ~a third for checkpointing's recompute. Holds
-# under LoRA: backward still traverses the frozen weights to reach the adapters.
+# Include checkpoint recomputation; LoRA backward still traverses frozen weights.
FLOPS_PER_PARAM_PER_TOKEN = 8.0
@@ -85,7 +80,7 @@ def total_tokens(self) -> int:
@property
def tokens_per_step(self) -> int:
- # The work per step is the padded batch, not `max_seq_len` (~16x larger).
+ # Estimate compute from padded batch length, not the truncation cap.
return self.avg_tokens_per_example * self.per_device_batch * self.grad_accum
@property
diff --git a/packages/lev/src/lev/train/evaluate.py b/packages/lev/src/lev/train/evaluate.py
index 1bdb5df..5b18507 100644
--- a/packages/lev/src/lev/train/evaluate.py
+++ b/packages/lev/src/lev/train/evaluate.py
@@ -1,5 +1,4 @@
-"""Evaluate checkpoints through the training forward path, per source, before and after
-fitted temperatures."""
+"""Evaluate checkpoints per source before and after temperature scaling."""
from __future__ import annotations
@@ -100,11 +99,7 @@ def evaluate_split(
limit_per_source: int | None = None,
batch_size: int = 16,
) -> tuple[EvalReport, EvalReport]:
- """Return (uncalibrated, calibrated) reports over `split`.
-
- Both come from one forward pass, with the temperature applied to stored raw
- logits, so any difference is the temperature alone.
- """
+ """Return raw and calibrated reports from the same stored logits."""
import torch
from ..data.build import load_split
@@ -157,8 +152,7 @@ def evaluate_split(
resolved = str(resolve_checkpoint(checkpoint_dir))
plain = EvalReport(split=split, checkpoint=resolved, calibrated=False)
tuned = EvalReport(split=split, checkpoint=resolved, calibrated=bool(profile.temperatures))
- # A source's rows may differ in option count, which sets the temperature, so
- # each group is scored at its own temperature and then reported per source.
+ # Apply option-count temperatures per group before aggregating by source.
per_source: dict[tuple[str, str, str], list[tuple[list[float], int, float]]] = {}
for (source, qtype, mode, width), samples in collected.items():
t = profile.temperature(qtype, mode, width)
diff --git a/packages/lev/src/lev/train/loop.py b/packages/lev/src/lev/train/loop.py
index 7e01e5e..eeed601 100644
--- a/packages/lev/src/lev/train/loop.py
+++ b/packages/lev/src/lev/train/loop.py
@@ -32,13 +32,14 @@ def build_model(config: TrainConfig, model_cache: str | None = None):
from transformers import AutoModelForCausalLM, AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained(config.model_id, cache_dir=model_cache)
+ if tokenizer is None:
+ raise RuntimeError(f"could not load tokenizer for {config.model_id!r}")
if tokenizer.pad_token_id is None:
- # Rows are right-padded and `last_positions` comes from the attention mask,
- # so the pad token only has to exist.
+ # Right padding uses mask-derived positions, so an existing token can serve as padding.
tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = "right"
- # bfloat16 on CPU is glacial, and `make smoke-local` runs on a laptop.
+ # Use fp32 for CPU smoke runs.
dtype = getattr(torch, config.dtype)
if dtype is torch.bfloat16 and not torch.cuda.is_available():
print("no CUDA; training in float32 instead of bfloat16")
@@ -48,8 +49,7 @@ def build_model(config: TrainConfig, model_cache: str | None = None):
model: Any = AutoModelForCausalLM.from_pretrained(
config.model_id,
dtype=dtype,
- # Multi-GPU only: on an Apple machine accelerate dispatches to MPS and
- # the process dies with SIGSEGV part-way through loading.
+ # Restrict automatic device mapping to CUDA; MPS loading can crash.
device_map="auto" if torch.cuda.device_count() > 1 else None,
cache_dir=model_cache,
)
@@ -58,8 +58,7 @@ def build_model(config: TrainConfig, model_cache: str | None = None):
if config.gradient_checkpointing:
model.gradient_checkpointing_enable()
- # Without it, checkpointed activations have no grad_fn and the LoRA
- # adapters get no gradient: a silent no-op run.
+ # Enable input gradients so checkpointing propagates gradients to LoRA adapters.
model.enable_input_require_grads()
if config.use_lora:
@@ -102,8 +101,7 @@ def decision_loss(logits, targets, config: TrainConfig, ordinal=False, soft_targ
if soft_targets is not None and (~hard).any():
target_dist[~hard] = soft_targets[~hard].to(probs.dtype)
- # `torch.where`, not a product: a padded slot has logit -inf and target 0, and
- # `0 * -inf` is NaN, which poisons the batch mean.
+ # Use where to avoid NaN from padded slots with 0 * -inf.
weighted = torch.where(target_dist > 0, target_dist * log_probs, torch.zeros_like(log_probs))
ce = -weighted.sum(dim=-1).mean()
brier = ((probs - target_dist) ** 2).sum(dim=-1).mean()
@@ -120,18 +118,13 @@ def decision_loss(logits, targets, config: TrainConfig, ordinal=False, soft_targ
def candidate_logits(model, batch, head=None):
- """One forward pass -> `(B, K)` logits over the batch's candidate set.
-
- Mode A gathers label-token logits at the answer boundary. Mode B scores pooled
- candidate strings through the head: one extra short forward, not one per option.
- """
+ """Compute (B, K) scores using label logits or pooled candidate embeddings."""
import torch
from ..router import Mode
if batch.mode is Mode.LABEL_TOKEN:
- # Project only answer positions to avoid a full (B, seq, vocab) allocation.
- # Rows with different answer positions must pick their own output column.
+ # Project answer positions only; each row selects its own output column.
keep, column = torch.unique(batch.last_positions, return_inverse=True)
out = model(
input_ids=batch.input_ids,
@@ -163,13 +156,11 @@ def candidate_logits(model, batch, head=None):
flat = batch.candidate_index.reshape(-1)
reprs = candidate_repr[flat].view(*batch.candidate_index.shape, -1) # (B, K, H)
- # Cast hidden states up to the fp32 head. Casting to the bf16 activations'
- # dtype instead raises on GPU, and is invisible on CPU, where both are fp32.
+ # Match hidden states to the fp32 head, not the bf16 backbone.
target_dtype = next(head.parameters()).dtype
logits = head(question_repr.to(target_dtype), reprs.to(target_dtype), batch.candidate_mask)
- # Mode B already masks padded slots; Mode A's gather returns a real (wrong)
- # vocabulary logit there.
+ # Mask Mode A padding after gathering vocabulary logits.
return logits.float().masked_fill(batch.candidate_mask.to(logits.device), float("-inf"))
@@ -218,8 +209,7 @@ def prepare_data(config: TrainConfig, data_dir: str):
if not train:
raise ValueError(f"{root / 'train.jsonl'} is empty")
- # Repeats the contamination guard on the manifest's Hub ids: the only check
- # that sees a manifest edited after the build. Skipped without a manifest.
+ # Recheck manifest Hub ids in case the manifest changed after data assembly.
manifest_path = root / MANIFEST
if manifest_path.is_file():
from ..data.contamination import assert_clean
@@ -240,19 +230,15 @@ def run_training(
max_steps: int | None = None,
fresh: bool = False,
) -> dict:
- """Train, checkpointing periodically so a long run survives a preemption.
+ """Train with periodic checkpoints; auto-resume the newest step unless fresh=True.
- With no `resume_from`, the newest `step-N` under `config.output_dir` is resumed;
- `fresh=True` starts over. Resume restores optimiser, schedule and data order, so a
- finished run trains nothing; pre-ADR-021 checkpoints restore weights only.
- """
+ Resume optimizer, schedule, and data order when saved; otherwise restore weights only."""
import torch
from torch.optim import AdamW
from transformers import get_cosine_schedule_with_warmup
config.validate()
- # Everything cheap and fallible happens before the model is touched.
data = prepare_data(config, data_dir)
output = Path(config.output_dir)
output.mkdir(parents=True, exist_ok=True)
@@ -260,8 +246,7 @@ def run_training(
if resume_from is None and not fresh and (found := latest_checkpoint(output)) is not None:
resume_from = str(found)
print(f"resuming from {found} (pass --fresh to start over)", flush=True)
- # Move, not delete, the previous run, so auto-resume (highest step) and the
- # calibration lookup cannot pick it up.
+ # Preserve the previous run outside active resume and calibration lookups.
if fresh and (moved := set_aside_previous_run(output)) is not None:
print(f"previous run moved to {moved}", flush=True)
@@ -291,8 +276,7 @@ def run_training(
train = data["train"]
steps_per_epoch = max(1, len(train) // config.examples_per_step)
total_steps = max_steps or steps_per_epoch * config.epochs
- # Sized in optimiser steps: sized in micro-steps, the cosine would finish
- # `grad_accum` times early and the tail would train at a learning rate of 0.
+ # Size the schedule in optimizer steps so accumulation does not finish it early.
optimiser_steps = max(1, total_steps // config.grad_accum)
scheduler = get_cosine_schedule_with_warmup(
optimiser, int(optimiser_steps * config.warmup_ratio), optimiser_steps
@@ -331,8 +315,7 @@ def run_training(
model.train()
def training_state() -> dict:
- # Read at save time: the loop below rebinds `epoch`, `step_in_epoch` and
- # `rng_before_epoch`, and every save happens after at least one step.
+ # Read loop position and RNG state at save time, after they are updated.
return {
"step": step,
"epoch": epoch,
@@ -348,7 +331,6 @@ def training_state() -> dict:
# Before the shuffle, so a resumed run reproduces this epoch's order.
rng_before_epoch = rng.getstate()
order = list(train)
- # Shuffled before bucketing so windows differ per epoch.
rng.shuffle(order)
groups = batcher(order, epoch=epoch)
step_in_epoch = 0
@@ -405,7 +387,7 @@ def training_state() -> dict:
state=training_state(),
)
last_saved = step
- # So a run that dies late still leaves its loss curve.
+ # Persist history periodically to survive late preemption.
_write_history(output, history, step)
if step >= total_steps:
break
@@ -424,10 +406,7 @@ def training_state() -> dict:
def set_aside_previous_run(output: Path) -> Path | None:
- """Move prior checkpoints, history and calibration under `superseded-`.
-
- Return the directory, or None if there was nothing to move.
- """
+ """Move previous run artifacts into a timestamped directory, or return None if empty."""
stale = list(output.glob("step-*")) + [
output / name for name in (HISTORY, CALIBRATION) if (output / name).exists()
]
diff --git a/packages/lev/src/lev/train/progress.py b/packages/lev/src/lev/train/progress.py
index 5c06ed6..8277e2c 100644
--- a/packages/lev/src/lev/train/progress.py
+++ b/packages/lev/src/lev/train/progress.py
@@ -6,12 +6,7 @@
class ProgressLog:
- """Periodic one-line progress, flushed: on Modal stdout is the only view, and it is
- block-buffered when not a tty.
-
- Losses are per mode (Mode B starts near `ln(K)`). Throughput counts real tokens from the
- attention mask, so it can contradict a wrong plan (ADR-016).
- """
+ """Flush periodic per-mode losses and throughput measured from real tokens."""
def __init__(self, total_steps: int, log_every: int):
self.total = total_steps
@@ -31,8 +26,7 @@ def record(self, step: int, loss: float, mode: str, lr: float, tokens: int = 0)
now = time.monotonic()
elapsed = max(now - self.start, 1e-9)
- # Over the window, not since start: startup skews a cumulative rate ~8x (ADR-017).
- # A coarse clock can report zero on a fast window; never divide by it.
+ # Use windowed rates to exclude startup delay; guard against zero elapsed time.
span = max(now - self.mark, 1e-9)
rate = (done - self.mark_step) / span
remaining = (self.total - done) / rate if rate else 0.0
diff --git a/packages/lev/src/lev/types.py b/packages/lev/src/lev/types.py
index 2e12b5e..f8c205e 100644
--- a/packages/lev/src/lev/types.py
+++ b/packages/lev/src/lev/types.py
@@ -1,8 +1,4 @@
-"""TypeSafe-compatible request and response models for `/v1/systemone`.
-
-lev also exposes Noul rating probabilities and confidence. Clients that only
-read the shared `noul` probability can ignore these extra fields.
-"""
+"""TypeSafe-compatible models; lev adds optional Noul rating probabilities and confidence."""
from __future__ import annotations
@@ -60,9 +56,7 @@ def _level_count(cls, v: list) -> list:
def question_payload(question: Question) -> dict:
- """The question as a levbench task file carries it. `criteria` is required on
- a Choice and a Score and optional on a Noul, so an absent Noul criteria map
- stays absent rather than round-tripping as an empty one."""
+ """Serialize a task question, preserving absent Noul criteria."""
payload: dict = {"type": question.type, "instructions": question.instructions}
if isinstance(question, Choice | Score) or question.criteria:
payload["criteria"] = question.criteria
diff --git a/packages/lev/tests/conftest.py b/packages/lev/tests/conftest.py
index 95f0f2b..834b950 100644
--- a/packages/lev/tests/conftest.py
+++ b/packages/lev/tests/conftest.py
@@ -1,5 +1,4 @@
-"""torch is imported only inside `BatchingTokenizer.__call__`, so the torch-free half of
-the suite still collects on a bare `uv sync`."""
+"""Shared fixtures; import Torch only when tensor batching is needed."""
from __future__ import annotations
@@ -11,8 +10,7 @@
class FakeTokenizer:
- """Strings in `single_tokens` encode to one id -- exactly what `labels.single_token_codes`
- checks, so a fake tests the routing decision faithfully."""
+ """Control which strings encode as a single token."""
def __init__(self, single_tokens: set[str]):
self.single_tokens = single_tokens
@@ -20,8 +18,7 @@ def __init__(self, single_tokens: set[str]):
def encode(self, text: str, add_special_tokens: bool = True) -> list[int]:
if text in self.single_tokens:
return [1]
- # Even a single character must encode to >1 token, or it looks single-token by
- # accident and the router test proves nothing.
+ # Encode unlisted characters as multiple tokens to avoid accidental routing matches.
return [1] * (len(text) + 1)
@@ -50,8 +47,7 @@ def poor_tokenizer() -> FakeTokenizer:
class BatchingTokenizer(FakeTokenizer):
- """One id per character so prompts get different lengths; equal lengths would hide the
- bug `last_positions` exists to prevent."""
+ """Encode one id per character to expose padding and last-token errors."""
pad_token_id = 0
padding_side = "right"
diff --git a/packages/lev/tests/test_batcher.py b/packages/lev/tests/test_batcher.py
index 0a9031c..de8a811 100644
--- a/packages/lev/tests/test_batcher.py
+++ b/packages/lev/tests/test_batcher.py
@@ -15,8 +15,7 @@ def request(name: str, rows: int = 1, width: int = 10):
class GatedEngine:
- """Records each forward's requests; the first forward waits for `release`,
- so requests submitted meanwhile queue up behind it."""
+ """Hold the first forward so later requests queue behind it."""
def __init__(self):
self.forwards: list[list[str]] = []
@@ -185,8 +184,6 @@ async def until(condition, timeout=2.0):
def test_a_held_request_can_still_be_cancelled_before_it_runs():
- """With a budget of one row, `c` is held back while `b` runs; its client
- leaving then must keep it off the GPU."""
async def run():
engine = SteppedEngine()
@@ -232,7 +229,7 @@ async def run():
def test_an_abandoned_running_request_still_counts_as_pending():
- """A client leaving does not free the GPU, so it must not free a slot."""
+ """Disconnecting does not release GPU capacity already in use."""
async def run():
engine = SteppedEngine()
@@ -252,8 +249,6 @@ async def run():
def test_a_failing_disconnect_watch_still_returns_the_answer():
- """A server whose `receive()` raises cannot report disconnects; the request
- must be answered, not dropped as if its client had left."""
async def broken_watch():
raise RuntimeError("receive() is not available")
@@ -270,8 +265,7 @@ async def run():
def test_an_error_outside_the_forward_fails_its_batch_and_the_worker_keeps_serving():
- """One answer short trips the `zip(strict=True)` after the forward, outside
- the forward's own error handling."""
+ """A short response list triggers zip(strict=True) outside forward error handling."""
async def run():
engine = SteppedEngine(short_by=1)
diff --git a/packages/lev/tests/test_cache_fork.py b/packages/lev/tests/test_cache_fork.py
index 13e96cd..999c64a 100644
--- a/packages/lev/tests/test_cache_fork.py
+++ b/packages/lev/tests/test_cache_fork.py
@@ -1,4 +1,4 @@
-"""Qwen3.5's `LinearAttentionLayer` has no `batch_repeat_interleave`; a fork built on it raised."""
+"""Test prefix-cache copying with mixed full- and linear-attention layers."""
from __future__ import annotations
@@ -51,7 +51,6 @@ def test_every_forked_row_is_a_copy_of_row_zero():
def test_fork_does_not_mutate_the_source():
- """The prefix cache must stay reusable for the next request."""
original = hybrid_cache()
before_keys = original.layers[0].keys.clone()
before_conv = original.layers[1].conv_states[0].clone()
@@ -64,7 +63,7 @@ def test_fork_does_not_mutate_the_source():
def test_the_old_approach_still_fails():
- """Pins why `_fork` avoids `batch_repeat_interleave`, so nobody 'simplifies' it back."""
+ """Hybrid caches do not support batch_repeat_interleave."""
linear = cache_utils.LinearAttentionLayer()
assert not hasattr(linear, "batch_repeat_interleave"), (
"LinearAttentionLayer now implements batch_repeat_interleave; "
diff --git a/packages/lev/tests/test_calibration.py b/packages/lev/tests/test_calibration.py
index 6dc538f..197a085 100644
--- a/packages/lev/tests/test_calibration.py
+++ b/packages/lev/tests/test_calibration.py
@@ -1,4 +1,4 @@
-"""Calibration must actually reduce ECE, and must refuse to cheat."""
+"""Test calibration improvement and held-out split rejection."""
from __future__ import annotations
@@ -11,14 +11,13 @@
expected_calibration_error,
fit,
fit_temperature,
- nll,
+ negative_log_likelihood,
softmax,
)
def overconfident_samples(n: int = 600, seed: int = 7, sharpness: float = 4.0):
- """The untuned-backbone failure (ECE 0.4252): the correct class wins ~70% of the time,
- but the margin implies ~96%."""
+ """Generate roughly 70% accuracy at 96% confidence."""
rng = random.Random(seed)
samples = []
for _ in range(n):
@@ -56,19 +55,17 @@ def test_fitting_reduces_ece_on_overconfident_logits(self):
[softmax(x, t) for x, _ in samples], [y for _, y in samples]
)
assert after < before, f"ECE got worse: {before:.4f} -> {after:.4f}"
- # The measured effect on real backbones is 0.43 -> 0.08. Demand a real
- # improvement, not a rounding-level one.
+ # Require a meaningful calibration improvement.
assert after < before / 2, f"expected a large reduction, {before:.4f} -> {after:.4f}"
- def test_fitting_minimises_nll(self):
+ def test_fitting_minimises_negative_log_likelihood(self):
samples = overconfident_samples()
t = fit_temperature(samples)
- best = nll(samples, t)
+ best = negative_log_likelihood(samples, t)
for other in (t * 0.5, t * 1.5, 1.0):
- assert best <= nll(samples, other) + 1e-6
+ assert best <= negative_log_likelihood(samples, other) + 1e-6
def test_a_flat_objective_returns_the_identity(self):
- # All-equal logits: every temperature gives the same NLL, so none is fitted.
samples = [([0.0, 0.0], i % 2) for i in range(400)]
assert fit_temperature(samples) == 1.0
@@ -141,8 +138,7 @@ def test_banded_temperature_falls_back_to_the_unbanded_bucket(self):
class TestTransferSelectedCalibration:
def family_rows(self):
- """Equally confident rows where large easy families dominate a small hard one, so the
- row fit is overconfident on the hard family."""
+ """Large easy families outnumber a small hard family at equal confidence."""
rows = []
for fam, n, acc in (
("easy1", 400, 0.97),
diff --git a/packages/lev/tests/test_collate.py b/packages/lev/tests/test_collate.py
index b2bdc31..cc3912f 100644
--- a/packages/lev/tests/test_collate.py
+++ b/packages/lev/tests/test_collate.py
@@ -1,4 +1,4 @@
-"""torch is in the `train` extra, so these skip on a bare `uv sync`."""
+"""Tensor collation tests; skip when Torch is unavailable."""
from __future__ import annotations
@@ -20,7 +20,7 @@ def collator(self, batching_tokenizer):
return DecisionCollator(batching_tokenizer, max_seq_len=512)
def test_last_position_is_the_final_real_token_not_seq_minus_one(self, collator):
- """Rows are right-padded, so `seq - 1` reads a pad token's logits: noise, and no error."""
+ """Right padding makes seq - 1 point to a pad token."""
batch = collator(
[an_example(a_choice(), state="short"), an_example(a_choice(), state="x" * 200)]
)
@@ -77,7 +77,6 @@ def test_a_large_option_set_routes_to_mode_b_with_a_candidate_pool(self, collato
assert batch.candidate_input_ids.shape[0] == 40
def test_the_candidate_pool_is_shared_across_rows(self, collator):
- """A 77-option question in a batch of 8 must not cost 8x77 encodes."""
rows = [an_example(a_choice(40), target=i) for i in range(4)]
batch = collator(rows)
assert batch.candidate_input_ids.shape[0] == 40, "pool grew with batch size"
@@ -102,8 +101,6 @@ def test_batches_are_homogeneous_and_correctly_sized(self, batching_tokenizer):
assert sum(len(g) for g in groups) == 12
def test_routes_are_resolved_once_per_question_not_per_row(self, batching_tokenizer):
- """Re-tokenising an option set for each of 200k rows is the slowest
- thing in the pipeline, and it produces the same answer every time."""
batcher = ModeBatcher(batching_tokenizer, batch_size=2)
rows = [an_example(a_choice(4), source="same") for _ in range(50)]
resolved = {id(batcher.route_for(row)) for row in rows}
@@ -120,8 +117,7 @@ def test_batcher_and_collator_can_share_one_route_cache(self, batching_tokenizer
class TestLengthBucketing:
- """A batch pads to its longest row; on the real mixture random batching wasted 4.43x
- on padding and bucketing 1.43x (ADR-017)."""
+ """Test padding savings without global length ordering."""
def rows(self, n=512):
rng = random.Random(0)
@@ -143,8 +139,7 @@ def test_bucketing_cuts_padding_waste(self, batching_tokenizer):
before = self.waste(list(unbucketed(rows)))
after = self.waste(list(bucketed(rows)))
assert before > 2.0, f"fixture is not bimodal enough to show the effect ({before:.2f})"
- # Relative, not absolute: the absolute floor depends on where the window
- # boundary falls, and the multiple of compute saved is what matters.
+ # Compare relative padding because window boundaries affect the absolute count.
assert after < before / 2, f"bucketing only got {before:.2f}x -> {after:.2f}x"
assert after < 1.5, f"bucketing left {after:.2f}x waste"
@@ -155,14 +150,12 @@ def test_no_example_is_lost_or_duplicated(self, batching_tokenizer):
assert {id(e) for e in batched} == {id(e) for e in rows}
def test_batches_stay_mode_homogeneous(self, batching_tokenizer):
- """Bucketing must not undo the mode grouping the collator relies on."""
rows = self.rows(100) + [an_example(a_choice(40), target=0) for _ in range(60)]
batcher = ModeBatcher(batching_tokenizer, 16, bucket_window=4)
for group in batcher(rows):
assert len({batcher.route_for(e).mode for e in group}) == 1
def test_batch_order_differs_between_epochs(self, batching_tokenizer):
- """Length must not track step number, or the schedule correlates with it."""
rows = self.rows(256)
batcher = ModeBatcher(batching_tokenizer, 32, bucket_window=4)
first = [len(str(b[0].state)) for b in batcher(rows, epoch=0)]
diff --git a/packages/lev/tests/test_contamination.py b/packages/lev/tests/test_contamination.py
index a391c00..6c3185b 100644
--- a/packages/lev/tests/test_contamination.py
+++ b/packages/lev/tests/test_contamination.py
@@ -1,4 +1,4 @@
-"""The guard must catch aliases, not just exact names, and must raise not warn."""
+"""Test blocked dataset names, aliases, and re-hosts."""
from __future__ import annotations
@@ -73,8 +73,7 @@ def test_innocent_datasets_pass(self, innocent):
assert resolve(innocent) is None
def test_subsets_that_did_not_run_are_blocked_too(self):
- """The 7 subsets `s1-fast` skipped are still evaluation data: easy to
- forget, since the completed-run leaderboard never mentions them."""
+ """All evaluation subsets are blocked, including those absent from completed runs."""
unrun = [
"massive-de-DE",
"squad2",
diff --git a/packages/lev/tests/test_core.py b/packages/lev/tests/test_core.py
index 0d9f056..08c9afa 100644
--- a/packages/lev/tests/test_core.py
+++ b/packages/lev/tests/test_core.py
@@ -66,8 +66,7 @@ def test_rejects_zero_and_oversized_option_counts(self):
label_codes(100_000)
def test_single_token_check_uses_the_space_prefix(self, rich_tokenizer):
- # The scored token follows "Answer:", so it is space-prefixed. Verifying
- # the bare code would pass while the real token differs.
+ # Verify space-prefixed codes at the actual answer boundary.
assert single_token_codes(rich_tokenizer, 5) == ["A", "B", "C", "D", "E"]
assert single_token_codes(rich_tokenizer, 5, prefix="") is None
@@ -95,7 +94,6 @@ def test_small_choice_uses_mode_a(self, rich_tokenizer):
assert r.codes == ["A", "B", "C"]
def test_falls_through_to_mode_b_instead_of_rejecting(self, poor_tokenizer):
- """The differentiator: where other implementations reject, lev routes."""
q = Choice(criteria={k: None for k in "abcdefgh"})
r = route(q, poor_tokenizer)
assert r.mode is Mode.CANDIDATE_PATH
@@ -110,11 +108,13 @@ def test_policy_cap_forces_mode_b_below_the_tokenizer_limit(self, rich_tokenizer
def test_noul_routes_on_the_rating_scale(self, rich_tokenizer):
r = route(Noul(instructions="urgent?"), rich_tokenizer)
assert r.mode is Mode.LABEL_TOKEN
+ assert r.codes is not None
assert len(r.codes) == len(NOUL_RATING_TOKENS)
def test_score_routes_on_level_count(self, rich_tokenizer):
r = route(Score(criteria=["low", "mid", "high"]), rich_tokenizer)
- assert r.mode is Mode.LABEL_TOKEN and len(r.codes) == 3
+ assert r.mode is Mode.LABEL_TOKEN
+ assert r.codes is not None and len(r.codes) == 3
def test_route_all_is_per_question(self, poor_tokenizer):
routes = route_all(
@@ -179,8 +179,6 @@ def test_score_level_bounds(self):
class TestOptionCap:
def test_generous_tokenizer_still_routes_above_the_cap_to_mode_b(self):
- """An explicit `max_label_options` forces Mode B even when the tokenizer
- can express every code (Qwen3.5 encodes codes up to `BP` as one token)."""
generous = two_letter_tokenizer()
sixty = Choice(criteria={f"o{i}": None for i in range(60)})
assert single_token_codes(generous, 60) is not None, "fixture must be generous"
@@ -200,6 +198,7 @@ def test_binary_route_uses_two_lettered_codes(self, rich_tokenizer):
def test_default_route_is_still_the_rating_scale(self, rich_tokenizer):
r = route(Noul(instructions="safe?"), rich_tokenizer)
+ assert r.codes is not None
assert len(r.codes) == len(NOUL_RATING_TOKENS)
def test_binary_render_lists_yes_then_no(self):
@@ -232,8 +231,6 @@ def test_average_maps_reversed_row_back_to_canonical_slots(self):
assert merged == pytest.approx([0.7, 0.2, 0.1])
def test_average_cancels_a_first_position_bias(self):
- """A first-position bonus lands on different candidates in the two orders, so averaging
- spreads it back out."""
biased_forward = [0.6, 0.2, 0.2] # candidate 0 first, gets the bonus
biased_backward = [0.6, 0.2, 0.2] # candidate 2 first, gets the bonus
merged = average_orders([biased_forward, biased_backward], [None, [2, 1, 0]])
@@ -271,8 +268,7 @@ def test_binary_noul_is_read_both_ways(self, rich_tokenizer):
]
def test_score_keeps_its_level_order(self, rich_tokenizer):
- """Levels run low to high and training never reorders them; a reversed
- scale is a prompt the model has not seen (helpsteer2: -2.4 points)."""
+ """Training presents ordered levels from low to high."""
assert self.orders(Score(criteria=["low", "mid", "high"]), rich_tokenizer) == [None]
def test_rating_noul_keeps_its_scale(self, rich_tokenizer):
@@ -349,8 +345,7 @@ def test_label_prefix_is_bare_for_chat_and_space_for_plain(
assert label_prefix(Style.PLAIN) == " " and label_prefix(Style.CHAT) == ""
q = Choice(criteria={"a": None, "b": None, "c": None})
assert route(q, chat_tokenizer, label_prefix="").codes == ["A", "B", "C"]
- # A tokenizer with only space-prefixed letters cannot express bare codes:
- # the chat style falls through to Mode B there rather than misreading.
+ # Plain-only label tokens must fall back to Mode B for chat prompts.
assert route(q, rich_tokenizer, label_prefix="").mode is Mode.CANDIDATE_PATH
def test_plain_style_is_unchanged(self):
@@ -369,6 +364,7 @@ def test_default_scheme_stops_at_the_first_split_code(self):
def test_skipping_passes_over_split_codes_and_keeps_the_order(self):
tok = two_letter_tokenizer(missing="BQ")
codes = single_token_codes(tok, 77, skip_multi_token=True)
+ assert codes is not None
assert len(codes) == 77 and "BQ" not in codes and len(set(codes)) == 77
assert codes[:68] == single_token_codes(tok, 68), "codes below the split are unchanged"
assert codes[68] == "BR"
@@ -385,11 +381,16 @@ def test_route_uses_mode_a_past_the_split_only_when_asked(self):
class TestRoutePreviewMatchesServing:
def preview(self, tokenizer, request, tmp_path, monkeypatch, capsys) -> str:
- """Run `lev route` on `request`, with `tokenizer` standing in for the Hub's."""
+ """Run the route CLI with an injected tokenizer."""
from lev.cli import main
stub = types.ModuleType("transformers")
- stub.AutoTokenizer = types.SimpleNamespace(from_pretrained=lambda *_a, **_k: tokenizer)
+ monkeypatch.setattr(
+ stub,
+ "AutoTokenizer",
+ types.SimpleNamespace(from_pretrained=lambda *_a, **_k: tokenizer),
+ raising=False,
+ )
monkeypatch.setitem(sys.modules, "transformers", stub)
path = tmp_path / "request.json"
path.write_text(json.dumps(request))
@@ -398,8 +399,6 @@ def preview(self, tokenizer, request, tmp_path, monkeypatch, capsys) -> str:
return capsys.readouterr().out
def test_lev_route_previews_the_mode_the_server_uses(self, tmp_path, monkeypatch, capsys):
- """A 60-option Choice is Mode A when served, since the tokenizer can
- express 60 codes, so the preview must say Mode A too."""
request = {
"state": "turn the lights off",
"questions": {
@@ -420,31 +419,27 @@ def test_lev_route_prints_a_mode_b_route_without_codes(self, tmp_path, monkeypat
class TestSystemOneAcceptsPlainDicts:
- """`lev.load` hands users an engine; they pass questions as JSON-shaped dicts."""
-
def engine(self, tokenizer, **config):
from lev.model import DecisionEngine, EngineConfig
return DecisionEngine(model=None, tokenizer=tokenizer, config=EngineConfig(**config))
def test_a_dict_question_is_parsed_and_routed(self, rich_tokenizer):
- # Capped at two label options with no head loaded, a three-option Choice
- # needs Mode B, so routing refuses before any forward pass is attempted.
+ # Require a Mode B head when the label cap excludes this question.
engine = self.engine(rich_tokenizer, max_label_options=2)
questions = {"pick": {"type": "choice", "criteria": {"a": None, "b": None, "c": None}}}
with pytest.raises(RuntimeError, match="need Mode B"):
engine.system_one("state", questions)
def test_a_malformed_dict_is_a_value_error(self, rich_tokenizer):
- """ValueError is what the server turns into a 422."""
+ """The server maps ValueError to HTTP 422."""
engine = self.engine(rich_tokenizer)
with pytest.raises(ValueError, match="at least 2 options"):
engine.system_one("state", {"pick": {"type": "choice", "criteria": {"a": None}}})
class TestTokenizerIsUsedOneCallAtATime:
- """A fast tokenizer keeps padding and truncation as state on one shared Rust object, so
- overlapping calls raise "Already borrowed" or encode with another call's truncation."""
+ """Concurrent access to a fast tokenizer can corrupt settings or raise Already borrowed."""
class OverlapDetector:
def __init__(self, inner):
diff --git a/packages/lev/tests/test_data_pipeline.py b/packages/lev/tests/test_data_pipeline.py
index 83bcc3b..88576e1 100644
--- a/packages/lev/tests/test_data_pipeline.py
+++ b/packages/lev/tests/test_data_pipeline.py
@@ -1,5 +1,4 @@
-"""Offline: `load_source` takes an injected `load_dataset`, so nothing here checks that the
-real dataset ids resolve; a dead id surfaces on the next `lev data build`."""
+"""Test data assembly with fake loaders; upstream dataset availability is not checked."""
from __future__ import annotations
@@ -90,8 +89,7 @@ def test_mode_b_sources_exist_and_exceed_the_code_ceiling(self):
def test_mode_b_gets_a_material_share_of_the_mixture(self):
weights = default_weights()
share = sum(weights[n] for n in MODE_B_SOURCES)
- # Size-proportional weighting would starve Mode B; this pins the
- # deliberate over-weighting.
+ # Reserve enough weight to train Mode B.
assert share >= 0.2, f"Mode B is only {share:.1%} of the mixture"
def test_weights_are_a_distribution_over_known_sources(self):
@@ -111,18 +109,19 @@ def test_noul_needs_exactly_two_labels(self):
def test_score_keeps_level_order(self):
spec = REGISTRY["sst5"]
+ assert spec.label_names is not None
question = build_question(spec, list(spec.label_names))
assert question.criteria == list(spec.label_names)
def test_choice_options_are_humanised(self):
question = build_question(REGISTRY["banking77"], ["card_arrival", "age_limit"])
+ assert isinstance(question, Choice)
assert set(question.criteria) == {"card arrival", "age limit"}
class TestSampling:
def test_limit_samples_rather_than_truncates(self):
- """A head slice of a label-sorted corpus yields one class (`imdb[:400]` is
- 400 negative reviews)."""
+ """A head slice of a label-sorted corpus may contain only one class."""
spec = REGISTRY["dbpedia_14"]
loader = label_sorted_loader(14, 100, text_field="content")
rows = list(load_source(spec, limit=140, load_dataset=loader))
@@ -135,8 +134,7 @@ def test_sampling_is_reproducible(self):
assert a == b
def test_noul_labels_land_on_the_ends_of_the_rating_scale(self):
- """The raw class would supervise "yes" as rating 1, which
- `noul_probability` reads back as P(yes) = 0.125."""
+ """A binary label of 1 must become rating 8, not P(yes) = 0.125."""
loader = label_sorted_loader(2, 20)
rows = list(load_source(REGISTRY["imdb"], load_dataset=loader))
assert {e.target for e in rows} == {0, len(NOUL_RATING_TOKENS) - 1}
@@ -183,7 +181,7 @@ def test_splits_are_disjoint(self, examples):
assert not states[Split.CALIBRATION] & states[Split.TEST]
def test_assignment_is_stable_across_processes(self):
- """`hash()` is salted per process, so a split built on it reshuffles on every rebuild."""
+ """Python hash() is salted per process."""
import os
import subprocess
import sys
@@ -215,7 +213,7 @@ def test_label_only_outside_train_raises(self):
check_coverage(splits)
def test_option_never_observed_at_all_raises(self):
- """A sample holding 2 of 4 options passes the in-train check but not this."""
+ """Train coverage alone misses offered labels absent from every split."""
splits = {
Split.TRAIN: [traceable_example("s", 0, 1), traceable_example("s", 1, 2)],
Split.CALIBRATION: [],
@@ -225,7 +223,7 @@ def test_option_never_observed_at_all_raises(self):
check_coverage(splits)
def test_noul_is_exempt_from_the_offered_option_check(self):
- """Nine rating levels, two classes in the data. Not an error."""
+ """Binary data observes only the endpoints of the nine-level scale."""
rows = [
Example("t", "n", Noul(instructions="q"), t, Layout.STATE_FIRST, "imdb")
for t in (0, 8, 0, 8)
@@ -247,8 +245,6 @@ def mixture(self):
return list(build_mixture(spec, loaders))
def test_abstain_examples_carry_a_foreign_state(self, mixture):
- """An abstain row must be unanswerable, so its state is replaced, not just
- its label."""
abstained = [e for e in mixture if e.abstain]
assert abstained
assert all(not str(e.state).startswith(e.source) for e in abstained)
@@ -274,8 +270,7 @@ def test_the_threshold_is_the_single_letter_alphabet(self):
class TestAbstainDonors:
- """Adjacent sources cannot supply an abstain state: an imdb question is still
- answerable from a rotten_tomatoes review."""
+ """Adjacent sources may still answer the question and are unsuitable abstain donors."""
def test_adjacency_is_symmetric_and_excludes_self(self):
mapping = adjacency_map()
@@ -316,7 +311,6 @@ def test_no_abstain_state_comes_from_an_adjacent_source(self):
)
def test_a_source_adjacent_to_everything_still_yields_an_example(self):
- """Degrading to "any other source" beats emitting nothing."""
sources = ["a", "b"]
loaders = {
s: (lambda s=s: [traceable_example(s, 0, i) for i in range(10)]) for s in sources
@@ -333,7 +327,6 @@ def test_a_source_adjacent_to_everything_still_yields_an_example(self):
class TestModeBPrompt:
def test_mode_b_does_not_emit_an_empty_option_header(self):
- """`Options:` with nothing under it costs a header and informs nothing."""
choice = build_prompt(
"s", "q", Choice(instructions="pick", criteria={"a": None, "b": None}), None
).full
@@ -354,13 +347,13 @@ def test_mode_a_still_lists_every_option_with_its_code(self):
class TestReaders:
- """Per-row readers are pure functions; drive them with the real column shapes."""
-
def test_race_maps_the_answer_letter_to_an_index(self):
from lev.data.sources import _read_race
row = {"article": "A.", "question": "Q?", "options": ["w", "x", "y", "z"], "answer": "C"}
- state, options, target = _read_race(row, random.Random(0))
+ result = _read_race(row, random.Random(0))
+ assert result is not None
+ state, options, target = result
assert options[target] == "y"
assert state == {"passage": "A.", "question": "Q?"}
@@ -372,7 +365,9 @@ def test_keyed_choices_match_arc_numeric_keys(self):
"choices": {"label": ["1", "2", "3"], "text": ["a", "b", "c"]},
"answerKey": "3",
}
- _, options, target = _read_keyed_choices("question")(row, random.Random(0))
+ result = _read_keyed_choices("question")(row, random.Random(0))
+ assert result is not None
+ _, options, target = result
assert options[target] == "c"
def test_keyed_choices_attach_context_when_present(self):
@@ -384,7 +379,9 @@ def test_keyed_choices_attach_context_when_present(self):
"choices": {"label": ["A", "B"], "text": ["a", "b"]},
"answerKey": "B",
}
- state, _, _ = _read_keyed_choices("question_stem", "fact1")(row, random.Random(0))
+ result = _read_keyed_choices("question_stem", "fact1")(row, random.Random(0))
+ assert result is not None
+ state, _, _ = result
assert state == {"question": "Q?", "context": "F."}
def test_sciq_shuffles_but_keeps_the_gold_index_right(self):
@@ -400,7 +397,9 @@ def test_sciq_shuffles_but_keeps_the_gold_index_right(self):
}
seen_first = set()
for seed in range(20):
- _, options, target = _read_sciq(row, random.Random(seed))
+ result = _read_sciq(row, random.Random(seed))
+ assert result is not None
+ _, options, target = result
assert options[target] == "gold"
seen_first.add(options[0])
assert len(seen_first) > 1, "the gold answer must not always sit first"
@@ -422,16 +421,18 @@ def test_nli_skips_rows_without_gold(self):
from lev.data.sources import _read_nli
assert _read_nli({"premise": "p", "hypothesis": "h", "label": -1}, random.Random(0)) is None
- _, options, target = _read_nli(
- {"premise": "p", "hypothesis": "h", "label": 2}, random.Random(0)
- )
+ result = _read_nli({"premise": "p", "hypothesis": "h", "label": 2}, random.Random(0))
+ assert result is not None
+ _, options, target = result
assert options is None and target == 2
def test_toxigen_uses_the_dataset_threshold(self):
from lev.data.sources import _read_toxigen
- assert _read_toxigen({"text": "t", "toxicity_human": 2.9}, random.Random(0))[2] == 0
- assert _read_toxigen({"text": "t", "toxicity_human": 3.0}, random.Random(0))[2] == 1
+ for toxicity, expected in [(2.9, 0), (3.0, 1)]:
+ result = _read_toxigen({"text": "t", "toxicity_human": toxicity}, random.Random(0))
+ assert result is not None
+ assert result[2] == expected
assert _read_toxigen({"text": "t", "toxicity_human": None}, random.Random(0)) is None
def test_beavertails_yes_means_safe(self):
@@ -453,15 +454,16 @@ def test_ultrafeedback_flattens_completions_and_drops_bad_ratings(self):
],
}
rows = _read_ultrafeedback(row, random.Random(0))
+ assert rows is not None
assert [(r[0]["response"], r[2]) for r in rows] == [("a", 4), ("c", 0)]
def test_snips_unknown_category_is_skipped(self):
from lev.data.sources import SNIPS_INTENTS, _read_snips
assert _read_snips({"text": "t", "category": "Nope"}, random.Random(0)) is None
- assert _read_snips({"text": "t", "category": "GetWeather"}, random.Random(0))[2] == (
- SNIPS_INTENTS.index("GetWeather")
- )
+ result = _read_snips({"text": "t", "category": "GetWeather"}, random.Random(0))
+ assert result is not None
+ assert result[2] == SNIPS_INTENTS.index("GetWeather")
def test_humanise_splits_camel_case(self):
assert humanise("SearchScreeningEvent") == "Search Screening Event"
@@ -481,10 +483,12 @@ def test_per_row_options_become_per_row_questions(self):
]
examples = list(load_source(REGISTRY["race"], load_dataset=fixed_loader(rows)))
assert len(examples) == 8
- assert all(
- isinstance(e.question, Choice) and len(e.question.criteria) == 4 for e in examples
- )
- assert [list(e.question.criteria)[e.target] for e in examples] == ["w", "x", "y", "z"] * 2
+ answers = []
+ for e in examples:
+ assert isinstance(e.question, Choice)
+ assert len(e.question.criteria) == 4
+ answers.append(list(e.question.criteria)[e.target])
+ assert answers == ["w", "x", "y", "z"] * 2
def test_noul_reader_targets_land_on_the_rating_ends(self):
rows = [{"text1": "a", "text2": "b", "label": 1}, {"text1": "c", "text2": "d", "label": 0}]
@@ -493,8 +497,7 @@ def test_noul_reader_targets_land_on_the_rating_ends(self):
assert examples[0].state == {"sentence1": "a", "sentence2": "b"}
def test_every_noul_source_carries_a_negation(self):
- """The aegis2 failure: Noul learned yes = good because no training
- question ever made yes the bad outcome."""
+ """Both polarities prevent yes from always meaning a good outcome."""
for name, spec in REGISTRY.items():
if spec.primitive == "noul":
assert spec.negations, f"{name} has no negated question"
@@ -617,7 +620,9 @@ def test_nli_fever_uses_the_string_label_not_the_integer(self):
"fever_gold_label": "REFUTES",
"label": 2,
}
- state, options, target = _read_nli_fever(row, random.Random(0))
+ result = _read_nli_fever(row, random.Random(0))
+ assert result is not None
+ state, options, target = result
assert state == {"claim": "claim", "evidence": "evidence"}
assert options is None and FEVER_LABELS[target] == "REFUTES"
assert _read_nli_fever({**row, "fever_gold_label": "weird"}, random.Random(0)) is None
@@ -665,6 +670,7 @@ def test_yes_no_wrapper_emits_one_true_and_one_false_per_row(self):
row = {"article": "A.", "question": "Q?", "options": ["w", "x", "y", "z"], "answer": "C"}
rows = yes_no_from_choices(_read_race)(row, random.Random(0))
+ assert rows is not None
assert [r[2] for r in rows] == [1, 0]
assert rows[0][0]["proposed_answer"] == "y" and rows[1][0]["proposed_answer"] != "y"
assert rows[0][0]["passage"] == "A." and rows[0][1] is None
@@ -680,7 +686,10 @@ def test_large_taxonomy_rows_split_between_full_and_cut_sets(self):
subsample_fraction=1.0,
augment={"big": Augment(min_options=15, keep_full_fraction=0.5)},
)
- sizes = [len(e.question.criteria) for e in build_mixture(spec, {"big": lambda: pool})]
+ sizes = []
+ for e in build_mixture(spec, {"big": lambda: pool}):
+ assert isinstance(e.question, Choice)
+ sizes.append(len(e.question.criteria))
full = sum(s == 40 for s in sizes)
assert 120 < full < 280, f"about half should keep the full set, got {full}/400"
assert min(sizes) >= 15, "a cut set fell below min_options"
@@ -713,8 +722,7 @@ def build(self, out):
)
def test_every_split_draws_from_every_source(self, tmp_path):
- """The loaders are built in a loop; a late-bound closure would make every
- split draw from the last source while coverage and round-trip checks pass."""
+ """Late-bound loader closures can cross source contents while preserving source labels."""
from lev.data.build import SPLIT_FILES, read_jsonl
manifest = self.build(tmp_path)
@@ -722,10 +730,9 @@ def test_every_split_draws_from_every_source(self, tmp_path):
for split, filename in SPLIT_FILES.items():
rows = read_jsonl(tmp_path / filename)
assert {e.source for e in rows} == {"ag_news", "imdb"}, split.value
- # A row is labelled with the pool it was drawn from, so a crossed
- # loader leaves the labels right and the content wrong. Check both.
for e in rows:
if not e.abstain:
+ assert isinstance(e.state, str)
assert e.state.startswith(prefix[e.source]), (
f"{split.value}: {e.source} row holds {e.state!r}"
)
@@ -733,8 +740,6 @@ def test_every_split_draws_from_every_source(self, tmp_path):
assert set(manifest["sources"]) == {"ag_news", "imdb"}
def test_held_out_splits_keep_the_canonical_question(self, tmp_path):
- """The test split keeps each source's canonical wording, since an exported
- eval needs one question per source."""
from lev.data.build import SPLIT_FILES, read_jsonl
self.build(tmp_path)
diff --git a/packages/lev/tests/test_engine_forward.py b/packages/lev/tests/test_engine_forward.py
index b2d9dcf..e362ec7 100644
--- a/packages/lev/tests/test_engine_forward.py
+++ b/packages/lev/tests/test_engine_forward.py
@@ -76,5 +76,6 @@ def test_without_score_averaging_a_large_request_answers_as_with_no_limit(tiny_m
unlimited = DecisionEngine(tiny_model, TinyTokenizer(), EngineConfig(max_batch_tokens=None))
state = "a long support message " * 16
prepared = limited.prepare(state, questions)
+ assert limited.config.max_batch_tokens is not None
assert prepared.rows == 80 and prepared.rows * prepared.width > limited.config.max_batch_tokens
assert limited.system_one(state, questions) == unlimited.system_one(state, questions)
diff --git a/packages/lev/tests/test_eval_export.py b/packages/lev/tests/test_eval_export.py
index 06cab7e..867be83 100644
--- a/packages/lev/tests/test_eval_export.py
+++ b/packages/lev/tests/test_eval_export.py
@@ -1,4 +1,4 @@
-"""The lev -> levbench handoff: a file, because levbench must not import lev."""
+"""Test the task-file contract between lev and levbench."""
from __future__ import annotations
@@ -36,8 +36,7 @@ def write_test_split(directory, rows):
class TestRoundTrip:
def test_same_options_in_a_different_order_are_different_questions(self, tmp_path):
- """A question cache keyed on sorted payloads moved shuffled rows' targets onto wrong
- options (ADR-024)."""
+ """Sorting cache keys changes what positional targets refer to (ADR-024)."""
forward = Choice(instructions="q", criteria={"a": None, "b": None, "c": None})
backward = Choice(instructions="q", criteria={"c": None, "b": None, "a": None})
rows = [an_example(forward, target=0, state="x"), an_example(backward, target=0, state="y")]
@@ -45,13 +44,14 @@ def test_same_options_in_a_different_order_are_different_questions(self, tmp_pat
write_jsonl(path, rows)
back = read_jsonl(path)
+ assert isinstance(back[0].question, Choice)
+ assert isinstance(back[1].question, Choice)
assert list(back[0].question.criteria)[back[0].target] == "a"
assert list(back[1].question.criteria)[back[1].target] == "c"
assert back[0].question is not back[1].question
def test_round_trip_guard_catches_an_order_blind_reader(self, tmp_path, monkeypatch):
- """Re-introduce the ADR-024 bug -- a question cache keyed on sorted
- payloads -- and the build-time guard must refuse the file and name the row."""
+ """Inject an order-blind cache and require the guard to reject it."""
import lev.data.build as build
from lev.data.build import verify_round_trip
@@ -108,7 +108,6 @@ def test_score_truth_is_the_level_index(self):
assert truth_for(SCORE, 2) == 2
def test_noul_truth_collapses_the_rating_the_way_the_readout_will(self):
- """Stored as a rating 0-8, compared as a bool. Both ends must map."""
assert truth_for(NOUL, 8) is True
assert truth_for(NOUL, 0) is False
@@ -126,7 +125,6 @@ def test_one_file_per_source_plus_an_index(self, tmp_path):
assert json.loads((out / "index.json").read_text())["total_items"] == 160
def test_levbench_can_read_what_was_written(self, tmp_path):
- """The actual contract. levbench never imports lev; this file is the seam."""
pytest.importorskip("levbench")
from levbench.tasks import load_task_file
@@ -141,7 +139,6 @@ def test_levbench_can_read_what_was_written(self, tmp_path):
assert items[0].labels["a"] in questions["a"].criteria
def test_abstain_rows_are_excluded(self, tmp_path):
- """Scoring them measures abstention, not accuracy."""
rows = [example(CHOICE, i % 2, i, source="a") for i in range(MIN_USEFUL_ITEMS + 10)]
rows += [example(CHOICE, 0, 999, source="a", abstain=True) for _ in range(20)]
out = tmp_path / "eval"
@@ -149,7 +146,6 @@ def test_abstain_rows_are_excluded(self, tmp_path):
assert index["total_items"] == MIN_USEFUL_ITEMS + 10
def test_an_eval_too_small_to_measure_anything_raises(self, tmp_path):
- """At n=24 the interval is +/-16 points. That is not a measurement."""
rows = [example(CHOICE, i % 2, i, source="a") for i in range(24)]
with pytest.raises(ValueError, match="cannot tell you"):
export(write_test_split(tmp_path / "mix", rows), tmp_path / "eval")
@@ -157,8 +153,7 @@ def test_an_eval_too_small_to_measure_anything_raises(self, tmp_path):
class TestVariableQuestions:
def test_sources_whose_question_varies_per_row_are_skipped_and_listed(self, tmp_path):
- """A task file has one question for all its items, so per-row options must not be
- silently written under the first row's option set."""
+ """One task file cannot represent different option sets per row."""
rows = [example(CHOICE, i % 2, i, source="a") for i in range(MIN_USEFUL_ITEMS + 10)]
rows += [
example(
diff --git a/packages/lev/tests/test_evaluate.py b/packages/lev/tests/test_evaluate.py
index 4e5a877..0d65d51 100644
--- a/packages/lev/tests/test_evaluate.py
+++ b/packages/lev/tests/test_evaluate.py
@@ -29,8 +29,6 @@ def test_perfect_predictions_score_accuracy_one(self):
class TestAccuracyInterval:
- """The report must state its own resolution: the eval is not reproducible."""
-
def test_interval_shrinks_with_n(self):
assert accuracy_interval(0.86, 200) > accuracy_interval(0.86, 1800)
@@ -38,7 +36,7 @@ def test_matches_the_normal_approximation(self):
assert accuracy_interval(0.86, 200) == pytest.approx(0.048, abs=0.001)
def test_a_certain_estimate_still_reports_a_finite_interval(self):
- """p=1.0 gives zero variance; the formula must not claim infinite precision."""
+ """Perfect observed accuracy does not imply zero uncertainty."""
assert 0.0 <= accuracy_interval(1.0, 200) < 0.01
def test_no_items_means_no_information(self):
diff --git a/packages/lev/tests/test_loss.py b/packages/lev/tests/test_loss.py
index d73cd82..d0669c8 100644
--- a/packages/lev/tests/test_loss.py
+++ b/packages/lev/tests/test_loss.py
@@ -1,4 +1,4 @@
-"""torch is in the `train` extra, so these skip on a bare `uv sync`."""
+"""Decision-loss tests; skip when Torch is unavailable."""
from __future__ import annotations
@@ -14,8 +14,7 @@
class TestDecisionLoss:
def test_padded_slots_do_not_produce_nan(self, train_config):
- """The first smoke run's bug: a padded slot has logit -inf and target 0,
- so a plain product is `0 * -inf` = NaN, poisoning the batch mean."""
+ """Padded slots must avoid 0 * -inf, which produces NaN."""
logits = torch.tensor(
[[1.0, 2.0, 0.5, 0.1], [0.3, 1.2, float("-inf"), float("-inf")]],
requires_grad=True,
@@ -26,7 +25,6 @@ def test_padded_slots_do_not_produce_nan(self, train_config):
assert torch.isfinite(logits.grad).all()
def test_matches_cross_entropy_when_only_ce_is_enabled(self, train_config):
- """Supporting soft targets must not change the hard-target answer."""
import torch.nn.functional as F
train_config.brier_weight = 0.0
@@ -40,7 +38,6 @@ def test_matches_cross_entropy_when_only_ce_is_enabled(self, train_config):
@pytest.mark.parametrize("weight", [0.5, 1.0])
def test_brier_adds_the_weighted_multiclass_brier_score(self, train_config, weight):
- """A padded slot (probability 0, target 0) must add nothing to the Brier term."""
logits = torch.tensor(
[[8.0, 0.0, 0.0], [2.0, 0.0, 0.0], [0.0, 3.0, float("-inf")]],
)
@@ -88,7 +85,6 @@ def test_ordinal_term_prefers_a_near_miss(self, train_config):
assert near < far, "an ordered scale must not treat every wrong level alike"
def test_ordinal_applies_only_to_the_rows_flagged(self, train_config):
- """`ordinal` is per-row: a Score and a Choice can share a batch."""
train_config.ordinal_weight = 5.0
logits = torch.randn(4, 5)
targets = torch.tensor([0, 1, 2, 3])
@@ -103,14 +99,11 @@ def test_ordinal_applies_only_to_the_rows_flagged(self, train_config):
class TestOrdinalCoverage:
def test_noul_rows_are_flagged_ordinal(self, batching_tokenizer):
- """The scale runs "0 = certainly no" to "8 = certainly yes": without the
- ordinal term, rating 4 is as wrong as rating 0 when the truth is 8."""
collator = DecisionCollator(batching_tokenizer, max_seq_len=512)
batch = collator([an_example(Noul(instructions="urgent?"), target=8)])
assert batch.ordinal.tolist() == [True]
def test_choice_rows_are_not_flagged_ordinal(self, batching_tokenizer):
- """Choice options are unordered symbols; a distance term is meaningless."""
collator = DecisionCollator(batching_tokenizer, max_seq_len=512)
assert collator([an_example(a_choice(4))]).ordinal.tolist() == [False]
diff --git a/packages/lev/tests/test_packaging.py b/packages/lev/tests/test_packaging.py
index c412472..13f14e9 100644
--- a/packages/lev/tests/test_packaging.py
+++ b/packages/lev/tests/test_packaging.py
@@ -1,4 +1,4 @@
-"""Docs run commands from the repo root, so an extra defined only on a member is unreachable."""
+"""Test workspace extras and documented commands from the repository root."""
from __future__ import annotations
@@ -58,8 +58,7 @@ def test_known_extras_are_present(expected):
def _inspect_cli(tool: str) -> tuple[set[str], dict[str, set[str]]]:
- """Introspects the real parser rather than regexing the source, so it checks what the
- parser actually accepts."""
+ """Inspect the live parser to collect accepted commands and flags."""
import argparse
import importlib
@@ -117,8 +116,7 @@ def test_every_documented_backend_exists():
def _documented_commands(tool: str) -> dict[str, set[str]]:
- """Only fenced blocks and inline backticks count; prose like "levbench must not import
- lev" would otherwise match."""
+ """Extract commands from fenced blocks and inline code, excluding prose."""
found: dict[str, set[str]] = {}
sources = [ROOT / "README.md", ROOT / "Makefile", ROOT / "CONTRIBUTING.md"]
sources += sorted((ROOT / "docs").glob("*.md"))
@@ -127,8 +125,7 @@ def _documented_commands(tool: str) -> dict[str, set[str]]:
if not src.is_file():
continue
text = src.read_text()
- # A Makefile is all code apart from its `## help text`, which is prose
- # and mentions commands the way a sentence does.
+ # Exclude Makefile help comments from command extraction.
code = (
[re.sub(r"##.*", "", text)]
if src.suffix != ".md"
diff --git a/packages/lev/tests/test_presentation.py b/packages/lev/tests/test_presentation.py
index 9f58e30..830f091 100644
--- a/packages/lev/tests/test_presentation.py
+++ b/packages/lev/tests/test_presentation.py
@@ -1,10 +1,4 @@
-"""Presentation checks (ADR-029), opt-in Score order averaging and the request row limit.
-
-The checks run against `FakeEngine`: a real `DecisionEngine` without weights routes,
-renders and counts every request, as serving does, and only the logits are scripted.
-So a probe lev would reject, a row count that disagrees with the engine, or an answer
-shape the checks misread fails here without weights.
-"""
+"""Test presentation probes using real routing and rendering with scripted logits."""
from __future__ import annotations
@@ -56,10 +50,7 @@ def encode(self, text: str, add_special_tokens: bool = True) -> list[int]:
class FakeEngine:
- """logit = content[text] + key_pull[key] + slot0 (in slot 0) + a packing term that
- grows with the questions per call, read in every order the engine renders and
- averaged as lev does. Routing, orders, row counts and the row limit are the real
- engine's."""
+ """Use real routing and orders with scripted content, key, slot, and packing effects."""
def __init__(self, slot0=0.0, content=None, packing=0.0, key_pull=None, **config):
self.real = DecisionEngine(model=None, tokenizer=Tokenizer(), config=EngineConfig(**config))
@@ -86,6 +77,7 @@ def system_one(self, state, questions):
rows: dict[str, list] = {}
for variant in prepared.variants:
q = prepared.questions[variant.name]
+ assert isinstance(q, Choice | Score)
items = list(q.criteria.items()) if q.type == "choice" else list(enumerate(q.criteria))
shown = items if variant.order is None else [items[i] for i in variant.order]
z = [
@@ -98,6 +90,7 @@ def system_one(self, state, questions):
rows.setdefault(variant.name, []).append((softmax(z), variant.order))
answers = {}
for name, q in prepared.questions.items():
+ assert isinstance(q, Choice | Score)
p = average_orders([r[0] for r in rows[name]], [r[1] for r in rows[name]])
if q.type == "score":
answers[name] = ScoreAnswer(
@@ -186,7 +179,7 @@ def test_a_planted_slot_zero_penalty_is_measured_exactly(self):
assert r[kind]["first_slot"]["decisions"] == 18
def test_ties_are_split_between_the_tied_slots(self):
- """A flat readout ties all three slots; argmax would credit slot 0 every time."""
+ """Argmax would incorrectly credit slot zero for every flat distribution."""
r = run(FakeEngine(), ["a", "b"])
for kind in KINDS:
assert r[kind]["first_slot"]["metric"] == pytest.approx(1 / 3)
@@ -203,7 +196,7 @@ def test_a_choice_key_pull_cancels_and_a_position_effect_does_not(self):
@pytest.mark.parametrize("mode", ["reversed", "cyclic"])
def test_the_identical_check_reads_one_order_whatever_the_engine_averages(self, mode):
- """Averaged, a pure slot bias cancels (cyclic) or hides the middle slots (reversed)."""
+ """Order averaging hides the raw position bias this probe measures."""
single = run(FakeEngine(slot0=-2.0, order_average=False), ["a"])
averaged = run(FakeEngine(slot0=-2.0, score_order_average=mode), ["a"])
for kind in KINDS:
@@ -223,8 +216,6 @@ def test_the_gate_is_the_proposal_and_can_be_moved(self):
class TestCallGroups:
- """Calls hold at most `max_rows` rows, counted by the engine for every question."""
-
@pytest.mark.parametrize("mode", ["off", "reversed", "cyclic"])
def test_every_call_fits_the_limit_and_asks_everything_once(self, mode):
engine = FakeEngine(score_order_average=mode)
@@ -244,11 +235,13 @@ def test_the_grouping_does_not_change_an_isolated_readout(self):
engine = FakeEngine(slot0=-1.0, content={"Soon": 1.0}, score_order_average="cyclic")
assert run(engine, ["a", "b"], max_rows=5) == run(engine, ["a", "b"])
- def test_identical_and_permuted_questions_never_share_a_call(self):
+ def test_identical_and_permuted_questions_never_share_a_call(self, monkeypatch):
engine = FakeEngine()
seen = []
real = engine.system_one
- engine.system_one = lambda state, qs: seen.append(set(qs)) or real(state, qs)
+ monkeypatch.setattr(
+ engine, "system_one", lambda state, qs: seen.append(set(qs)) or real(state, qs)
+ )
ask(engine, ["a"])
for qids in seen:
assert len({q.split("|")[1] for q in qids}) == 1
@@ -303,8 +296,6 @@ def test_the_cli_is_registered(self):
class TestScoreOrderAverage:
- """Off by default: a Score keeps its level order unless the config opts in."""
-
def orders(self, question, tokenizer, **config):
engine = DecisionEngine(model=None, tokenizer=tokenizer, config=EngineConfig(**config))
return engine._orders(question, route(question, tokenizer))
@@ -349,7 +340,6 @@ def test_schema_first_never_doubles_the_prefix(self, rich_tokenizer):
) == [None]
def test_cyclic_averaging_cancels_a_slot_bias_on_equal_content(self):
- """Each level sits in each slot once, so a pure position bias averages out."""
k = 4
orders = [None] + [[(j + r) % k for j in range(k)] for r in range(1, k)]
biased = softmax([2.0, 0.0, 0.0, 0.0])
@@ -365,8 +355,6 @@ def test_cyclic_averaging_cancels_a_slot_bias_on_equal_content(self):
class TestQuestionRows:
- """`question_rows` is what `prepare` renders, so callers never recount rows."""
-
@pytest.mark.parametrize("mode", ["off", "reversed", "cyclic"])
@pytest.mark.parametrize("order_average", [True, False])
@pytest.mark.parametrize("noul_readout", ["rating", "binary"])
@@ -408,8 +396,6 @@ def test_schema_first_is_one_row_per_question(self, rich_tokenizer):
class TestAveragedTokenLimit:
- """Only a request Score averaging grew can be refused, and only past the batch budget."""
-
def engine(self, tokenizer, **config) -> DecisionEngine:
return DecisionEngine(model=None, tokenizer=tokenizer, config=EngineConfig(**config))
@@ -424,7 +410,7 @@ def test_a_long_state_with_few_averaged_rows_is_refused(self, rich_tokenizer):
assert self.engine(rich_tokenizer).prepare(LONG, question).rows == 1
def test_without_score_averaging_a_large_request_is_never_refused(self, rich_tokenizer):
- """#4's batcher runs an oversized request alone and unsplit, so main accepts it."""
+ """The batcher runs oversized requests alone."""
prepared = self.engine(rich_tokenizer).prepare("x" * 400, BIG)
assert prepared.rows == 80 and prepared.rows * prepared.width > MAX_BATCH_TOKENS
assert [v.order for v in prepared.variants] == [None, [3, 2, 1, 0]] * 20 + [None] * 40
@@ -486,8 +472,6 @@ def test_the_server_answers_422(self, monkeypatch, rich_tokenizer):
class TestCommittedReport:
- """`data/presentation-checks.json` is what `--compare` reads in CI."""
-
def test_it_parses_and_carries_every_metric(self):
path = Path(__file__).resolve().parents[3] / "data" / "presentation-checks.json"
@@ -502,7 +486,7 @@ def test_it_parses_and_carries_every_metric(self):
class TestCompareGuardsItsSettings:
- """`--compare` compares like with like: settings first (exit 2), then metrics (exit 1)."""
+ """Settings mismatches exit 2; metric failures exit 1."""
def cli(self, monkeypatch, tmp_path, engine, *extra):
from lev.cli import main
diff --git a/packages/lev/tests/test_progress.py b/packages/lev/tests/test_progress.py
index 97e2e8f..e57879b 100644
--- a/packages/lev/tests/test_progress.py
+++ b/packages/lev/tests/test_progress.py
@@ -1,4 +1,4 @@
-"""torch is in the `train` extra, so these skip on a bare `uv sync`."""
+"""Progress-reporting tests; skip when Torch is unavailable."""
from __future__ import annotations
@@ -13,8 +13,6 @@
class TestProgressLog:
- """A silent run is indistinguishable from a hung one."""
-
def make(self, total=100, every=10):
return ProgressLog(total, every)
@@ -27,14 +25,13 @@ def test_logs_only_on_the_interval(self, capsys):
assert "step 10/100" in capsys.readouterr().out
def test_always_logs_the_final_step(self, capsys):
- """An interval that does not divide the total must not swallow the end."""
log = self.make(total=7, every=10)
for step in range(7):
log.record(step, 1.0, "A", 1e-4, tokens=10)
assert "step 7/7" in capsys.readouterr().out
def test_modes_are_reported_separately(self, capsys):
- """Mode B starts near ln(K); blending it with Mode A hides both."""
+ """Mode B starts near ln(K), so averaging modes hides their separate losses."""
log = self.make(total=2, every=2)
log.record(0, 1.0, "A", 1e-4, tokens=10)
log.record(1, 5.0, "B", 1e-4, tokens=10)
@@ -52,13 +49,10 @@ def test_window_resets_between_reports(self, capsys):
assert "A=1.0000" in capsys.readouterr().out, "stale window carried forward"
def test_throughput_counts_real_tokens(self, capsys):
- """Counted from the attention mask, not the config's average, which was once wrong by 10x
- (ADR-016) and would have hidden the mistake."""
log = self.make(total=1, every=1)
log.record(0, 1.0, "A", 1e-4, tokens=4096)
out = capsys.readouterr().out
- # Parse the value rather than substring-match it: a fast window makes
- # the rate large, and "4096000000 tok/s" contains "0 tok/s".
+ # Parse the rate so large values cannot falsely match a zero-rate substring.
match = re.search(r"([\d,]+) tok/s", out)
assert match, out
assert int(match.group(1).replace(",", "")) > 0
@@ -75,8 +69,7 @@ def test_hms_formats_hours(self):
class TestWindowedRate:
- """A cumulative rate never stops paying the startup cost: the 0.8B smoke read 0.33 it/s
- against a true 2.50, and the ETA was 7.5x off (ADR-017)."""
+ """Windowed rates exclude startup delay from steady-state throughput and ETA."""
@pytest.fixture
def clock(self, monkeypatch):
@@ -100,7 +93,9 @@ def test_rate_reflects_the_window_not_the_startup_stall(self, capsys, clock):
second = capsys.readouterr().out
def rate(out):
- return float(re.search(r"([\d.]+) it/s", out).group(1))
+ match = re.search(r"([\d.]+) it/s", out)
+ assert match is not None, out
+ return float(match.group(1))
assert rate(second) == pytest.approx(2.5, abs=0.05), second
assert rate(second) > 5 * rate(first), f"{rate(first)} -> {rate(second)}"
diff --git a/packages/lev/tests/test_release.py b/packages/lev/tests/test_release.py
index 975bebc..2b890dc 100644
--- a/packages/lev/tests/test_release.py
+++ b/packages/lev/tests/test_release.py
@@ -10,7 +10,7 @@
def fake_checkpoint(root, step=18750, calibrated=True):
- """The files `save_checkpoint` and `calibrate` leave behind, contents faked."""
+ """Create placeholder files matching a saved checkpoint and calibration."""
out = root / "4b"
step_dir = out / f"step-{step}"
step_dir.mkdir(parents=True)
@@ -97,8 +97,7 @@ def test_a_missing_path_that_is_not_a_hub_id_fails_clearly(self, spec):
fetch_checkpoint(spec)
def test_a_mistyped_local_path_that_looks_like_a_hub_id_fails_clearly(self, monkeypatch):
- """`checkpoints/lev-instuct` has one `/`, so it is tried on the Hub; a
- missing repo must read as a missing path, not a raw Hub 404."""
+ """A missing relative path may also parse as a Hub id."""
# Both ship with the `train` extra, absent on a bare `uv sync`.
huggingface_hub = pytest.importorskip("huggingface_hub")
httpx = pytest.importorskip("httpx")
@@ -115,7 +114,6 @@ def not_on_the_hub(repo, **kwargs):
def test_model_card_describes_routing_as_it_is_served_and_trained(tmp_path):
- """The card ships with the weights, so it must not carry a stale `27+` Mode B threshold."""
out = fake_checkpoint(tmp_path)
build_release(out, tmp_path / "release", preset="4b")
card = (tmp_path / "release" / MODEL_CARD).read_text()
diff --git a/packages/lev/tests/test_s1bench.py b/packages/lev/tests/test_s1bench.py
index 267a744..bd7eb1b 100644
--- a/packages/lev/tests/test_s1bench.py
+++ b/packages/lev/tests/test_s1bench.py
@@ -1,4 +1,4 @@
-"""The eval door opens only onto blocked subsets and has no route into the training mixture."""
+"""Test that S1Bench evaluation loading remains isolated from training."""
from __future__ import annotations
@@ -22,7 +22,6 @@ def test_every_blocked_subset_still_raises_on_the_training_path(self, subset: st
assert_clean([subset])
def test_adding_an_eval_door_did_not_unblock_the_hf_ids_it_reads(self) -> None:
- """The loader reads `tals/vitaminc`; the mixture still must not."""
for spec in EVAL_SUBSETS.values():
with pytest.raises(ContaminationError):
assert_clean([spec.hf_id])
@@ -41,15 +40,13 @@ def test_aliases_resolve_to_their_subset(self, alias: str) -> None:
@pytest.mark.parametrize("name", ["ag_news", "imdb", "banking77", "some/private-corpus"])
def test_training_sources_are_refused(self, name: str) -> None:
- """The door must not become a general-purpose dataset loader."""
with pytest.raises(ContaminationError):
get_subset(name)
class TestStructuralSeparation:
def test_s1bench_cannot_reach_the_mixture_builder(self) -> None:
- """Importing `mixture` would give the eval loaders a type the training path consumes.
- A fresh interpreter, since in-process the suite's own imports are visible."""
+ """Use a fresh interpreter so existing test-suite imports do not mask dependencies."""
probe = (
"import sys; import lev.data.s1bench; "
"leaked = sorted(m for m in sys.modules "
@@ -103,15 +100,12 @@ def test_definition_is_self_consistent_and_names_its_source(self, name: str) ->
@pytest.mark.parametrize("name", sorted(EVAL_SUBSETS))
def test_record_counts_match_the_board(self, name: str) -> None:
- """helpsteer2 is one record short of the board's 250: the manifest's
- token-length filter dropped rows."""
+ """The pinned helpsteer2 filter yields 249 records, one fewer than the board."""
board = jev_board(snapshot())[name]
expected = board["total"] - (1 if name == "helpsteer2" else 0)
assert definition(name)["count"] == expected
def test_jev_numbers_match_the_snapshot(self) -> None:
- """Read from `data/s1bench-snapshot.json` rather than restated, so a
- regenerated snapshot fails here."""
data = snapshot()
board = jev_board(data)
for name, subset in EVAL_SUBSETS.items():
diff --git a/packages/lev/tests/test_training_loop.py b/packages/lev/tests/test_training_loop.py
index e543079..b2588f0 100644
--- a/packages/lev/tests/test_training_loop.py
+++ b/packages/lev/tests/test_training_loop.py
@@ -1,4 +1,4 @@
-"""torch is in the `train` extra, so these skip on a bare `uv sync`."""
+"""Training-loop tests; skip when Torch is unavailable."""
from __future__ import annotations
@@ -72,8 +72,7 @@ def smoke_config(tmp_path, *, epochs, checkpoint_every):
def run_resumable(
tmp_path, monkeypatch, *, max_steps, checkpoint_every=3, fresh=False, persist=True
):
- """Checkpoints land on disk, so a second call in the same `tmp_path` resumes.
- Returns (saved steps, batches seen, summary)."""
+ """Run with on-disk checkpoints; return saved steps, observed batches, and summary."""
import lev.train.loop as loop
from lev.train.checkpoints import TRAINING_STATE
from lev.train.collate import DecisionCollator
@@ -105,8 +104,7 @@ def fake_save(m, h, tk, out, step, cb=None, state=None):
class TestCheckpointCadence:
- """Only data loading, the backbone, the forward pass and the checkpoint write are stubbed;
- batching, routing, the step counter and checkpoint decisions are the shipping code."""
+ """Exercise real batching and checkpoint decisions with model and storage stubs."""
def run(self, tmp_path, monkeypatch, *, max_steps, checkpoint_every, epochs=1):
import lev.train.loop as loop
@@ -124,13 +122,10 @@ def run(self, tmp_path, monkeypatch, *, max_steps, checkpoint_every, epochs=1):
return saved
def test_the_final_step_is_not_checkpointed_twice(self, tmp_path, monkeypatch):
- """A total that is a multiple of `checkpoint_every` saves once; on Modal a
- duplicate is a second multi-hundred-MB adapter write and Volume commit."""
saved = self.run(tmp_path, monkeypatch, max_steps=12, checkpoint_every=6)
assert saved == [6, 12], f"expected [6, 12], got {saved}"
def test_an_uneven_total_still_saves_at_the_end(self, tmp_path, monkeypatch):
- """Dedup must not swallow the final save when it is not on the interval."""
saved = self.run(tmp_path, monkeypatch, max_steps=10, checkpoint_every=4)
assert saved == [4, 8, 10], f"expected [4, 8, 10], got {saved}"
@@ -140,9 +135,6 @@ def test_checkpointing_off_still_saves_once(self, tmp_path, monkeypatch):
class TestResume:
- """A preempted run continues at the step it stopped, through the batches it had not seen,
- on the schedule it had reached."""
-
def test_resumes_at_the_saved_step_and_sees_only_the_remaining_batches(
self, tmp_path, monkeypatch
):
@@ -163,6 +155,7 @@ def test_saved_state_records_the_schedule_position(self, tmp_path, monkeypatch):
run_resumable(tmp_path, monkeypatch, max_steps=6)
state = load_training_state(tmp_path / "out" / "step-6")
+ assert state is not None
assert (state["step"], state["epoch"], state["step_in_epoch"]) == (6, 0, 6)
assert state["scheduler"]["last_epoch"] == 6
@@ -173,16 +166,14 @@ def test_fresh_ignores_the_checkpoint(self, tmp_path, monkeypatch):
assert summary["history"][0]["step"] == 0
def test_weights_only_checkpoint_restarts_the_schedule(self, tmp_path, monkeypatch):
- """A checkpoint without training state (pre-ADR-021) restarts the step count
- from zero. `load_checkpoint` is stubbed, so weight restore is not checked."""
+ """Weight loading is stubbed; this checks schedule restart only."""
run_resumable(tmp_path, monkeypatch, max_steps=6, persist=False)
saved, _, summary = run_resumable(tmp_path, monkeypatch, max_steps=6)
assert saved == [3, 6]
assert summary["history"][0]["step"] == 0
def test_rerunning_a_finished_run_trains_nothing(self, tmp_path, monkeypatch):
- """7 steps at checkpoint_every=3 ends between saves, so the final checkpoint must carry
- training state or a rerun resumes it as weights-only and trains from step 0."""
+ """An off-interval final checkpoint must carry resume state."""
run_resumable(tmp_path, monkeypatch, max_steps=7)
saved, seen, summary = run_resumable(tmp_path, monkeypatch, max_steps=7)
assert seen == [], "a finished run must not train again"
@@ -198,8 +189,7 @@ def test_a_final_checkpoint_between_saves_resumes_where_it_stopped(self, tmp_pat
assert [h["step"] for h in summary["history"]] == list(range(10))
def test_resume_across_an_epoch_boundary(self, tmp_path, monkeypatch):
- """64 rows at batch 2 is 32 steps per epoch; stopping at 33 is one step
- into epoch 1, and the resumed run must skip exactly that one."""
+ """At batch size two, step 33 is the first step of the second 64-row epoch."""
_, full, _ = run_resumable(tmp_path / "ref", monkeypatch, max_steps=36)
run_resumable(tmp_path, monkeypatch, max_steps=33)
_, seen, summary = run_resumable(tmp_path, monkeypatch, max_steps=36)
@@ -228,8 +218,7 @@ def test_stale_checkpoints_and_calibration_are_moved_not_deleted(self, tmp_path)
assert set_aside_previous_run(out) is None
def test_fresh_run_does_not_resume_from_a_higher_stale_step(self, tmp_path, monkeypatch):
- """The hazard: a fresh run writes step-3, step-6 beside a stale step-18750;
- a preemption's auto-resume would take the stale one. `--fresh` moves it."""
+ """Stale higher-step checkpoints must not survive in the active run directory."""
from lev.train.checkpoints import latest_checkpoint
stale = tmp_path / "out" / "step-18750"
@@ -238,13 +227,13 @@ def test_fresh_run_does_not_resume_from_a_higher_stale_step(self, tmp_path, monk
saved, _, summary = run_resumable(tmp_path, monkeypatch, max_steps=6, fresh=True)
assert saved == [3, 6]
assert summary["history"][0]["step"] == 0
- assert latest_checkpoint(tmp_path / "out").name == "step-6"
+ latest = latest_checkpoint(tmp_path / "out")
+ assert latest is not None and latest.name == "step-6"
class TestModeAProjectsOnlyTheAnswerPositions:
def test_logits_to_keep_selects_distinct_positions_and_rows_read_their_own(self):
- """Mode A must not materialise (B, seq, V): the model is asked for the
- distinct answer positions only, and each row reads its own column."""
+ """Project only answer positions and select the correct output column per row."""
from types import SimpleNamespace
from lev.router import Mode
@@ -256,8 +245,7 @@ class FakeModel:
def __call__(self, input_ids, attention_mask, logits_to_keep):
seen["keep"] = logits_to_keep.tolist()
b, v = input_ids.shape[0], 10
- # logits[row, j, tok] = 100*row + 10*keep[j] + tok, so the value
- # read identifies which row and position it came from.
+ # Encode row and position in each logit to expose indexing errors.
keep = logits_to_keep.float()
rows = torch.arange(b).float()[:, None, None] * 100
pos = keep[None, :, None] * 10
diff --git a/packages/levbench/src/levbench/__init__.py b/packages/levbench/src/levbench/__init__.py
index 6eaf970..42c86e8 100644
--- a/packages/levbench/src/levbench/__init__.py
+++ b/packages/levbench/src/levbench/__init__.py
@@ -1,5 +1 @@
-"""Benchmark harness for any `/v1/systemone` server: Jev, lev, or an LLM baseline.
-
-Import submodules directly (`from levbench import runner, metrics`); the package
-binds nothing at import time, so importing one module does not pull in the rest.
-"""
+"""Benchmark System One servers; import submodules directly."""
diff --git a/packages/levbench/src/levbench/batching.py b/packages/levbench/src/levbench/batching.py
index e80458b..e763022 100644
--- a/packages/levbench/src/levbench/batching.py
+++ b/packages/levbench/src/levbench/batching.py
@@ -1,10 +1,4 @@
-"""The batching sweep: one state, N questions, one call versus N calls.
-
-Measures the central economic claim: input tokens are billed per *request*,
-so reading a large state is amortised across every question asked in the same
-call. If it holds, batched cost stays roughly flat as N grows while split cost
-grows linearly.
-"""
+"""Compare one batched call with separate calls over a shared state."""
from __future__ import annotations
@@ -16,8 +10,7 @@
from . import pricing
from .runner import call_once
-# One shared document; all three primitives, since output-token cost differs
-# between them.
+# Include all three primitives because their output costs differ.
QUESTION_BANK: list[tuple[str, Any]] = [
("mentions_retention", Noul(instructions="The document discusses data retention periods.")),
("mentions_consent", Noul(instructions="The document discusses user consent.")),
@@ -96,17 +89,18 @@ def sweep(
errors: list[tuple[int, str]] | None = None,
) -> list[SweepRow]:
"""Run the sweep, appending any per-count failure to `errors`."""
- bank = bank or QUESTION_BANK
+ bank = QUESTION_BANK if bank is None else bank
rows: list[SweepRow] = []
errors = errors if errors is not None else []
for n in counts:
+ if n < 1:
+ raise ValueError(f"Question count must be positive, got {n}")
if n > len(bank):
raise ValueError(f"Asked for {n} questions but the bank holds {len(bank)}")
selected = dict(bank[:n])
- # The default sweep (1,2,4,8,13) sends 33 sequential requests; each count
- # is isolated so a late rate limit keeps the rows already paid for.
+ # Preserve completed rows if a later count hits an API failure.
try:
batched = call_once(client, state, selected)
batched_cost = pricing.cost_usd(model, batched.input_tokens, batched.output_tokens)
@@ -117,7 +111,7 @@ def sweep(
one = call_once(client, state, {key: question})
split_cost += pricing.cost_usd(model, one.input_tokens, one.output_tokens)
split_seconds += one.seconds
- except Exception as exc: # noqa: BLE001 -- any API failure, report and move on
+ except Exception as exc:
errors.append((n, f"{type(exc).__name__}: {exc}"))
continue
diff --git a/packages/levbench/src/levbench/cli.py b/packages/levbench/src/levbench/cli.py
index 00bc770..4dee4d6 100644
--- a/packages/levbench/src/levbench/cli.py
+++ b/packages/levbench/src/levbench/cli.py
@@ -8,6 +8,7 @@
import argparse
import os
import sys
+from collections.abc import Sequence
from contextlib import nullcontext
from pathlib import Path
@@ -20,8 +21,7 @@
def repo_data_dir() -> Path:
- """Find the repo's `data/` by walking up from this file, so it survives the
- package moving within the repo."""
+ """Find the repository data directory relative to this module."""
for parent in Path(__file__).resolve().parents:
if (candidate := parent / "data").is_dir():
return candidate
@@ -41,9 +41,8 @@ def _require_key(backend: str, base_url: str | None = None) -> None:
sys.exit(f"{needed} is not set.\n{hint}")
-def _task_files(path: str | None) -> list[Path | None]:
- """`None` for the built-in fixture, else one entry per task file; a generated
- eval is a directory with one file per source (ADR-015)."""
+def _task_files(path: str | None) -> Sequence[Path | None]:
+ """Return task files, or None to use the built-in fixture."""
if path is None:
return [None]
target = Path(path)
diff --git a/packages/levbench/src/levbench/confidence_id.py b/packages/levbench/src/levbench/confidence_id.py
index 189ea69..69ebba3 100644
--- a/packages/levbench/src/levbench/confidence_id.py
+++ b/packages/levbench/src/levbench/confidence_id.py
@@ -1,14 +1,9 @@
-"""Identify which statistic the server's `confidence` field actually is.
-
-TypeSafe documents it only as "a statistic computed from the probability distribution".
-Jev's is `norm_max_prob` rounded to 2 dp (docs/FINDINGS.md §3); LitJev states normalized
-Gini, so a LitJev server is a known-answer test.
-"""
+"""Identify a server's confidence formula from its answer distributions."""
from __future__ import annotations
import math
-from collections.abc import Callable
+from collections.abc import Callable, Iterable
from dataclasses import dataclass
from typing import Any
@@ -43,8 +38,7 @@ def top_two_margin(p: Distribution) -> float:
def normalized_max_prob(p: Distribution) -> float:
- """Chance-corrected max probability: a raw 0.5 means something different over
- 2 candidates than over 20."""
+ """Normalize maximum probability relative to uniform chance."""
k = len(p)
if k <= 1:
return 1.0
@@ -69,8 +63,7 @@ def top_two_ratio(p: Distribution) -> float:
def detect_precision(values: list[float], max_decimals: int = 6) -> float:
- """The quantisation step every value is a multiple of, or 0 if none is; the API
- rounds before sending, which sets the floor on any match."""
+ """Find a shared quantization step, or return zero if none matches."""
for decimals in range(1, max_decimals + 1):
unit = 10.0**-decimals
if all(abs(v / unit - round(v / unit)) < 1e-6 for v in values):
@@ -79,8 +72,7 @@ def detect_precision(values: list[float], max_decimals: int = 6) -> float:
def rounding_sensitivity(fn: Callable[[Distribution], float], p: Distribution, eps: float) -> float:
- """How far `fn` can move when each probability is off by up to `eps`: `max_prob`
- passes it through, `norm_max_prob` amplifies it by K/(K-1), `gini` by about 2K/(K-1)."""
+ """Bound how probability rounding affects a confidence formula."""
base = fn(p)
raised = fn([min(1.0, x + eps) for x in p])
lowered = fn([max(0.0, x - eps) for x in p])
@@ -97,12 +89,11 @@ class Fit:
@property
def matches(self) -> bool:
- """Within the API's rounding plus how far it moves this statistic; a flat 5e-3
- rejects Jev's `norm_max_prob`, which amplifies 2 dp rounding to ~0.01 (FINDINGS §3)."""
+ """Allow quantization error amplified by the candidate formula."""
return self.max_abs_error <= self.tolerance
-def collect(answers: list[Any]) -> list[tuple[Distribution, float]]:
+def collect(answers: Iterable[Any]) -> list[tuple[Distribution, float]]:
"""Pull (distribution, reported confidence) from answers that have both."""
samples: list[tuple[Distribution, float]] = []
for answer in answers:
@@ -116,17 +107,15 @@ def collect(answers: list[Any]) -> list[tuple[Distribution, float]]:
def identify(samples: list[tuple[Distribution, float]]) -> list[Fit]:
"""Rank candidate formulas by how closely they reproduce `confidence`."""
+ if not samples:
+ return []
everything = [v for dist, reported in samples for v in (*dist, reported)]
eps = detect_precision(everything) / 2
fits: list[Fit] = []
for name, fn in CANDIDATES.items():
errors = [abs(fn(dist) - reported) for dist, reported in samples]
- if not errors:
- continue
- tolerance = max(
- (eps + rounding_sensitivity(fn, dist, eps) for dist, _ in samples), default=0.0
- )
+ tolerance = max(eps + rounding_sensitivity(fn, dist, eps) for dist, _ in samples)
fits.append(
Fit(
name=name,
@@ -140,8 +129,7 @@ def identify(samples: list[tuple[Distribution, float]]) -> list[Fit]:
def identify_by_size(samples: list[tuple[Distribution, float]]) -> dict[int, list[Fit]]:
- """`identify` per candidate-set size: a formula that holds for one size reads as
- "no match" when pooled, and candidate count tracks question type."""
+ """Fit confidence formulas separately for each candidate-set size."""
by_size: dict[int, list[tuple[Distribution, float]]] = {}
for dist, reported in samples:
by_size.setdefault(len(dist), []).append((dist, reported))
@@ -151,7 +139,7 @@ def identify_by_size(samples: list[tuple[Distribution, float]]) -> dict[int, lis
def worst_residuals(
samples: list[tuple[Distribution, float]], name: str, limit: int = 3
) -> list[tuple[Distribution, float, float]]:
- """The samples a candidate fits worst, as (distribution, reported, predicted)."""
+ """Return the worst fits as (distribution, reported, predicted)."""
fn = CANDIDATES[name]
scored = [(dist, reported, fn(dist)) for dist, reported in samples]
return sorted(scored, key=lambda row: abs(row[2] - row[1]), reverse=True)[:limit]
@@ -184,7 +172,7 @@ def format_fits(fits: list[Fit]) -> str:
def format_diagnosis(samples: list[tuple[Distribution, float]]) -> str:
- """Per-size fits and the worst residuals, for when nothing matches overall."""
+ """Format per-size fits and worst residuals."""
out: list[str] = []
off = [abs(sum(dist) - 1.0) for dist, _ in samples]
diff --git a/packages/levbench/src/levbench/metrics.py b/packages/levbench/src/levbench/metrics.py
index e9de062..372051e 100644
--- a/packages/levbench/src/levbench/metrics.py
+++ b/packages/levbench/src/levbench/metrics.py
@@ -1,9 +1,4 @@
-"""Accuracy and calibration over typed answer distributions.
-
-ECE uses top probability, since servers define their `confidence` fields
-differently (lev: normalized Gini; Jev: chance-corrected max probability). Noul
-is represented as {True: p, False: 1-p} for every backend.
-"""
+"""Score typed distributions; ECE uses top probability, independent of vendor confidence."""
from __future__ import annotations
@@ -11,8 +6,7 @@
from dataclasses import dataclass, field
from typing import Any
-# Log clamp, so a confidently wrong answer scores badly rather than infinitely
-# badly (the scikit-learn convention).
+# Clamp zero probabilities so confidently wrong answers have finite log loss.
_EPS = 1e-15
@@ -34,8 +28,7 @@ def predicted_label(answer: Any) -> Any:
if kind == "noul":
return float(answer.noul) >= 0.5
if kind == "choice":
- # Not a recomputed argmax: they differ on ties, and `choice` is what a
- # caller acts on.
+ # Preserve the model's choice, including its tie-breaking behavior.
return answer.choice
if kind == "score":
return max(answer.probabilities.items(), key=lambda kv: kv[1])[0]
@@ -43,7 +36,7 @@ def predicted_label(answer: Any) -> Any:
def top_probability(answer: Any) -> float:
- """The most likely label's probability (max(p, 1-p) for a Noul): what ECE bins on."""
+ """Return the top-label probability used for ECE."""
return max(to_distribution(answer).values())
@@ -134,10 +127,9 @@ def calibration(
def selective_accuracy(
records: list[tuple[dict[Any, float], Any, Any, float]], threshold: float
) -> tuple[float, float]:
- """Accuracy on answers at or above a confidence threshold, plus the kept fraction;
- if accuracy does not rise with the threshold, confidence is useless for routing."""
+ """Return accuracy and coverage above a confidence threshold."""
kept = [r for r in records if r[3] >= threshold]
if not kept:
return 0.0, 0.0
- acc = sum(1 for _, pred, truth, _ in kept if pred == truth) / len(kept)
+ acc = sum(pred == truth for _, pred, truth, _ in kept) / len(kept)
return acc, len(kept) / len(records)
diff --git a/packages/levbench/src/levbench/pricing.py b/packages/levbench/src/levbench/pricing.py
index 62a7ab2..1d341c6 100644
--- a/packages/levbench/src/levbench/pricing.py
+++ b/packages/levbench/src/levbench/pricing.py
@@ -1,9 +1,4 @@
-"""Per-million-token prices, and cost accounting for a single call.
-
-Jev prices are from typesafe.ai (input $0.042/M, output free). Anthropic prices
-are first-party API rates. Both are vendor list prices and go stale -- override
-via `PRICES` rather than editing call sites.
-"""
+"""Compute call costs from per-million-token list prices; override rates via PRICES."""
from __future__ import annotations
@@ -26,9 +21,7 @@ class Price:
}
-# A self-hosted model costs GPU time, not tokens. An unlisted `jev*` name is
-# priced as jev-latest; any other unlisted name, including an unlisted
-# `claude-*`, counts as self-hosted and costs $0.
+# Unknown jev names use jev-latest rates; other unknown models have zero token cost.
SELF_HOSTED = Price(0.0, 0.0)
diff --git a/packages/levbench/src/levbench/runner.py b/packages/levbench/src/levbench/runner.py
index 2cf3228..a96a302 100644
--- a/packages/levbench/src/levbench/runner.py
+++ b/packages/levbench/src/levbench/runner.py
@@ -1,21 +1,18 @@
-"""Backend-agnostic evaluation runner.
-
-Every backend exposes the same `system_one(state, questions)` call and answer
-types, so the benchmark is written once and only the client changes.
-"""
+"""Evaluate backends through the shared system_one interface."""
from __future__ import annotations
import os
import statistics
import time
+from collections.abc import Sequence
from dataclasses import dataclass, field
from typing import Any
from dotenv import load_dotenv
from . import metrics, pricing
-from .tasks import Item
+from .tasks import FileItem, Item
load_dotenv()
@@ -27,8 +24,7 @@ class CallResult:
served_by: str
input_tokens: int
output_tokens: int
- # Adapter-only retries (schema-invalid LLM output, transient provider errors);
- # always 0 on the TypeSafe SDK, where schema conformance is structural.
+ # Adapter retries; the TypeSafe SDK reports zero.
schema_retries: int = 0
transient_retries: int = 0
@@ -85,11 +81,7 @@ def build_client(
base_url: str | None = None,
timeout: float | None = None,
):
- """Return `(client, model_name)` for jev, lev or anthropic.
-
- Jev and lev share the TypeSafe protocol. lev defaults to localhost, needs no
- credential and uses self-hosted pricing. Anthropic uses the LLM adapter.
- """
+ """Build a Jev, local lev, or Anthropic client and return its model name."""
if backend in ("jev", "lev"):
if backend == "lev":
base_url = base_url or DEFAULT_LOCAL_BASE_URL
@@ -103,8 +95,7 @@ def build_client(
kwargs["timeout"] = DEFAULT_LOCAL_TIMEOUT
if base_url:
kwargs["base_url"] = base_url
- # Never forward the hosted API key to a custom endpoint. Empty local
- # keys also fall back, since an empty Bearer header is invalid.
+ # Use local credentials for custom endpoints; empty keys produce invalid Bearer headers.
kwargs["api_key"] = os.environ.get("LEVBENCH_LOCAL_API_KEY") or "local"
return TypeSafeClient(**kwargs), resolved
@@ -126,8 +117,7 @@ def build_client(
def _token_count(usage: Any, total_field: str, base_field: str) -> int:
- """Prefer the adapter's `*_total` (it includes billed retries) over the SDK's base
- field; a genuine 0 counts, but an absent count raises rather than booking it free."""
+ """Prefer billed retry totals; accept zero counts and reject missing counts."""
for field_name in (total_field, base_field):
value = getattr(usage, field_name, None)
if value is not None:
@@ -167,12 +157,11 @@ def run_eval(
client,
backend: str,
model: str,
- items: list[Item],
+ items: Sequence[Item | FileItem],
questions: dict[str, Any],
concurrency: int = 1,
) -> EvalReport:
- """Score every item, with up to `concurrency` requests in flight and results in
- item order; concurrency lowers wall time, not the per-call latency reported."""
+ """Evaluate concurrently, returning results in item order and per-call latencies."""
from concurrent.futures import ThreadPoolExecutor
report = EvalReport(backend=backend, model=model)
@@ -188,14 +177,16 @@ def run_eval(
for item, result in zip(items, results, strict=True):
report.calls.append(result)
+ labels = item.labels
for name, answer in result.answers.items():
- truth = item.labels[name]
+ truth = labels[name]
+ distribution = metrics.to_distribution(answer)
records[name].append(
(
- metrics.to_distribution(answer),
+ distribution,
metrics.predicted_label(answer),
truth,
- metrics.top_probability(answer),
+ max(distribution.values()),
)
)
@@ -225,8 +216,7 @@ def _format_summary(report: EvalReport) -> list[str]:
if n:
lines.append(f"latency p50 {report.pct(0.5):.3f}s")
lines.append(f"latency mean {statistics.mean(report.latencies):.3f}s")
- # Under 100 calls the p95 falls among the few slowest calls, which are
- # mostly connection setup; report the slowest call instead.
+ # Report the maximum for small samples, where p95 is dominated by connection setup.
if n >= 100:
lines.append(f"latency p95 {report.pct(0.95):.3f}s")
else:
diff --git a/packages/levbench/src/levbench/snake/__init__.py b/packages/levbench/src/levbench/snake/__init__.py
index 762b949..9e3763d 100644
--- a/packages/levbench/src/levbench/snake/__init__.py
+++ b/packages/levbench/src/levbench/snake/__init__.py
@@ -1,9 +1,4 @@
-"""A decision model plays Snake, one `/v1/systemone` call per move.
-
-Ported from laya-mlx (Apache-2.0, github.com/mizorewww/laya-mlx) with its rules,
-planner and prompts verbatim, so runs are comparable (ADR-022). The planner scores
-both Noul answers live; the compact prompt states them (docs/FINDINGS.md §12).
-"""
+"""Snake benchmark ported from laya-mlx (Apache-2.0, github.com/mizorewww/laya-mlx; ADR-022)."""
from .game import DIRECTIONS, SnakeGame
from .policy import Decision, ModelPolicy, PlannerClient
diff --git a/packages/levbench/src/levbench/snake/game.py b/packages/levbench/src/levbench/snake/game.py
index 8c0e8ec..585208c 100644
--- a/packages/levbench/src/levbench/snake/game.py
+++ b/packages/levbench/src/levbench/snake/game.py
@@ -1,9 +1,6 @@
-"""Deterministic Snake rules and a Hamiltonian-cycle safety planner.
+"""Snake rules and cycle-order safety planning, following laya-mlx.
-A safe move preserves cycle order without crossing the tail or skipping food.
-The shield rejects moves outside that policy, including legal moves that do
-not cause an immediate collision. Rules follow laya-mlx so runs are comparable.
-"""
+Safe moves preserve cycle order without crossing the tail or skipping food."""
from __future__ import annotations
@@ -61,7 +58,7 @@ def __init__(self, width: int = 24, height: int = 16, seed: int = 7, initial_len
def head(self) -> tuple[int, int]:
return self.body[0]
- def _spawn_food(self):
+ def _spawn_food(self) -> tuple[int, int] | None:
occupied = set(self.body)
empty = [cell for cell in self.cycle if cell not in occupied]
return self.rng.choice(empty) if empty else None
@@ -84,6 +81,7 @@ def legal_reason(self, direction: str) -> str:
def moves(self) -> list[MoveInfo]:
if not self.alive or self.won:
return []
+ assert self.food is not None, "an unfinished game must have food"
head_index = self.indices[self.head]
tail_distance = (self.indices[self.body[-1]] - head_index) % self.capacity
food_distance = (self.indices[self.food] - head_index) % self.capacity
diff --git a/packages/levbench/src/levbench/snake/policy.py b/packages/levbench/src/levbench/snake/policy.py
index 423dee7..04b5fe4 100644
--- a/packages/levbench/src/levbench/snake/policy.py
+++ b/packages/levbench/src/levbench/snake/policy.py
@@ -1,8 +1,6 @@
-"""Turn a board into one System One call, and the answer into a move.
+"""Turn Snake boards into model decisions using laya-mlx prompts.
-Both prompts are laya-mlx's verbatim. `compact` states the two Noul answers in the state
-text, so a model that reads its input scores 100% on them; `detailed` uses prose.
-"""
+The compact prompt supplies the two Noul truths; the detailed prompt uses prose."""
from __future__ import annotations
@@ -27,8 +25,7 @@ class Decision:
executed: str
safe_directions: list[str]
intervened: bool
- # `dead_end_risk` is 1 - the `risk` answer, which asks whether a safe route exists;
- # the `*_truth` fields are what the planner knows.
+ # risk asks about a safe route; dead_end_risk is its complement. Truth comes from the planner.
dead_end_risk: float
food_reachable: float
route_truth: bool
@@ -46,8 +43,7 @@ def to_dict(self) -> dict:
def describe(game: SnakeGame, prompt: str) -> tuple[str, dict[str, Any], dict]:
- """The state text and questions for one board, plus the planner's view; separate
- from `decide` so the wording can be tested and logged without a call."""
+ """Build prompts and planner truth without calling the server."""
if prompt not in PROMPTS:
raise ValueError(f"prompt must be one of {PROMPTS}, got {prompt!r}")
moves = game.moves()
@@ -120,8 +116,7 @@ def describe(game: SnakeGame, prompt: str) -> tuple[str, dict[str, Any], dict]:
class PlannerClient:
- """A stand-in server answering from the planner in SDK shapes: the reference row
- every model is measured against, and a way to run the display with no server."""
+ """Return planner decisions in SDK shapes for the offline baseline."""
served = "planner"
@@ -146,8 +141,7 @@ def system_one(self, state, questions):
class ModelPolicy:
- """Asks the server for a move; with `guarded`, executes the likeliest safe one and
- raises if no safe move exists."""
+ """Ask for a move; guarded mode picks the likeliest safe move or raises if none exists."""
def __init__(self, client, *, guarded: bool = True, prompt: str = "compact"):
if prompt not in PROMPTS:
diff --git a/packages/levbench/src/levbench/snake/run.py b/packages/levbench/src/levbench/snake/run.py
index 6dae240..da529bf 100644
--- a/packages/levbench/src/levbench/snake/run.py
+++ b/packages/levbench/src/levbench/snake/run.py
@@ -1,8 +1,4 @@
-"""The game loop: one decision per move, rounds, pacing, a replayable record.
-
-Rounds and the JSONL record follow laya (ADR-022). Shielded, a finished board starts the
-next seed; unassisted, the first round's end ends the run, so survival means something.
-"""
+"""Play and record Snake; guarded runs advance seeds, unassisted runs end after one round."""
from __future__ import annotations
@@ -122,14 +118,10 @@ def play(
keys: Callable[[], str] | None = None,
on_step: Callable[[dict, dict, dict], None] | None = None,
) -> RunSummary:
- """Play until `steps` moves, `duration` seconds, Q, or -- unassisted -- a round's end.
-
- Ctrl-C also stops the run. `fps` paces moves to a budget; None waits only
- on inference. `display` is a `LiveDisplay`; `keys` returns pressed keys
- (SPACE pause, +/- or arrow up/down speed, R next round on the next seed,
- Q quit); `on_step` gets `(game, decision, stats)` as dicts for headless
- status lines.
- """
+ """Play until the move/time limit, quit, interrupt, or an unassisted round ends.
+
+ fps controls pacing. Keys pause, change speed, advance rounds, or quit;
+ on_step receives game, decision, and stats dictionaries."""
game = SnakeGame(width, height, seed, initial_length)
policy = ModelPolicy(client, guarded=guarded, prompt=prompt)
recorder = Recorder(record)
@@ -171,6 +163,8 @@ def play(
"food_acc": 0.0,
}
deaths = 0
+ round_number = 1
+ route_correct = food_correct = 0
best_length = len(game.body)
served = model
timestamps: deque[float] = deque(maxlen=60)
@@ -178,8 +172,10 @@ def play(
started = time.perf_counter()
def next_round() -> SnakeGame:
- stats["round"] += 1
- return SnakeGame(width, height, seed + stats["round"] - 1, initial_length)
+ nonlocal round_number
+ round_number += 1
+ stats["round"] = round_number
+ return SnakeGame(width, height, seed + round_number - 1, initial_length)
try:
while True:
@@ -229,12 +225,10 @@ def next_round() -> SnakeGame:
)
stats["steps"] = len(decisions)
n = len(decisions)
- stats["route_acc"] = (
- sum(((1 - d.dead_end_risk) >= 0.5) == d.route_truth for d in decisions) / n
- )
- stats["food_acc"] = (
- sum((d.food_reachable >= 0.5) == d.food_truth for d in decisions) / n
- )
+ route_correct += ((1 - decision.dead_end_risk) >= 0.5) == decision.route_truth
+ food_correct += (decision.food_reachable >= 0.5) == decision.food_truth
+ stats["route_acc"] = route_correct / n
+ stats["food_acc"] = food_correct / n
shown_game, shown_decision = game.snapshot(), decision.to_dict()
if display:
@@ -285,7 +279,7 @@ def next_round() -> SnakeGame:
seed=seed,
best_length=best_length,
seconds=seconds,
- rounds=stats["round"],
+ rounds=round_number,
deaths=deaths,
best_score=max(stats["best"], game.score),
)
diff --git a/packages/levbench/src/levbench/snake/ui.py b/packages/levbench/src/levbench/snake/ui.py
index e5eba5b..741ff14 100644
--- a/packages/levbench/src/levbench/snake/ui.py
+++ b/packages/levbench/src/levbench/snake/ui.py
@@ -1,10 +1,6 @@
-"""A fixed-cell terminal composition, shared by the live display and replay.
+"""Live and replay terminal display adapted from laya-mlx (Apache-2.0).
-laya-mlx's layout (Apache-2.0), with the right-hand panel adapted to a client
-of a remote server: the backend and served model replace the local engine
-lines, and the planner's ground truth sits beside the model's two estimates.
-`rich` is optional: without it the loop prints one status line per step.
-"""
+Show server metadata and planner truth beside model estimates; Rich is optional."""
from __future__ import annotations
diff --git a/packages/levbench/src/levbench/tasks.py b/packages/levbench/src/levbench/tasks.py
index 5761c2d..af8a75a 100644
--- a/packages/levbench/src/levbench/tasks.py
+++ b/packages/levbench/src/levbench/tasks.py
@@ -1,14 +1,10 @@
-"""Built-in smoke fixture and file-backed benchmark tasks.
-
-The 24 support tickets below are too few to measure training improvements.
-Use held-out task files for benchmarks. Each JSON file has `questions` and
-`items`; each item has a `state` and `labels` keyed by question name.
-"""
+"""Smoke fixture and task-file loading; use held-out files to measure model improvements."""
from __future__ import annotations
import json
import math
+from collections.abc import Sequence
from dataclasses import dataclass, field
from pathlib import Path
from typing import Any
@@ -225,15 +221,14 @@ def load_task_file(path: str | Path) -> tuple[list[FileItem], dict[str, Any]]:
return items, built
-def dataset(path: str | Path | None = None) -> tuple[list[Any], dict[str, Any]]:
+def dataset(path: str | Path | None = None) -> tuple[Sequence[Item | FileItem], dict[str, Any]]:
if path is not None:
return load_task_file(path)
return ITEMS, questions()
def detectable_difference(n: int, baseline: float = 0.8, z: float = 1.96) -> float:
- """Roughly the smallest accuracy change `n` items can distinguish: the one-proportion
- normal-approximation half-width (comparing two runs needs more; ADR-015)."""
+ """Estimate the accuracy margin (95% at default z); comparing runs needs more data."""
if n <= 0:
return 1.0
return min(1.0, z * math.sqrt(baseline * (1 - baseline) / n))
diff --git a/packages/levbench/tests/fake_transport.py b/packages/levbench/tests/fake_transport.py
index 6be8c14..6882ab7 100644
--- a/packages/levbench/tests/fake_transport.py
+++ b/packages/levbench/tests/fake_transport.py
@@ -1,7 +1,4 @@
-"""A fake Jev endpoint: real request parsing, wire-shaped responses, no API key.
-
-It tests plumbing only: answers are deterministic pseudo-random, so its accuracy is meaningless.
-"""
+"""Fake Jev transport with deterministic pseudo-random answers; tests plumbing, not accuracy."""
from __future__ import annotations
@@ -16,33 +13,34 @@ def _rand(*parts: str) -> float:
return int.from_bytes(digest[:8], "big") / 2**64
+def _probabilities(key: str, options: list[str] | range, state_sig: str) -> dict:
+ weights = [_rand(state_sig, key, str(option)) + 1e-6 for option in options]
+ total = sum(weights)
+ return {o: round(w / total, 4) for o, w in zip(options, weights, strict=True)}
+
+
def _answer(key: str, question: dict, state_sig: str) -> dict:
kind = question["type"]
if kind == "noul":
return {"type": "noul", "noul": round(_rand(state_sig, key, "noul"), 4)}
if kind == "choice":
- options = list(question["criteria"])
- weights = [_rand(state_sig, key, o) + 1e-6 for o in options]
- total = sum(weights)
- probs = {o: round(w / total, 4) for o, w in zip(options, weights, strict=True)}
+ probs = _probabilities(key, list(question["criteria"]), state_sig)
return {
"type": "choice",
- "choice": max(probs, key=probs.get),
+ "choice": max(probs, key=probs.__getitem__),
"probabilities": probs,
- "confidence": round(max(probs.values()), 4),
+ "confidence": max(probs.values()),
}
if kind == "score":
levels = question["criteria"]
- weights = [_rand(state_sig, key, str(i)) + 1e-6 for i in range(len(levels))]
- total = sum(weights)
- probs = {i: round(w / total, 4) for i, w in enumerate(weights)}
+ probs = _probabilities(key, range(len(levels)), state_sig)
return {
"type": "score",
"score": round(sum(i * p for i, p in probs.items()), 4),
"probabilities": probs,
- "confidence": round(max(probs.values()), 4),
+ "confidence": max(probs.values()),
"legend": {i: lv for i, lv in enumerate(levels)},
}
@@ -58,8 +56,7 @@ def handler(request: httpx2.Request) -> httpx2.Response:
questions = body["questions"]
answers = {k: _answer(k, q, state_sig) for k, q in questions.items()}
- # Input scales with the state (billed once per request), output with the
- # question count: the shape the batching sweep must find from responses.
+ # Bill state tokens once per request and output tokens per question.
input_tokens = max(1, len(state_text) // 4)
output_tokens = 4 * len(questions)
diff --git a/packages/levbench/tests/test_offline.py b/packages/levbench/tests/test_offline.py
index 8606f84..cb01ac4 100644
--- a/packages/levbench/tests/test_offline.py
+++ b/packages/levbench/tests/test_offline.py
@@ -1,4 +1,4 @@
-"""End-to-end plumbing check against the fake transport. No API key needed."""
+"""Offline checks for benchmark metrics, SDK plumbing, and reporting. No API key needed."""
from __future__ import annotations
@@ -6,23 +6,19 @@
import json
import math
import random
-import sys
from pathlib import Path
from threading import Barrier
from types import SimpleNamespace
-PKG = Path(__file__).resolve().parents[1]
-ROOT = next(p for p in PKG.parents if (p / "data").is_dir())
-sys.path.insert(0, str(PKG / "src"))
-sys.path.insert(0, str(PKG / "tests"))
+import httpx2
+import pytest
+from fake_transport import transport
+from levbench import batching, confidence_id, metrics, runner
+from levbench.pricing import cost_usd
+from levbench.tasks import FileItem, dataset, detectable_difference, load_task_file
+from typesafe_sdk import Choice, Noul, TypeSafeClient
-import httpx2 # noqa: E402
-import pytest # noqa: E402
-from fake_transport import transport # noqa: E402
-from levbench import batching, confidence_id, metrics, runner # noqa: E402
-from levbench.pricing import cost_usd # noqa: E402
-from levbench.tasks import FileItem, dataset, detectable_difference, load_task_file # noqa: E402
-from typesafe_sdk import Choice, Noul, TypeSafeClient # noqa: E402
+ROOT = next(p for p in Path(__file__).resolve().parents if (p / "data").is_dir())
def fake_client() -> TypeSafeClient:
@@ -31,51 +27,48 @@ def fake_client() -> TypeSafeClient:
def test_metrics_are_arithmetically_right() -> None:
dist = {"a": 0.7, "b": 0.2, "c": 0.1}
- assert abs(metrics.log_loss(dist, "a") - (-math.log(0.7))) < 1e-12
+ assert metrics.log_loss(dist, "a") == pytest.approx(-math.log(0.7), abs=1e-12, rel=0)
# Brier: (0.7-1)^2 + 0.2^2 + 0.1^2 = 0.09 + 0.04 + 0.01
- assert abs(metrics.brier(dist, "a") - 0.14) < 1e-12
+ assert metrics.brier(dist, "a") == pytest.approx(0.14, abs=1e-12, rel=0)
# A perfectly calibrated set: 70% confident, 70% accurate -> ECE 0.
recs = [({True: 0.7, False: 0.3}, True, True, 0.7)] * 7
recs += [({True: 0.7, False: 0.3}, True, False, 0.7)] * 3
cal = metrics.calibration(recs)
- assert abs(cal.accuracy - 0.7) < 1e-12
+ assert cal.accuracy == pytest.approx(0.7, abs=1e-12, rel=0)
assert cal.ece < 1e-12, f"expected zero ECE, got {cal.ece}"
# A badly overconfident set: 100% confident, 50% accurate -> ECE 0.5.
bad = [({True: 1.0, False: 0.0}, True, True, 1.0)] * 5
bad += [({True: 1.0, False: 0.0}, True, False, 1.0)] * 5
- assert abs(metrics.calibration(bad).ece - 0.5) < 1e-12
+ assert metrics.calibration(bad).ece == pytest.approx(0.5, abs=1e-12, rel=0)
def test_every_primitive_flattens_to_a_distribution() -> None:
- """Every primitive must flatten, including the fieldless Noul."""
items, questions = dataset()
result = runner.call_once(fake_client(), items[0].state, questions)
noul = result.answers["is_urgent"]
- assert not hasattr(noul, "confidence") or noul.confidence is None
+ assert getattr(noul, "confidence", None) is None
dist = metrics.to_distribution(noul)
- assert abs(sum(dist.values()) - 1.0) < 1e-9
+ assert sum(dist.values()) == pytest.approx(1.0, abs=1e-9, rel=0)
assert metrics.top_probability(noul) == max(dist.values())
choice = result.answers["department"]
assert metrics.predicted_label(choice) == choice.choice
- assert abs(sum(metrics.to_distribution(choice).values()) - 1.0) < 1e-3
+ assert sum(metrics.to_distribution(choice).values()) == pytest.approx(1.0, abs=1e-3, rel=0)
score = result.answers["frustration"]
by_level = metrics.to_distribution(score)
expected = sum(level * p for level, p in by_level.items())
- assert abs(score.score - expected) < 1e-2, "score must equal the probability-weighted mean"
+ assert score.score == pytest.approx(expected, abs=1e-2, rel=0)
def test_ece_bins_on_the_top_probability_not_the_vendor_confidence_field() -> None:
- """lev's `confidence` is Gini concentration and Jev's is chance-corrected max
- probability, so ECE bins on the answer's own top probability instead."""
+ """Vendor confidence formulas differ; ECE must use the answer's top probability."""
class Server:
def system_one(self, state, questions):
- # 90% on the right answer, but a vendor confidence of 0.2.
answer = SimpleNamespace(
type="choice", choice="a", probabilities={"a": 0.9, "b": 0.1}, confidence=0.2
)
@@ -88,7 +81,7 @@ def system_one(self, state, questions):
items = [FileItem(f"s{i}", {"q": "a" if i < 9 else "b"}) for i in range(10)]
questions = {"q": Choice(instructions="?", criteria={"a": None, "b": None})}
report = runner.run_eval(Server(), "lev", "stub", items, questions)
- # Every answer is 0.9 confident and 9 of 10 are right: perfectly calibrated.
+ # Top probability is 0.9 and 9 of 10 predictions are correct: ECE is zero.
assert report.per_question["q"].ece == pytest.approx(0.0, abs=1e-9)
@@ -103,12 +96,11 @@ def test_eval_runs_end_to_end_over_the_built_in_fixture() -> None:
def test_sweep_arithmetic_over_a_fixed_billing_shape() -> None:
- """SweepRow arithmetic over the fake transport's per-request billing, not
- Jev's real billing, which only a live `levbench sweep` can measure."""
+ """Synthetic billing verifies sweep arithmetic; live billing requires a live sweep."""
state = (ROOT / "data" / "sample_policy.md").read_text()
rows = batching.sweep(fake_client(), "jev-latest", state, [1, 2, 4, 8, 13])
- assert abs(rows[0].cost_ratio - 1.0) < 0.01, (
+ assert rows[0].cost_ratio == pytest.approx(1.0, abs=0.01, rel=0), (
"at N=1 batched and split are the same call; ratio must be ~1"
)
ratios = [r.cost_ratio for r in rows]
@@ -119,51 +111,53 @@ def test_sweep_arithmetic_over_a_fixed_billing_shape() -> None:
assert "batching sweep" in batching.format_sweep(rows, "jev-latest", len(state))
-def test_confidence_identifier_recovers_a_planted_formula() -> None:
- """If planted formulas were not identified uniquely, a match against a real server
- would prove nothing."""
-
- def server(conf_fn):
- def handle(request):
- body = json.loads(request.content)
- answers = {}
- for key, question in body["questions"].items():
- options = list(question["criteria"])
- # Varied, non-uniform distributions, so candidate statistics
- # actually separate instead of coinciding.
- raw = [1 + hashlib.sha256(f"{key}:{o}".encode()).digest()[0] % 97 for o in options]
- total = sum(raw)
- probs = {o: r / total for o, r in zip(options, raw, strict=True)}
- answers[key] = {
- "type": "choice",
- "choice": max(probs, key=probs.get),
- "probabilities": probs,
- "confidence": conf_fn(list(probs.values())),
- }
- return httpx2.Response(
- 200,
- json={
- "model": "planted",
- "answers": answers,
- "usage": {"input_tokens": 10, "output_tokens": 0},
- },
- )
+@pytest.mark.parametrize("count", [0, -1])
+def test_sweep_rejects_non_positive_question_counts(count) -> None:
+ with pytest.raises(ValueError, match="must be positive"):
+ batching.sweep(None, "jev-latest", "state", [count])
+
+
+def test_sweep_respects_an_explicitly_empty_bank() -> None:
+ with pytest.raises(ValueError, match="bank holds 0"):
+ batching.sweep(None, "jev-latest", "state", [1], bank=[])
+
- return TypeSafeClient(api_key="x", model="planted", transport=httpx2.MockTransport(handle))
+@pytest.mark.parametrize("planted", confidence_id.CANDIDATES)
+def test_confidence_identifier_recovers_a_planted_formula(planted) -> None:
+ """A match is useful only if the identifier can distinguish candidate formulas."""
+
+ def handle(request):
+ body = json.loads(request.content)
+ answers = {}
+ for key, question in body["questions"].items():
+ options = list(question["criteria"])
+ # Non-uniform distributions keep candidate statistics distinguishable.
+ raw = [1 + hashlib.sha256(f"{key}:{o}".encode()).digest()[0] % 97 for o in options]
+ total = sum(raw)
+ probs = {o: r / total for o, r in zip(options, raw, strict=True)}
+ answers[key] = {
+ "type": "choice",
+ "choice": max(probs, key=probs.__getitem__),
+ "probabilities": probs,
+ "confidence": confidence_id.CANDIDATES[planted](list(probs.values())),
+ }
+ return httpx2.Response(
+ 200,
+ json={
+ "model": "planted",
+ "answers": answers,
+ "usage": {"input_tokens": 10, "output_tokens": 0},
+ },
+ )
+ client = TypeSafeClient(api_key="x", model="planted", transport=httpx2.MockTransport(handle))
questions = {
f"q{i}": Choice(instructions="pick", criteria={c: c for c in "abcde"[: 2 + i % 4]})
for i in range(6)
}
-
- for planted, fn in confidence_id.CANDIDATES.items():
- client = server(fn)
- answers = []
- for state in ("alpha", "beta", "gamma", "delta"):
- answers.extend(runner.call_once(client, state, questions).answers.values())
- fits = confidence_id.identify(confidence_id.collect(answers))
- winners = [f.name for f in fits if f.matches]
- assert winners == [planted], f"planted {planted}, identified {winners}"
+ answers = list(runner.call_once(client, "state", questions).answers.values())
+ fits = confidence_id.identify(confidence_id.collect(answers))
+ assert [f.name for f in fits if f.matches] == [planted]
def test_base_url_routes_to_a_local_clone() -> None:
@@ -191,23 +185,24 @@ def handle(request):
def test_local_base_url_never_forwards_the_real_key(monkeypatch) -> None:
- """A `--base-url` server must not receive TYPESAFE_API_KEY (the SDK always
- sends `Authorization: Bearer `); the hosted path still authenticates."""
+ """Custom endpoints use a local key; hosted requests retain the real credential."""
monkeypatch.setenv("TYPESAFE_API_KEY", "sk-REAL-SECRET-KEY")
monkeypatch.delenv("LEVBENCH_LOCAL_API_KEY", raising=False)
local, _ = runner.build_client("jev", "litjev", "http://127.0.0.1:8000")
+ assert isinstance(local, TypeSafeClient)
assert "REAL-SECRET" not in str(local._config.api_key), "leaked the real key"
assert local._config.api_key == "local"
hosted, _ = runner.build_client("jev", "jev-latest")
+ assert isinstance(hosted, TypeSafeClient)
assert hosted._config.api_key == "sk-REAL-SECRET-KEY", "hosted path lost its key"
def test_empty_local_key_env_var_falls_back(monkeypatch) -> None:
- """`LEVBENCH_LOCAL_API_KEY=` (from `.env.example`) once reached the SDK as "", which
- sends a malformed `Bearer ` header (`LocalProtocolError: Illegal header value`)."""
+ """An empty key produces an invalid Bearer header; use the local fallback."""
monkeypatch.setenv("LEVBENCH_LOCAL_API_KEY", "")
client, _ = runner.build_client("lev")
+ assert isinstance(client, TypeSafeClient)
assert client._config.api_key == "local", (
f"empty env var leaked through as {client._config.api_key!r}"
)
@@ -235,27 +230,20 @@ def _write_task_file(path, question_type="choice", n=5):
return path
-def test_eval_runs_over_a_generated_task_file(tmp_path) -> None:
- """The lev -> levbench seam: the harness reads back and scores the kind of
- file `lev data eval` writes."""
- items, questions = dataset(_write_task_file(tmp_path / "gen.json"))
- report = runner.run_eval(fake_client(), "jev", "jev-latest", items, questions)
- assert len(report.calls) == 5
- assert set(report.per_question) == {"q"}
- assert "accuracy" in runner.format_report(report)
-
-
@pytest.mark.parametrize("question_type", ["choice", "score", "noul"])
-def test_every_primitive_round_trips_through_a_task_file(tmp_path, question_type) -> None:
- items, questions = load_task_file(_write_task_file(tmp_path / "g.json", question_type))
+def test_eval_runs_over_a_generated_task_file(tmp_path, question_type) -> None:
+ """Exercise the task-file format produced by `lev data eval` for every primitive."""
+ items, questions = dataset(_write_task_file(tmp_path / "gen.json", question_type))
assert questions["q"].type == question_type
report = runner.run_eval(fake_client(), "jev", "jev-latest", items, questions)
+ assert len(report.calls) == 5
+ assert set(report.per_question) == {"q"}
assert report.per_question["q"] is not None
+ assert "accuracy" in runner.format_report(report)
def test_a_task_file_labelling_an_undefined_question_raises(tmp_path) -> None:
- """Otherwise the label is silently dropped and the eval scores fewer items
- than it was given."""
+ """Reject undefined questions so their labels cannot be silently excluded from scoring."""
path = tmp_path / "bad.json"
path.write_text(
json.dumps(
@@ -281,8 +269,7 @@ def test_detectable_difference_shrinks_with_n() -> None:
def test_pooling_can_hide_a_formula_that_holds_per_candidate_set_size() -> None:
- """The live Jev run's shape: nothing matched pooled, and the pooled view could not
- say which samples broke it."""
+ """Grouping by candidate-set size can expose formulas hidden by pooled samples."""
four = [0.7, 0.1, 0.1, 0.1]
three = [0.5, 0.3, 0.2]
samples = [(four, confidence_id.max_prob(four))] * 6
@@ -306,18 +293,17 @@ def test_worst_residuals_surfaces_the_samples_that_break_a_candidate() -> None:
distribution, reported, predicted = worst[0]
assert distribution == breaks_it
assert reported == 0.99
- assert abs(predicted - 0.5) < 1e-9
+ assert predicted == pytest.approx(0.5, abs=1e-9, rel=0)
def test_diagnosis_reports_whether_distributions_are_complete() -> None:
- """A truncated distribution makes every candidate wrong for the same reason."""
+ """Incomplete distributions can distort confidence formulas independently of fit."""
truncated = [([0.6, 0.3], 0.6)]
assert "sum to 1 within 0.1" in confidence_id.format_diagnosis(truncated)
def test_tolerance_accounts_for_how_much_a_statistic_amplifies_rounding() -> None:
- """Jev rounds to 2 dp and `norm_max_prob` amplifies that by K/(K-1) to 0.010, so a
- flat 5e-3 threshold rejected the correct formula."""
+ """Normalized max probability amplifies rounding by K/(K-1)."""
rng = random.Random(0)
samples = []
for size in (4, 3):
@@ -355,7 +341,7 @@ def test_parallel_requests_keep_item_order_and_report_wall_time(self):
class ConcurrentStub:
def system_one(self, state, questions):
- # All four workers must enter before any request can finish.
+ # The barrier makes serial execution fail instead of passing unnoticed.
barrier.wait(timeout=5)
return SimpleNamespace(
answers={
@@ -380,7 +366,7 @@ def test_cost_is_per_million_tokens_at_list_price(self) -> None:
assert cost_usd("Qwen/Qwen3.5-4B", 1_000_000, 1_000_000) == 0.0, "self-hosted is unmetered"
def test_billed_totals_win_over_base_counts(self) -> None:
- """The adapter's `*_total` fields include retried attempts -- what is billed."""
+ """Retry totals reflect billed usage."""
usage = SimpleNamespace(
input_tokens=10, input_tokens_total=25, output_tokens=2, output_tokens_total=4
)
@@ -391,14 +377,9 @@ def test_missing_token_counts_raise_rather_than_count_as_free(self) -> None:
runner._usage_ints(SimpleNamespace(input_tokens=None, output_tokens=None))
def test_a_substituted_model_is_flagged_in_the_report(self) -> None:
- """Pricing uses the requested model; if the server answered with another,
- the report has to say so."""
+ """Report model substitution because pricing uses the requested model."""
report = runner.EvalReport(backend="jev", model="jev-latest")
report.calls.append(runner.CallResult({}, 0.3, "jev-1.13.0", 10, 1))
assert "requested 'jev-latest' but server served ['jev-1.13.0']" in runner.format_report(
report
)
-
-
-if __name__ == "__main__":
- raise SystemExit(pytest.main([__file__, "-v"]))
diff --git a/packages/levbench/tests/test_snake.py b/packages/levbench/tests/test_snake.py
index 6c67406..b973f8e 100644
--- a/packages/levbench/tests/test_snake.py
+++ b/packages/levbench/tests/test_snake.py
@@ -4,7 +4,9 @@
import json
from collections import deque
+from pathlib import Path
from types import SimpleNamespace
+from typing import TypedDict
import pytest
from levbench.snake import (
@@ -20,37 +22,43 @@
from levbench.snake.policy import describe
from levbench.snake.ui import compose, layout_size, status_line
-SMALL_BOARD = {"width": 8, "height": 6, "initial_length": 3}
+class BoardSettings(TypedDict):
+ width: int
+ height: int
+ initial_length: int
-def planner_move(game):
+
+SMALL_BOARD: BoardSettings = {"width": 8, "height": 6, "initial_length": 3}
+
+
+def planner_move(game: SnakeGame) -> str:
return max((m for m in game.moves() if m.safe), key=lambda m: m.advance).direction
-def unsafe_direction(game):
+def unsafe_direction(game: SnakeGame) -> str:
return next(m.direction for m in game.moves() if not m.safe)
class StubServer:
- """Answers from a chosen direction and two Noul probabilities, SDK-shaped."""
+ """Answers from a chosen direction and two Noul probabilities"""
def __init__(self, pick=None, route=0.9, food=0.9, follow_planner=False, invert=False):
self.pick, self.route, self.food = pick, route, food
self.follow_planner, self.invert = follow_planner, invert
- self.calls: list[tuple[str, dict]] = []
def system_one(self, state, questions):
- self.calls.append((state, questions))
- options = list(questions["move"].criteria)
+ criteria = questions["move"].criteria
+ options = list(criteria)
if self.follow_planner:
- pick = next((o for o in options if "Best" in questions["move"].criteria[o]), options[0])
+ pick = next((o for o in options if "Best" in criteria[o]), options[0])
else:
pick = self.pick or options[0]
probs = {o: 0.04 for o in options}
probs[pick] = 1.0 - 0.04 * (len(options) - 1)
route, food = self.route, self.food
if self.invert:
- # Answer both Nouls from the state text, wrongly.
+ # Contradict the compact prompt to test scoring against planner truth.
route = 0.1 if "Safe route: yes" in state else 0.9
food = 0.1 if "reachable through empty cells: yes" in state else 0.9
return SimpleNamespace(
@@ -71,11 +79,10 @@ def test_cycle_visits_every_cell_once_with_adjacent_steps(self):
for a, b in zip(cycle, cycle[1:] + cycle[:1], strict=True):
assert abs(a[0] - b[0]) + abs(a[1] - b[1]) == 1
- def test_odd_by_odd_and_tiny_boards_are_rejected(self):
- with pytest.raises(ValueError):
- hamiltonian_cycle(5, 5)
+ @pytest.mark.parametrize(("width", "height"), [(5, 5), (3, 8)])
+ def test_odd_by_odd_and_tiny_boards_are_rejected(self, width, height):
with pytest.raises(ValueError):
- hamiltonian_cycle(3, 8)
+ hamiltonian_cycle(width, height)
def test_same_seed_same_game(self):
a, b = SnakeGame(seed=3), SnakeGame(seed=3)
@@ -98,17 +105,17 @@ def test_following_the_planner_never_dies(self):
def test_eating_grows_and_scores(self):
game = SnakeGame(**SMALL_BOARD, seed=1)
length = len(game.body)
- while True:
- eating = [m for m in game.moves() if m.safe and m.eats]
- if eating:
- assert game.step(eating[0].direction) is True
+ # The planner advances toward food without skipping it; one cycle is enough.
+ for _ in range(game.capacity):
+ if game.step(planner_move(game)):
break
- game.step(planner_move(game))
+ else:
+ pytest.fail("the planner did not reach food within one cycle")
assert len(game.body) == length + 1 and game.score == 1
def test_wall_and_reverse_are_illegal(self):
game = SnakeGame(**SMALL_BOARD)
- game.body = deque([(0, 0), (1, 0), (2, 0)]) # head in the top-left corner
+ game.body = deque([(0, 0), (1, 0), (2, 0)])
assert game.legal_reason("LEFT") == "wall"
assert game.legal_reason("UP") == "wall"
assert game.legal_reason("RIGHT") == "reverse"
@@ -147,7 +154,7 @@ def test_shield_executes_best_safe_move_and_keeps_raw_probabilities(self):
assert decision.proposed == bad
assert decision.executed in decision.safe_directions
assert decision.intervened
- assert max(decision.probabilities, key=decision.probabilities.get) == bad
+ assert max(decision.probabilities, key=decision.probabilities.__getitem__) == bad
def test_unassisted_executes_the_raw_choice(self):
game = SnakeGame()
@@ -242,14 +249,34 @@ def test_planner_never_needs_the_shield_and_reads_its_state(self):
class TestRounds:
def test_guarded_run_continues_past_a_cleared_board(self):
- """A 4x4 board with a planner-perfect player is cleared quickly; the run
- must roll into round 2 on the next seed rather than stop."""
+ """A small board clears within the step budget, forcing a new round."""
summary = play(
PlannerClient(), "planner", "planner", width=4, height=4, initial_length=2, steps=60
)
assert summary.rounds >= 2 and summary.steps == 60
assert summary.best_score >= 1
+ def test_live_accuracy_remains_cumulative_across_rounds(self):
+ observed = []
+ summary = play(
+ StubServer(follow_planner=True, route=0.2, food=0.9),
+ "lev",
+ "local",
+ width=4,
+ height=4,
+ initial_length=2,
+ steps=60,
+ on_step=lambda game, decision, stats: observed.append((decision, stats)),
+ )
+ assert summary.rounds >= 2
+ assert len(observed) == 60
+ for n, (_, stats) in enumerate(observed, start=1):
+ expected_food = sum(decision["food_truth"] for decision, _ in observed[:n]) / n
+ assert stats["route_acc"] == 0.0
+ assert stats["food_acc"] == pytest.approx(expected_food)
+ assert observed[-1][1]["route_acc"] == summary.route_accuracy
+ assert observed[-1][1]["food_acc"] == summary.food_accuracy
+
class TestDisplay:
def test_layout_fits_the_default_board(self):
@@ -288,7 +315,7 @@ def test_status_line_is_one_line(self):
class TestReplay:
- def test_recording_replays_every_frame(self, tmp_path):
+ def test_recording_replays_every_frame(self, tmp_path: Path):
record = tmp_path / "run.jsonl"
play(
PlannerClient(),
@@ -303,7 +330,7 @@ def test_recording_replays_every_frame(self, tmp_path):
class Screen:
def __init__(self):
- self.frames = []
+ self.frames: list[str] = []
def show(self, canvas):
self.frames.append(canvas.text())
@@ -312,20 +339,18 @@ def show(self, canvas):
assert replay(record, screen, speed=1000.0) == 8
assert "RECORDED RUN" in screen.frames[0]
- def test_broken_timestamps_are_rejected(self, tmp_path):
+ def test_broken_timestamps_are_rejected(self, tmp_path: Path):
bad = tmp_path / "bad.jsonl"
- bad.write_text(
- json.dumps({"type": "metadata", "format": "x"})
- + "\n"
- + json.dumps({"type": "frame", "at": 2.0, "game": {}, "decision": {}, "stats": {}})
- + "\n"
- + json.dumps({"type": "frame", "at": 1.0, "game": {}, "decision": {}, "stats": {}})
- + "\n"
- )
+ events = [
+ {"type": "metadata", "format": "x"},
+ {"type": "frame", "at": 2.0, "game": {}, "decision": {}, "stats": {}},
+ {"type": "frame", "at": 1.0, "game": {}, "decision": {}, "stats": {}},
+ ]
+ bad.write_text("\n".join(json.dumps(event) for event in events) + "\n")
with pytest.raises(ValueError, match="strictly increase"):
load_record(bad)
- def test_reusing_a_recording_path_replays_the_latest_run(self, tmp_path):
+ def test_reusing_a_recording_path_replays_the_latest_run(self, tmp_path: Path):
record = tmp_path / "run.jsonl"
for seed, steps in ((7, 8), (8, 3)):
play(
@@ -344,7 +369,7 @@ def test_reusing_a_recording_path_replays_the_latest_run(self, tmp_path):
events = [json.loads(line) for line in record.read_text().splitlines()]
assert sum(event["type"] == "frame" for event in events) == 11
- def test_an_empty_latest_run_does_not_borrow_previous_frames(self, tmp_path):
+ def test_an_empty_latest_run_does_not_borrow_previous_frames(self, tmp_path: Path):
record = tmp_path / "run.jsonl"
events = [
{"type": "metadata", "model": "previous"},
diff --git a/pyproject.toml b/pyproject.toml
index ca0e3d2..1340402 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -20,7 +20,6 @@ demo = ["levbench[demo]"]
dev = [
"pytest>=8.3",
"ruff>=0.14",
- # Resolve the SDK rather than the local modal/ directory during type checking.
"modal>=1.0",
]
@@ -32,6 +31,7 @@ lev = { workspace = true }
members = ["packages/*"]
[tool.pytest.ini_options]
+pythonpath = ["packages/levbench/src"]
testpaths = ["packages/levbench/tests", "packages/lev/tests"]
addopts = "-q"
diff --git a/scripts/publish_hf.sh b/scripts/publish_hf.sh
index ab01e4c..b732b6a 100755
--- a/scripts/publish_hf.sh
+++ b/scripts/publish_hf.sh
@@ -1,10 +1,11 @@
#!/usr/bin/env bash
-# Package and check the 4b-instruct release on Modal, pull it, add the hf/ model card
-# and upload it to a private Hugging Face repo.
+
+# Package, validate, pull, and publish the 4b-instruct release to Hugging Face.
#
-# scripts/publish_hf.sh # interfaze-ai/lev
-# REPO=interfaze-ai/other scripts/publish_hf.sh # another repo
-# DRY_RUN=1 scripts/publish_hf.sh # everything but the upload
+# scripts/publish_hf.sh # interfaze-ai/lev
+# REPO=interfaze-ai/other scripts/publish_hf.sh # custom repo
+# DRY_RUN=1 scripts/publish_hf.sh # skip upload
+
set -euo pipefail
PRESET="${PRESET:-4b-instruct}"
@@ -14,7 +15,7 @@ REPO="${REPO:-interfaze-ai/lev}"
cd "$(dirname "$0")/.."
release="weights/$NAME"
-# A token in .env (HF_TOKEN=...) wins over the one `hf auth login` stored.
+# Prefer HF_TOKEN from .env over `hf auth login`.
if [[ -f .env ]] && grep -q '^HF_TOKEN=' .env; then
set -a
# shellcheck source=/dev/null
@@ -22,20 +23,23 @@ if [[ -f .env ]] && grep -q '^HF_TOKEN=' .env; then
set +a
fi
-hf auth whoami >/dev/null || { echo "not logged in to the Hub: run 'hf auth login'" >&2; exit 1; }
+hf auth whoami >/dev/null || {
+ echo "not logged in to the Hub: run 'hf auth login'" >&2
+ exit 1
+}
modal run modal/app.py::export_checkpoint --preset "$PRESET" --name "$NAME"
-# Load the release as a user will before anything is pulled or published.
modal run modal/app.py::check_release --name "$NAME"
-# The target must be an existing directory: given a missing path, `volume get`
-# writes every file of the release onto that one path.
+
mkdir -p weights
modal volume get --force lev-checkpoints "releases/$NAME" weights/
+
cp hf/README.md "$release/README.md"
cp -R hf/assets "$release/"
echo "release ready in $release:"
ls -la "$release"
+
if [[ -n "${DRY_RUN:-}" ]]; then
echo "DRY_RUN set: not uploading to $REPO"
exit 0
From f7f851db2438a6ecec23f409bfad3bc3d49cd842 Mon Sep 17 00:00:00 2001
From: Abhinavexists
Date: Wed, 7 Oct 2026 23:30:37 +0530
Subject: [PATCH 15/39] refactor(train): share the checkpoint load and scoring
forward
---
docs/DECISIONS.md | 2 +-
packages/lev/src/lev/train/calibration_run.py | 41 +++++------------
packages/lev/src/lev/train/evaluate.py | 34 ++++----------
packages/lev/src/lev/train/loop.py | 35 +++++++++++++++
packages/lev/tests/test_training_loop.py | 44 +++++++++++++++++++
packages/levbench/pyproject.toml | 2 +-
packages/levbench/tests/test_snake.py | 2 +-
pyproject.toml | 6 +--
uv.lock | 8 ++--
9 files changed, 109 insertions(+), 65 deletions(-)
diff --git a/docs/DECISIONS.md b/docs/DECISIONS.md
index bc4610e..008fd2d 100644
--- a/docs/DECISIONS.md
+++ b/docs/DECISIONS.md
@@ -634,7 +634,7 @@ Accuracy, RPS, Choice and Noul are identical, to the last digit, to the runs bef
Calibration of the averaged modes. The shipped Score temperature (2.80) was fitted on single-order rows. Refit per mode on half of each bench (stratified by the gold level, seed 0), judged on the other half:
```text
- T refit held-out negative_log_likelihood held-out ECE held-out RPS
+ T refit NLL held-out ECE held-out RPS
shipped refit shipped refit shipped refit
en (144) off 3.96 0.935 0.909 0.130 0.108 0.152 0.147
reversed 3.16 0.839 0.838 0.111 0.119 0.126 0.126
diff --git a/packages/lev/src/lev/train/calibration_run.py b/packages/lev/src/lev/train/calibration_run.py
index 1eb3fc1..79a5137 100644
--- a/packages/lev/src/lev/train/calibration_run.py
+++ b/packages/lev/src/lev/train/calibration_run.py
@@ -69,46 +69,29 @@ def collect_logits(
limit: int | None = None,
) -> dict[str, list[tuple[list[float], int, str]]]:
"""Collect raw logits and task families by calibration bucket, skipping abstain rows."""
- import torch
from ..data.build import load_split
from ..data.sources import family_of
from ..data.splits import Split
- from ..prompt import Style
- from .checkpoints import load_checkpoint
- from .collate import DecisionCollator, ModeBatcher, RouteCache
from .config import PRESETS
- from .loop import build_head, build_model, candidate_logits, device_of, to_device
+ from .loop import load_for_eval, scored_rows
config = config or PRESETS["4b"]
- model, tokenizer = build_model(config, None)
- head = build_head(config, model) if config.train_mode_b_head else None
- load_checkpoint(model, head, checkpoint_dir)
- model.eval()
- if head is not None:
- head.eval()
+ model, tokenizer, head = load_for_eval(config, checkpoint_dir)
rows = [e for e in load_split(data_dir, Split(split)) if not e.abstain]
if limit:
rows = rows[:limit]
- routes = RouteCache(tokenizer, config.max_label_options, Style(config.prompt_style))
- collator = DecisionCollator(tokenizer, max_seq_len=config.max_seq_len, routes=routes)
- batcher = ModeBatcher(tokenizer, batch_size=batch_size, routes=routes)
- device = device_of(model)
-
buckets: dict[str, list[tuple[list[float], int, str]]] = {}
- with torch.no_grad():
- for group in batcher(rows):
- batch = to_device(collator(group), device)
- logits = candidate_logits(model, batch, head)
- for i, example in enumerate(group):
- width = int((~batch.candidate_mask[i]).sum())
- sample = (logits[i, :width].tolist(), example.target, family_of(example.source))
- # Banded for serving, unbanded as the fallback for thin bands.
- for key in {
- CalibrationProfile.key(example.question.type, batch.mode.value, width),
- CalibrationProfile.key(example.question.type, batch.mode.value),
- }:
- buckets.setdefault(key, []).append(sample)
+ for example, logits, mode, width in scored_rows(
+ model, tokenizer, head, config, rows, batch_size
+ ):
+ sample = (logits, example.target, family_of(example.source))
+ # Banded for serving, unbanded as the fallback for thin bands.
+ for key in {
+ CalibrationProfile.key(example.question.type, mode, width),
+ CalibrationProfile.key(example.question.type, mode),
+ }:
+ buckets.setdefault(key, []).append(sample)
return buckets
diff --git a/packages/lev/src/lev/train/evaluate.py b/packages/lev/src/lev/train/evaluate.py
index 5b18507..539c07a 100644
--- a/packages/lev/src/lev/train/evaluate.py
+++ b/packages/lev/src/lev/train/evaluate.py
@@ -100,25 +100,17 @@ def evaluate_split(
batch_size: int = 16,
) -> tuple[EvalReport, EvalReport]:
"""Return raw and calibrated reports from the same stored logits."""
- import torch
from ..data.build import load_split
from ..data.splits import Split
- from ..prompt import Style
- from .checkpoints import load_checkpoint, resolve_checkpoint
- from .collate import DecisionCollator, ModeBatcher, RouteCache
+ from .checkpoints import resolve_checkpoint
from .config import PRESETS
- from .loop import build_head, build_model, candidate_logits, device_of, to_device
+ from .loop import load_for_eval, scored_rows
config = config or PRESETS["4b"]
profile = profile or CalibrationProfile()
- model, tokenizer = build_model(config, None)
- head = build_head(config, model) if config.train_mode_b_head else None
- load_checkpoint(model, head, checkpoint_dir)
- model.eval()
- if head is not None:
- head.eval()
+ model, tokenizer, head = load_for_eval(config, checkpoint_dir)
rows = [row for row in load_split(data_dir, Split(split)) if not row.abstain]
if limit_per_source:
@@ -131,23 +123,13 @@ def evaluate_split(
kept.append(row)
rows = kept
- routes = RouteCache(tokenizer, config.max_label_options, Style(config.prompt_style))
- collator = DecisionCollator(tokenizer, max_seq_len=config.max_seq_len, routes=routes)
# Bucketed like training: each row is scored once whatever the order.
- batcher = ModeBatcher(
- tokenizer, batch_size=batch_size, bucket_window=config.bucket_window, routes=routes
- )
- device = device_of(model)
-
collected: dict[tuple[str, str, str, int], list[tuple[list[float], int]]] = {}
- with torch.no_grad():
- for group in batcher(rows):
- batch = to_device(collator(group), device)
- logits = candidate_logits(model, batch, head)
- for i, example in enumerate(group):
- width = int((~batch.candidate_mask[i]).sum())
- key = (example.source, example.question.type, batch.mode.value, width)
- collected.setdefault(key, []).append((logits[i, :width].tolist(), example.target))
+ for example, logits, mode, width in scored_rows(
+ model, tokenizer, head, config, rows, batch_size, bucket_window=config.bucket_window
+ ):
+ key = (example.source, example.question.type, mode, width)
+ collected.setdefault(key, []).append((logits, example.target))
resolved = str(resolve_checkpoint(checkpoint_dir))
plain = EvalReport(split=split, checkpoint=resolved, calibrated=False)
diff --git a/packages/lev/src/lev/train/loop.py b/packages/lev/src/lev/train/loop.py
index eeed601..7bcd3c6 100644
--- a/packages/lev/src/lev/train/loop.py
+++ b/packages/lev/src/lev/train/loop.py
@@ -179,6 +179,41 @@ def build_head(config: TrainConfig, model=None):
return head
+def load_for_eval(config: TrainConfig, checkpoint_dir, model_cache: str | None = None):
+ """Build the model and head, restore a checkpoint into them, and put both in eval mode."""
+ model, tokenizer = build_model(config, model_cache)
+ head = build_head(config, model) if config.train_mode_b_head else None
+ load_checkpoint(model, head, checkpoint_dir)
+ model.eval()
+ if head is not None:
+ head.eval()
+ return model, tokenizer, head
+
+
+def scored_rows(model, tokenizer, head, config: TrainConfig, rows, batch_size, bucket_window=64):
+ """Yield `(example, logits, mode, width)` per row, batched as training batches.
+
+ Callers differ only in how they bucket the result, so the forward pass lives here;
+ `bucket_window` is explicit because calibration and evaluation have always passed
+ different values.
+ """
+ import torch
+
+ routes = RouteCache(tokenizer, config.max_label_options, Style(config.prompt_style))
+ collator = DecisionCollator(tokenizer, max_seq_len=config.max_seq_len, routes=routes)
+ batcher = ModeBatcher(
+ tokenizer, batch_size=batch_size, bucket_window=bucket_window, routes=routes
+ )
+ device = device_of(model)
+ with torch.no_grad():
+ for group in batcher(rows):
+ batch = to_device(collator(group), device)
+ logits = candidate_logits(model, batch, head)
+ for i, example in enumerate(group):
+ width = int((~batch.candidate_mask[i]).sum())
+ yield example, logits[i, :width].tolist(), batch.mode.value, width
+
+
def device_of(model):
return next(model.parameters()).device
diff --git a/packages/lev/tests/test_training_loop.py b/packages/lev/tests/test_training_loop.py
index b2588f0..89de56b 100644
--- a/packages/lev/tests/test_training_loop.py
+++ b/packages/lev/tests/test_training_loop.py
@@ -264,3 +264,47 @@ def __call__(self, input_ids, attention_mask, logits_to_keep):
out = candidate_logits(FakeModel(), batch)
assert seen["keep"] == [4, 7], "only the distinct answer positions are projected"
assert out.tolist() == [[71.0, 72.0], [141.0, 142.0], [273.0, 274.0]]
+
+
+class TestScoredRows:
+ """`scored_rows` is the shared forward that calibration and evaluation both read."""
+
+ def rows_through(self, monkeypatch, n_rows):
+ from lev.train import loop
+ from lev.train.config import PRESETS
+
+ rows = [an_example(a_choice(4), target=i % 4, state=f"row {i}") for i in range(n_rows)]
+ monkeypatch.setattr(loop, "device_of", lambda m: torch.device("cpu"))
+ monkeypatch.setattr(loop, "to_device", lambda b, d: b)
+ monkeypatch.setattr(
+ loop,
+ "candidate_logits",
+ lambda m, b, h=None: torch.arange(
+ b.size * b.candidate_mask.size(1), dtype=torch.float
+ ).reshape(b.size, -1),
+ )
+ config = PRESETS["smoke"]
+ out = list(
+ loop.scored_rows(
+ StubModel(), make_batching_tokenizer(), None, config, rows, batch_size=8
+ )
+ )
+ return rows, out
+
+ def test_every_row_is_scored_exactly_once(self, monkeypatch):
+ rows, out = self.rows_through(monkeypatch, 32)
+ assert len(out) == len(rows)
+ assert {e.state for e, _, _, _ in out} == {r.state for r in rows}
+
+ def test_width_matches_the_option_count_and_logits_are_trimmed(self, monkeypatch):
+ from lev.router import Mode
+
+ _, out = self.rows_through(monkeypatch, 16)
+ for example, logits, mode, width in out:
+ assert width == len(example.question.criteria)
+ assert len(logits) == width
+ assert mode == Mode.LABEL_TOKEN.value
+
+ def test_bucket_window_does_not_drop_or_duplicate_rows(self, monkeypatch):
+ rows, wide = self.rows_through(monkeypatch, 24)
+ assert sorted(e.state for e, _, _, _ in wide) == sorted(r.state for r in rows)
diff --git a/packages/levbench/pyproject.toml b/packages/levbench/pyproject.toml
index 7307d71..9419238 100644
--- a/packages/levbench/pyproject.toml
+++ b/packages/levbench/pyproject.toml
@@ -1,6 +1,6 @@
[project]
name = "levbench"
-version = "0.1.0"
+version = "0.1.1"
description = "Benchmark harness for System One decision models (Jev and compatible servers)"
readme = "README.md"
requires-python = ">=3.12"
diff --git a/packages/levbench/tests/test_snake.py b/packages/levbench/tests/test_snake.py
index b973f8e..f730753 100644
--- a/packages/levbench/tests/test_snake.py
+++ b/packages/levbench/tests/test_snake.py
@@ -266,7 +266,7 @@ def test_live_accuracy_remains_cumulative_across_rounds(self):
height=4,
initial_length=2,
steps=60,
- on_step=lambda game, decision, stats: observed.append((decision, stats)),
+ on_step=lambda _, decision, stats: observed.append((decision, stats)),
)
assert summary.rounds >= 2
assert len(observed) == 60
diff --git a/pyproject.toml b/pyproject.toml
index 1340402..005b66c 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -1,7 +1,7 @@
[project]
-name = "lev-workspace"
-version = "0.1.1"
-description = "lev — umbrella workspace: the `lev` decision model and the `levbench` harness"
+name = "lev-4b"
+version = "0.1.2"
+description = "an open System One decision model"
readme = "README.md"
requires-python = ">=3.12"
license = { text = "Apache-2.0" }
diff --git a/uv.lock b/uv.lock
index 0a92e96..0c74dea 100644
--- a/uv.lock
+++ b/uv.lock
@@ -13,7 +13,7 @@ resolution-markers = [
[manifest]
members = [
"lev",
- "lev-workspace",
+ "lev-4b",
"levbench",
]
@@ -999,8 +999,8 @@ requires-dist = [
provides-extras = ["train", "serve", "modal"]
[[package]]
-name = "lev-workspace"
-version = "0.1.1"
+name = "lev-4b"
+version = "0.1.2"
source = { virtual = "." }
dependencies = [
{ name = "lev" },
@@ -1048,7 +1048,7 @@ dev = [
[[package]]
name = "levbench"
-version = "0.1.0"
+version = "0.1.1"
source = { editable = "packages/levbench" }
dependencies = [
{ name = "python-dotenv" },
From feb9865f0ad886f95349495c1e1db0d404c01591 Mon Sep 17 00:00:00 2001
From: Abhinavexists
Date: Thu, 15 Oct 2026 09:17:47 +0530
Subject: [PATCH 16/39] fix(modal): score the ordinal sweep with levbench's
label rule
---
modal/app.py | 14 +++++++++++---
1 file changed, 11 insertions(+), 3 deletions(-)
diff --git a/modal/app.py b/modal/app.py
index 3586ba5..4fce9fe 100644
--- a/modal/app.py
+++ b/modal/app.py
@@ -394,6 +394,15 @@ def check_release(name: str = "4b") -> dict:
return {"health": health, "response": served}
+def _predicted_label(answer):
+ """levbench's label rule, so a sweep is comparable with a `levbench eval` run."""
+ if answer.type == "noul":
+ return float(answer.noul) >= 0.5
+ if answer.type == "choice":
+ return answer.choice
+ return max(answer.probabilities.items(), key=lambda kv: kv[1])[0]
+
+
@app.function(gpu="H100", volumes=VOLUMES, secrets=SECRETS, timeout=2 * 60 * 60)
def diagnose_ordinal(
name: str = "lev",
@@ -426,9 +435,8 @@ def diagnose_ordinal(
picks: collections.Counter = collections.Counter()
for item in items:
answer = engine.system_one(item.state, {"decision": question}).answers["decision"]
- probs = {int(k): float(v) for k, v in answer.probabilities.items()}
- pick = max(probs, key=probs.get)
- picks[pick] += 1
+ pick = _predicted_label(answer)
+ picks[str(pick)] += 1
hits += pick == item.truth
row = report["subsets"].setdefault(subset, {})
row[f"{scale:g}"] = {
From 281704b101d18cb953fd4741f29effbfadc5d058 Mon Sep 17 00:00:00 2001
From: Abhinavexists
Date: Thu, 15 Oct 2026 09:17:47 +0530
Subject: [PATCH 17/39] feat(serve): apply the release adapter scale when
loading
---
packages/lev/src/lev/model.py | 24 +++++++++++++++++++++++-
1 file changed, 23 insertions(+), 1 deletion(-)
diff --git a/packages/lev/src/lev/model.py b/packages/lev/src/lev/model.py
index 00a7eaa..145e4a2 100644
--- a/packages/lev/src/lev/model.py
+++ b/packages/lev/src/lev/model.py
@@ -535,11 +535,16 @@ def load(
score_order_average: Literal["off", "reversed", "cyclic"] = "off",
max_batch_tokens: int | None = MAX_BATCH_TOKENS,
merge_adapter: bool = True,
+ adapter_scale: float | None = None,
) -> DecisionEngine:
"""Load an engine from a release, training output, or Hub id; otherwise use the frozen base.
Release metadata overrides the base model and prompt style. Load the checkpoint's
- tokenizer, head, and calibration when present; warn if calibration is missing."""
+ tokenizer, head, and calibration when present; warn if calibration is missing.
+
+ `adapter_scale` overrides the manifest's `adapter_scale`, a multiplier on the adapter's
+ trained strength: below 1.0 it interpolates back toward the frozen backbone, trading
+ in-distribution accuracy for out-of-distribution robustness (FINDINGS.md §19)."""
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
@@ -548,9 +553,11 @@ def load(
# Load the base model before applying the checkpoint's adapter.
resolved = resolve_checkpoint(checkpoint, cache_dir) if checkpoint else None
manifest_file = resolved / "lev_release.json" if resolved else None
+ released_scale = 1.0
if manifest_file and manifest_file.is_file():
manifest = json.loads(manifest_file.read_text())
prompt_style = manifest.get("prompt_style", prompt_style)
+ released_scale = float(manifest.get("adapter_scale", 1.0))
if manifest["base_model"] != model_id:
print(
f"release manifest names base {manifest['base_model']!r}; "
@@ -578,6 +585,9 @@ def load(
# Merge to reduce launches and fix graph weights; keep separate for adapter-scale sweeps.
model = PeftModel.from_pretrained(model, str(resolved))
+ scale = released_scale if adapter_scale is None else adapter_scale
+ if scale != 1.0:
+ _set_adapter_scale(model, scale)
if merge_adapter:
model = model.merge_and_unload()
head = _load_head(resolved / MODE_B_HEAD, model)
@@ -612,6 +622,18 @@ def load(
return engine
+def _set_adapter_scale(model, scale: float) -> None:
+ """Scale every LoRA layer relative to its trained strength, before any merge bakes it in."""
+ from peft.tuners.lora import LoraLayer
+
+ layers = [m for m in model.modules() if isinstance(m, LoraLayer)]
+ if not layers:
+ raise RuntimeError(f"adapter_scale {scale} requested, but the model has no LoRA layers")
+ for layer in layers:
+ for adapter in layer.scaling:
+ layer.set_scale(adapter, scale)
+
+
def _load_head(path: Path, model):
"""Load the saved Mode B head, or return None for Mode A-only serving."""
import torch
From 0a2a650a4a42c6d5201edc61f27d90d6ec31ceac Mon Sep 17 00:00:00 2001
From: Abhinavexists
Date: Thu, 15 Oct 2026 09:17:47 +0530
Subject: [PATCH 18/39] test(serve): pin the adapter-scale multiplier through
the merge
---
packages/lev/tests/test_adapter_scale.py | 71 ++++++++++++++++++++++++
1 file changed, 71 insertions(+)
create mode 100644 packages/lev/tests/test_adapter_scale.py
diff --git a/packages/lev/tests/test_adapter_scale.py b/packages/lev/tests/test_adapter_scale.py
new file mode 100644
index 0000000..ca08469
--- /dev/null
+++ b/packages/lev/tests/test_adapter_scale.py
@@ -0,0 +1,71 @@
+"""Pin the adapter-scale multiplier: relative to trained strength, and survives the merge."""
+
+from __future__ import annotations
+
+import pytest
+
+torch = pytest.importorskip("torch")
+qwen = pytest.importorskip("transformers.models.qwen3_5")
+peft = pytest.importorskip("peft")
+
+from lev.model import _set_adapter_scale # noqa: E402
+
+TARGET = "model.layers.1.self_attn.q_proj"
+
+
+def tiny_base():
+ torch.manual_seed(0)
+ config = qwen.Qwen3_5TextConfig(
+ vocab_size=64,
+ hidden_size=32,
+ intermediate_size=64,
+ num_hidden_layers=2,
+ layer_types=["linear_attention", "full_attention"],
+ num_attention_heads=2,
+ num_key_value_heads=1,
+ head_dim=16,
+ linear_num_key_heads=2,
+ linear_num_value_heads=2,
+ linear_key_head_dim=8,
+ linear_value_head_dim=8,
+ )
+ return qwen.Qwen3_5ForCausalLM(config).eval()
+
+
+def merged_delta(scale: float | None):
+ """Weight the merge adds to `TARGET`, with the adapter scaled by `scale` first."""
+ base = tiny_base()
+ before = base.get_submodule(TARGET).weight.detach().clone()
+
+ adapted = peft.get_peft_model(
+ base, peft.LoraConfig(r=4, lora_alpha=8, target_modules=["q_proj"])
+ )
+ # lora_B is zero-initialised, so an untouched adapter is a no-op whatever the scale.
+ for name, param in adapted.named_parameters():
+ if "lora_B" in name:
+ torch.nn.init.normal_(param, std=0.05)
+
+ if scale is not None:
+ _set_adapter_scale(adapted, scale)
+ merged = adapted.merge_and_unload()
+ return merged.get_submodule(TARGET).weight.detach() - before
+
+
+def test_the_merge_scales_the_adapter_linearly():
+ full = merged_delta(None)
+ assert full.abs().max() > 0, "the adapter must not be a no-op for this test to mean anything"
+ for scale in (0.25, 0.5):
+ torch.testing.assert_close(merged_delta(scale), full * scale, rtol=1e-4, atol=1e-6)
+
+
+def test_scale_one_is_the_trained_strength():
+ torch.testing.assert_close(merged_delta(1.0), merged_delta(None))
+
+
+def test_scale_zero_leaves_the_frozen_backbone():
+ assert merged_delta(0.0).abs().max() == 0
+
+
+def test_scaling_a_model_without_lora_layers_raises():
+ with pytest.raises(RuntimeError, match="no LoRA layers"):
+ _set_adapter_scale(tiny_base(), 0.25)
From e4351bd34731e2b2d6d830f7e996043e39973be9 Mon Sep 17 00:00:00 2001
From: Abhinavexists
Date: Thu, 15 Oct 2026 09:17:47 +0530
Subject: [PATCH 19/39] feat(modal): refit the temperature profile at a chosen
adapter scale
---
modal/app.py | 109 +++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 109 insertions(+)
diff --git a/modal/app.py b/modal/app.py
index 4fce9fe..71e3d7e 100644
--- a/modal/app.py
+++ b/modal/app.py
@@ -394,6 +394,115 @@ def check_release(name: str = "4b") -> dict:
return {"health": health, "response": served}
+@app.function(gpu="H100", volumes=VOLUMES, secrets=SECRETS, timeout=4 * 60 * 60)
+def fit_at_adapter_scale(
+ name: str = "lev",
+ preset: str = "4b-instruct",
+ scale: float = 0.25,
+ scales: str = "",
+ split: str = "calibration",
+ search_rows: int = 6000,
+ apply: bool = False,
+) -> dict:
+ """Refit the temperature profile with the adapter scaled to `scale`.
+
+ Shrinking the adapter trades in-distribution accuracy for out-of-distribution
+ robustness, so `split` cannot choose the scale: it is drawn from the training
+ mixture, and accuracy there rises monotonically to scale 1.0 (0.6233 at 0.125,
+ 0.8133 at 1.0). The scale is chosen instead by leave-one-family-out over S1Bench,
+ which holds each family out of its own selection; that put it at 0.25 for 6 of 7
+ families. Temperatures still belong on `split`, which is what this fits.
+
+ Passing `scales` sweeps them on `search_rows` rows and reports in-distribution
+ sensitivity, without selecting. `apply` writes the scale and profile into the release.
+ """
+ import json
+ import random
+
+ from lev import load
+ from lev.calibrate import CalibrationProfile, fit_for_transfer
+ from lev.data.build import load_split
+ from lev.data.sources import family_of
+ from lev.data.splits import Split
+ from lev.train.checkpoints import CALIBRATION
+ from lev.train.config import PRESETS
+ from lev.train.evaluate import score
+ from lev.train.loop import scored_rows
+ from peft.tuners.lora import LoraLayer
+
+ release = Path(f"{RELEASES_DIR}/{name}")
+ engine = load(str(release), cache_dir=MODELS_DIR, merge_adapter=False)
+ layers = [m for m in engine.model.modules() if isinstance(m, LoraLayer)]
+ if not layers:
+ raise RuntimeError("adapter already merged; nothing to scale")
+
+ config = PRESETS[preset]
+ everything = [r for r in load_split(DATA_DIR, Split(split)) if not r.abstain]
+ # The search only has to separate scales that differ by points, so it runs on a
+ # shuffled subsample; the chosen scale is then fitted on the whole split.
+ search_set = everything
+ if search_rows and search_rows < len(everything):
+ search_set = random.Random(17).sample(everything, search_rows)
+ print(
+ f"{len(layers)} lora layers, {len(everything)} {split} rows, "
+ f"searching on {len(search_set)}",
+ flush=True,
+ )
+
+ def collect(scale: float, rows):
+ for m in layers:
+ for adapter in m.scaling:
+ m.set_scale(adapter, scale)
+ buckets: dict[str, list[tuple[list[float], int, str]]] = {}
+ flat: list[tuple[list[float], int]] = []
+ for example, logits, mode, width in scored_rows(
+ engine.model, engine.tokenizer, engine.mode_b_head, config, rows, 32
+ ):
+ flat.append((logits, example.target))
+ sample = (logits, example.target, family_of(example.source))
+ for key in {
+ CalibrationProfile.key(example.question.type, mode, width),
+ CalibrationProfile.key(example.question.type, mode),
+ }:
+ buckets.setdefault(key, []).append(sample)
+ return buckets, flat
+
+ results: dict[str, dict] = {}
+ for swept in [float(x) for x in scales.split(",") if x.strip()]:
+ _, flat = collect(swept, search_set)
+ accuracy, ece, brier, log_loss = score(flat, 1.0)
+ results[f"{swept:g}"] = {
+ "accuracy": round(accuracy, 4),
+ "ece_uncalibrated": round(ece, 4),
+ "brier": round(brier, 4),
+ "log_loss": round(log_loss, 4),
+ }
+ print(f" scale {swept:<7g} {split} accuracy {accuracy:.4f} ECE {ece:.4f}", flush=True)
+
+ print(f"fitting temperatures at scale {scale} on all {len(everything)} rows", flush=True)
+ buckets, flat = collect(scale, everything)
+ profile, report = fit_for_transfer(buckets, split_name=split)
+ after = score(flat, 1.0)
+ out = {
+ "scale": scale,
+ "fitted_on": split,
+ "per_scale": results,
+ "search_rows": len(search_set) if results else 0,
+ "temperatures": profile.temperatures,
+ "calibration_accuracy": round(after[0], 4),
+ "calibration_ece_uncalibrated": round(after[1], 4),
+ }
+ if apply:
+ profile.save(release / CALIBRATION)
+ manifest = json.loads((release / "lev_release.json").read_text())
+ manifest["adapter_scale"] = scale
+ (release / "lev_release.json").write_text(json.dumps(manifest, indent=2) + "\n")
+ checkpoints.commit()
+ print(f"wrote scale {scale} and refitted temperatures into {release}")
+ print(json.dumps(out, indent=2))
+ return out
+
+
def _predicted_label(answer):
"""levbench's label rule, so a sweep is comparable with a `levbench eval` run."""
if answer.type == "noul":
From ae21270f2685a4b72826313004201295f71b6db1 Mon Sep 17 00:00:00 2001
From: Abhinavexists
Date: Thu, 15 Oct 2026 09:17:47 +0530
Subject: [PATCH 20/39] docs(findings): record the adapter-scale result and its
selection
---
docs/FINDINGS.md | 35 +++++++++++++++++++++++++++++++++++
1 file changed, 35 insertions(+)
diff --git a/docs/FINDINGS.md b/docs/FINDINGS.md
index c53ed27..a9c2d35 100644
--- a/docs/FINDINGS.md
+++ b/docs/FINDINGS.md
@@ -677,3 +677,38 @@ A user on a 3090 reported that lev "suffers in boundary conditions. When the ans
**What should, in order:** graded or smoothed Noul targets through the existing `soft_targets` path, instead of one-hot 0/8; a paired consistency loss across whitespace and paraphrase variants of the same row; distillation toward the frozen model's Noul distribution, which these measurements show is well behaved at the boundary. Stripping trailing whitespace from the state before rendering is a near-free partial patch.
**Limits.** The sample is 30 items per subset with one seed, so per-subset rates are noisy (civil_comments, the worst, is 8 of 30 counting the layout flips). bf16 on MPS is not the reporter's CUDA. On the same weights, batching alone moves p by up to 0.07 in bf16 (ADR-030), so small differences between perturbations are within that floor; spreads of 0.5 or more are not.
+
+## 19. The adapter scale is an out-of-distribution knob, and it is worth +3.9 macro points
+
+The released adapter is applied at full strength. Scaling it down at serving time interpolates back toward the frozen backbone — WiSE-FT ([arXiv 2109.01903](https://arxiv.org/abs/2109.01903)), except applied to a LoRA scale rather than a weight average. Swept 2026-10-07/08 on one H100, over all 13 pinned S1Bench subsets, by setting `LoraLayer.set_scale` on each of the 128 adapter layers. Scale 1.0 scores 0.6892 against the 0.6893 measured in §17, which is the control for the whole table.
+
+```text
+ 0 0.25 0.5 0.75 1 LOFO scale
+ aegis2 0.4320 0.8360 0.8200 0.8040 0.8000 <- 0.25
+ boolq 0.4200 0.8533 0.8467 0.8367 0.8267 <- 0.25
+ civil_comments 0.8900 0.7867 0.7767 0.7800 0.7600 <- 0.25
+ helpsteer2 0.3133 0.3574 0.3855 0.3896 0.3855 <- 0.25
+ massive-de-DE 0.8000 0.8257 0.8486 0.8429 0.8229 <- 0.25
+ massive-en-US 0.8257 0.8686 0.8743 0.8686 0.8571 <- 0.25
+ multinli 0.7893 0.8495 0.8662 0.8729 0.8896 <- 0.25
+ paws 0.5160 0.8520 0.8360 0.8040 0.7760 <- 0.25
+ pubmedqa 0.7040 0.7160 0.7160 0.7160 0.7320 <- 0.25
+ squad2 0.5017 0.8227 0.8161 0.8294 0.8127 <- 0.25
+ summeval-consistency 0.8264 0.8056 0.6250 0.3958 0.2708 <- 0.5
+ summeval-relevance 0.4750 0.3792 0.3458 0.3667 0.3583 <- 0.5
+ vitaminc-dev 0.6912 0.7279 0.7179 0.6828 0.6678 <- 0.25
+ -------------------------------------------------------------------
+ macro 0.6296 0.7447 0.7288 0.7069 0.6892
+```
+
+**Scale 0.25 beats full strength by 5.5 macro points — but that number is test-selected.**
+Choosing the scale on the same 13 subsets we report is fitting the benchmark, which is exactly what the contamination guard exists to forbid. Two honest selections:
+
+**The calibration split cannot do it.** Swept 2026-10-08 on 6,000 of its 48,691 rows at seven scales: accuracy is monotone in scale — 0.6233, 0.6508, 0.6810, 0.7112, 0.7393, 0.7942, 0.8133 at 0.125 through 1.0 — and picks 1.0, the scale already shipped. The split is drawn from the training mixture, so it is in-distribution, and shrinking the adapter *costs* in-distribution accuracy while buying out-of-distribution robustness. An in-distribution split is structurally blind to the trade, whatever its size. (Uncalibrated ECE moves the other way, 0.0619 at 0.125 to 0.1400 at 1.0.)
+
+**Leave-one-family-out over S1Bench can.** The 13 subsets share task families with training but no datasets. Grouping them into 7 families — safety, qa-bool, nli, paraphrase, intent, summeval, rating — and scoring each family at the scale chosen on the *other* six gives a macro of **0.7282**, against 0.6892 shipped and 0.7610 for Jev. Six of seven families pick 0.25 independently; only summeval picks 0.5, because summeval-consistency is the one subset that prefers a weak adapter (0.8264 frozen, 0.2708 at full strength) and the remaining subsets cannot see that when it is held
+out. The estimate is robust to the grouping: merging helpsteer2 into summeval, splitting vitaminc off NLI, and leave-one-*subset*-out give 0.7447, 0.7282 and 0.7447.
+
+**This needs no retraining.** The scale is a serving-time multiplier on weights we already ship.
+
+**Limits.** The grid is coarse — 0, 0.25, 0.5, 0.75, 1.0 — and the optimum sits at its low end next to 0.25, so a finer sweep may do better; it would have to be selected under the same leave-one-out discipline to stay quotable. Accuracy is an argmax and so is invariant to temperature, which is why these numbers hold even though the shipped temperatures were fitted at scale 1.0; every *calibration* figure at any other scale requires a refit, which is what `fit_at_adapter_scale` in `modal/app.py` does. Per-subset n runs from 144 (summeval-consistency) to 599 (vitaminc-dev), so a single-subset move under about 6 points is noise at the small end and about 3 at the large end; the macro, pooled over all 13, is tighter than either. Closing the gap to Jev takes 3.3 points more than this buys.
From 5117993b1658ee1b7a4f02a9d2b979c7134aa8b3 Mon Sep 17 00:00:00 2001
From: Abhinavexists
Date: Thu, 15 Oct 2026 09:17:47 +0530
Subject: [PATCH 21/39] refactor(train): derive the bucket window from the
training config
---
packages/lev/src/lev/train/evaluate.py | 2 +-
packages/lev/src/lev/train/loop.py | 11 +++--------
2 files changed, 4 insertions(+), 9 deletions(-)
diff --git a/packages/lev/src/lev/train/evaluate.py b/packages/lev/src/lev/train/evaluate.py
index 539c07a..f473f01 100644
--- a/packages/lev/src/lev/train/evaluate.py
+++ b/packages/lev/src/lev/train/evaluate.py
@@ -126,7 +126,7 @@ def evaluate_split(
# Bucketed like training: each row is scored once whatever the order.
collected: dict[tuple[str, str, str, int], list[tuple[list[float], int]]] = {}
for example, logits, mode, width in scored_rows(
- model, tokenizer, head, config, rows, batch_size, bucket_window=config.bucket_window
+ model, tokenizer, head, config, rows, batch_size
):
key = (example.source, example.question.type, mode, width)
collected.setdefault(key, []).append((logits, example.target))
diff --git a/packages/lev/src/lev/train/loop.py b/packages/lev/src/lev/train/loop.py
index 7bcd3c6..b4a5b3a 100644
--- a/packages/lev/src/lev/train/loop.py
+++ b/packages/lev/src/lev/train/loop.py
@@ -190,19 +190,14 @@ def load_for_eval(config: TrainConfig, checkpoint_dir, model_cache: str | None =
return model, tokenizer, head
-def scored_rows(model, tokenizer, head, config: TrainConfig, rows, batch_size, bucket_window=64):
- """Yield `(example, logits, mode, width)` per row, batched as training batches.
-
- Callers differ only in how they bucket the result, so the forward pass lives here;
- `bucket_window` is explicit because calibration and evaluation have always passed
- different values.
- """
+def scored_rows(model, tokenizer, head, config: TrainConfig, rows, batch_size):
+ """Yield `(example, logits, mode, width)` per row, batched as training batches."""
import torch
routes = RouteCache(tokenizer, config.max_label_options, Style(config.prompt_style))
collator = DecisionCollator(tokenizer, max_seq_len=config.max_seq_len, routes=routes)
batcher = ModeBatcher(
- tokenizer, batch_size=batch_size, bucket_window=bucket_window, routes=routes
+ tokenizer, batch_size=batch_size, bucket_window=config.bucket_window, routes=routes
)
device = device_of(model)
with torch.no_grad():
From 06b43a8da36979a0d8c1f5a78457bf2a506d283d Mon Sep 17 00:00:00 2001
From: Abhinavexists
Date: Thu, 15 Oct 2026 09:17:47 +0530
Subject: [PATCH 22/39] test(train): cover the ragged final batch in the
scored-rows round trip
---
packages/lev/tests/test_training_loop.py | 9 +++------
1 file changed, 3 insertions(+), 6 deletions(-)
diff --git a/packages/lev/tests/test_training_loop.py b/packages/lev/tests/test_training_loop.py
index 89de56b..163339c 100644
--- a/packages/lev/tests/test_training_loop.py
+++ b/packages/lev/tests/test_training_loop.py
@@ -291,8 +291,9 @@ def rows_through(self, monkeypatch, n_rows):
)
return rows, out
- def test_every_row_is_scored_exactly_once(self, monkeypatch):
- rows, out = self.rows_through(monkeypatch, 32)
+ @pytest.mark.parametrize("n_rows", [32, 29])
+ def test_every_row_is_scored_exactly_once(self, monkeypatch, n_rows):
+ rows, out = self.rows_through(monkeypatch, n_rows)
assert len(out) == len(rows)
assert {e.state for e, _, _, _ in out} == {r.state for r in rows}
@@ -304,7 +305,3 @@ def test_width_matches_the_option_count_and_logits_are_trimmed(self, monkeypatch
assert width == len(example.question.criteria)
assert len(logits) == width
assert mode == Mode.LABEL_TOKEN.value
-
- def test_bucket_window_does_not_drop_or_duplicate_rows(self, monkeypatch):
- rows, wide = self.rows_through(monkeypatch, 24)
- assert sorted(e.state for e, _, _, _ in wide) == sorted(r.state for r in rows)
From 0bce0d376c23ac5de9a46faf1c5d37842ca741d0 Mon Sep 17 00:00:00 2001
From: Abhinavexists
Date: Thu, 15 Oct 2026 09:17:47 +0530
Subject: [PATCH 23/39] refactor(train): tighten the training loop's types
---
packages/lev/src/lev/train/loop.py | 15 ++++++++++++---
1 file changed, 12 insertions(+), 3 deletions(-)
diff --git a/packages/lev/src/lev/train/loop.py b/packages/lev/src/lev/train/loop.py
index b4a5b3a..5abbd38 100644
--- a/packages/lev/src/lev/train/loop.py
+++ b/packages/lev/src/lev/train/loop.py
@@ -8,7 +8,10 @@
from collections.abc import Callable
from dataclasses import fields
from pathlib import Path
-from typing import Any
+from typing import TYPE_CHECKING, Any
+
+if TYPE_CHECKING:
+ import torch
from ..prompt import Style
from .checkpoints import (
@@ -77,7 +80,13 @@ def build_model(config: TrainConfig, model_cache: str | None = None):
return model, tokenizer
-def decision_loss(logits, targets, config: TrainConfig, ordinal=False, soft_targets=None):
+def decision_loss(
+ logits,
+ targets,
+ config: TrainConfig,
+ ordinal: bool | torch.Tensor = False,
+ soft_targets=None,
+):
"""Cross-entropy, Brier and ordinal loss over padded candidate sets.
Args:
@@ -400,7 +409,7 @@ def training_state() -> dict:
step,
value,
batch.mode.value,
- scheduler.get_last_lr()[0],
+ float(scheduler.get_last_lr()[0]),
tokens=int(batch.attention_mask.sum()),
)
step += 1
From 6efa5d7a0b2a4b43f017667b31911707c21e1b17 Mon Sep 17 00:00:00 2001
From: Abhinavexists
Date: Thu, 15 Oct 2026 09:17:47 +0530
Subject: [PATCH 24/39] test(train): tighten the training-loop tests' types
---
packages/lev/tests/test_training_loop.py | 9 ++++++---
1 file changed, 6 insertions(+), 3 deletions(-)
diff --git a/packages/lev/tests/test_training_loop.py b/packages/lev/tests/test_training_loop.py
index 163339c..f64b09a 100644
--- a/packages/lev/tests/test_training_loop.py
+++ b/packages/lev/tests/test_training_loop.py
@@ -78,7 +78,7 @@ def run_resumable(
from lev.train.collate import DecisionCollator
saved: list[int] = []
- seen: list[tuple[str, ...]] = []
+ seen: list[tuple[object, ...]] = []
class RecordingCollator(DecisionCollator):
def __call__(self, examples):
@@ -295,13 +295,16 @@ def rows_through(self, monkeypatch, n_rows):
def test_every_row_is_scored_exactly_once(self, monkeypatch, n_rows):
rows, out = self.rows_through(monkeypatch, n_rows)
assert len(out) == len(rows)
- assert {e.state for e, _, _, _ in out} == {r.state for r in rows}
+ assert len(out) == len(rows)
+ assert all(any(e.state == r.state for e, _, _, _ in out) for r in rows)
def test_width_matches_the_option_count_and_logits_are_trimmed(self, monkeypatch):
from lev.router import Mode
_, out = self.rows_through(monkeypatch, 16)
for example, logits, mode, width in out:
- assert width == len(example.question.criteria)
+ criteria = example.question.criteria
+ assert criteria is not None
+ assert width == len(criteria)
assert len(logits) == width
assert mode == Mode.LABEL_TOKEN.value
From 594e284332cafc5887ff1fa4dfc42fc5435cf07c Mon Sep 17 00:00:00 2001
From: Abhinavexists
Date: Thu, 15 Oct 2026 09:17:47 +0530
Subject: [PATCH 25/39] test(packaging): drop the dead help-comment strip and
share the doc scan
---
packages/lev/tests/test_packaging.py | 44 +++++++++++-----------------
1 file changed, 17 insertions(+), 27 deletions(-)
diff --git a/packages/lev/tests/test_packaging.py b/packages/lev/tests/test_packaging.py
index 13f14e9..b926f52 100644
--- a/packages/lev/tests/test_packaging.py
+++ b/packages/lev/tests/test_packaging.py
@@ -20,18 +20,24 @@ def _extras(pyproject: Path) -> set[str]:
return set(data.get("project", {}).get("optional-dependencies", {}))
-def _documented_extras() -> dict[str, set[str]]:
- found: dict[str, set[str]] = {}
+def _doc_sources() -> list[Path]:
sources = [ROOT / "README.md", ROOT / "Makefile", ROOT / "CONTRIBUTING.md"]
- sources += sorted((ROOT / "docs").glob("*.md"))
- for src in sources:
- if not src.is_file():
- continue
- for name in re.findall(r"--extra[= ]([a-z][a-z0-9-]*)", src.read_text()):
- found.setdefault(name, set()).add(src.name)
+ return [p for p in sources + sorted((ROOT / "docs").glob("*.md")) if p.is_file()]
+
+
+def _documented_values(pattern: str) -> dict[str, set[str]]:
+ """Map each value `pattern` captures to the documentation files that mention it."""
+ found: dict[str, set[str]] = {}
+ for src in _doc_sources():
+ for value in re.findall(pattern, src.read_text()):
+ found.setdefault(value, set()).add(src.name)
return found
+def _documented_extras() -> dict[str, set[str]]:
+ return _documented_values(r"--extra[= ]([a-z][a-z0-9-]*)")
+
+
def test_root_forwards_every_member_extra():
root = _extras(ROOT / "pyproject.toml")
for member in sorted((ROOT / "packages").glob("*/pyproject.toml")):
@@ -91,21 +97,10 @@ def capture(self, args=None, namespace=None):
return subcommands, choices
-def _documented_flag_values(flag: str) -> dict[str, set[str]]:
- found: dict[str, set[str]] = {}
- sources = [ROOT / "README.md", ROOT / "Makefile", ROOT / "CONTRIBUTING.md"]
- sources += sorted((ROOT / "docs").glob("*.md"))
- for src in sources:
- if src.is_file():
- for value in re.findall(rf"{flag}[= ]([a-z][a-z0-9-]*)", src.read_text()):
- found.setdefault(value, set()).add(src.name)
- return found
-
-
def test_every_documented_backend_exists():
_, choices = _inspect_cli("levbench")
accepted = choices["--backend"]
- for value, where in sorted(_documented_flag_values("--backend").items()):
+ for value, where in sorted(_documented_values(r"--backend[= ]([a-z][a-z0-9-]*)").items()):
assert value in accepted, (
f"{sorted(where)} document `--backend {value}`, but the CLI accepts "
f"only {sorted(accepted)}."
@@ -118,16 +113,11 @@ def test_every_documented_backend_exists():
def _documented_commands(tool: str) -> dict[str, set[str]]:
"""Extract commands from fenced blocks and inline code, excluding prose."""
found: dict[str, set[str]] = {}
- sources = [ROOT / "README.md", ROOT / "Makefile", ROOT / "CONTRIBUTING.md"]
- sources += sorted((ROOT / "docs").glob("*.md"))
pattern = re.compile(rf"(?:^|\s)(?:uv run )?{tool} ([a-z][a-z0-9-]*)", re.M)
- for src in sources:
- if not src.is_file():
- continue
+ for src in _doc_sources():
text = src.read_text()
- # Exclude Makefile help comments from command extraction.
code = (
- [re.sub(r"##.*", "", text)]
+ [text]
if src.suffix != ".md"
else [block or span for block, span in CODE_SPANS.findall(text)]
)
From 3031d43dbc1465d70b1b53278b9a676f01578c50 Mon Sep 17 00:00:00 2001
From: Abhinavexists
Date: Thu, 15 Oct 2026 09:17:47 +0530
Subject: [PATCH 26/39] docs(claude): give agents the project context
---
CLAUDE.md | 71 ++++++++++++++++++++++++++++++++++++-------------------
1 file changed, 47 insertions(+), 24 deletions(-)
diff --git a/CLAUDE.md b/CLAUDE.md
index 208444c..e09cc73 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -1,34 +1,57 @@
-## Comments
+# lev
+
+lev is an open System One decision model: `f(state, questions) → typed, calibrated distributions`, served over the same `/v1/systemone` wire format as TypeSafe's Jev, with no token generation. It is `Qwen/Qwen3.5-4B` with a rank-32 LoRA adapter, a set-attention candidate head, and per-bucket temperature calibration, released on the Hub as [`interfaze-ai/lev`](https://huggingface.co/interfaze-ai/lev). `levbench` is the harness that measures it, and Jev, on S1Bench.
+
+@CONTRIBUTING.md
+
+## Where the facts live
-Keep comments rare and information-dense.
+Read the relevant doc before changing anything it covers; most "obvious" improvements here were already tried and are recorded with their evidence.
-- Do not add comments by default.
-- Do not explain code that is already obvious from the implementation.
-- Never narrate the implementation line-by-line.
-- Do not add comments for simple assignments, conditionals, loops, function calls, or standard library usage.
-- Prefer better naming and structure over explanatory comments.
-- Comments should explain information that cannot be expressed clearly in the code.
+| Doc | Holds |
+| --- | --- |
+| [`docs/STATUS.md`](docs/STATUS.md) | What is built, how far each part is exercised, the current headline numbers |
+| [`docs/DECISIONS.md`](docs/DECISIONS.md) | ADR-001 to ADR-030; the next decision is ADR-031 |
+| [`docs/FINDINGS.md`](docs/FINDINGS.md) | Every measurement and head-to-head, in order, with provenance |
+| [`docs/ARCHITECTURE.md`](docs/ARCHITECTURE.md) | The design, and the trade-offs refused |
+| [`docs/TRAINING.md`](docs/TRAINING.md) | The training order, presets, budgets, resume |
+| [`docs/SETUP.md`](docs/SETUP.md) | Local and Modal setup, volumes, deploy knobs |
+| [`hf/README.md`](hf/README.md) | The Hub model card; keep it consistent with `README.md` |
-Add a comment only when it explains one of:
+## The model
-- non-obvious intent or rationale
-- an important business rule
-- an external API/protocol assumption
-- a performance or security constraint
-- an edge case that is easy to miss
-- a workaround for a known bug or limitation
+- **Readout.** Mode A reads label-token logits at the answer boundary whenever every label code is a single token — 68 options for this tokenizer, since the 69th code, `BQ`, is two. Above that, Mode B scores candidate text with the trained head. Training and serving route the same way (ADR-025, ADR-026); serving also skips split codes (ADR-028). Routing is in `router.py`.
+- **Calibration** is per bucket, keyed `{type}:{mode}[:{band}]`, selected by leave-one-family-out transfer ECE. `calibrate.fit` refuses `test`, `eval` and `holdout` splits — fit only on the dedicated `calibration` split.
+- **The release** (`lev-4b`) is preset `4b-instruct`, step 18750, chat prompt style. Its `lev_release.json` manifest sets the base model, prompt style and optional `adapter_scale`, and `lev.load` honours all three.
+- **The released weights were trained at commit `c6ccad7`.** Changes to data, sources, prompts or the training loop after it are not in the weights until a retrain; do not attribute a benchmark move to them.
-Keep implementation comments to one or two concise lines whenever possible.
+## Running things
-Do not write multi-paragraph comments or verbose explanatory blocks.
+- **Anything that needs a GPU runs on Modal**, through `modal/app.py` and the `make` targets. Do not download checkpoints or run models locally to check something — probe the deployed endpoint or add a `modal run` function.
+- **Pass `--preset 4b-instruct` to every Modal function you call directly.** Their defaults are preset `4b`, the plain base-model run, not the release; the `make` targets pass `PRESET=4b-instruct` for you. `modal deploy` likewise serves `4b` unless `LEV_SERVE_PRESET` is set, which `make deploy` does.
+- `make deploy` replaces the live endpoint. `modal/app.py` must import inside the container, where `python-dotenv` is not installed — keep its import guarded, or every container crash-loops.
+- Modal volumes: `lev-models` (base weights), `lev-checkpoints` (runs and `releases/`), `lev-data` (mixture and splits).
-For public APIs, use docstrings/documentation when the API contract, parameters, return behavior, exceptions, or usage constraints need to be documented.
+## Benchmarks
-When modifying existing code:
+- S1Bench is 13 subsets pinned by Bespoke Nimble manifests at commit `62076b4`, in `packages/lev/src/lev/data/s1bench_subsets/`. All 13 are banned from training (ADR-009).
+- **Never select anything on S1Bench** — a hyperparameter, a scale, a prompt. Select with leave-one-family-out over the subsets, as FINDINGS §19 does, or on data outside both training and S1Bench. The `calibration` split cannot select out-of-distribution knobs: it is drawn from the training mixture.
+- Compare against Jev through identical task files, one complete run each.
+
+## Safety
+
+- **Ask before any paid call**: `levbench` against Jev, the Anthropic backend, or a sweep. They bill per token.
+- `levbench` never sends `TYPESAFE_API_KEY` to a `--base-url` host; a custom endpoint gets `LEVBENCH_LOCAL_API_KEY` or `local`. Keep it that way.
+- Do not read `.env`.
+- This repository is mirrored by hand to the public [`InterfazeAI/lev`](https://github.com/InterfazeAI/lev); the two diverge, so a change meant for release must be ported, not assumed.
+
+## Comments
-- Preserve useful existing comments.
-- Remove comments that are made obsolete by the change.
-- Do not rewrite comments merely for stylistic reasons.
-- Do not perform unrelated comment cleanup.
+The code should explain itself; a comment exists only for what it cannot say.
-Default assumption: if the code is understandable without the comment, do not write the comment.
+- **One line is the unit.** A comment is one line, a docstring is one line. Multi-line is earned — a module's two or three lines of orientation, or tensor shapes and wire formats. A second paragraph on a function belongs in `docs/` with a pointer from the code.
+- **Do not restate the code.** If a reader would have to check the prose against the implementation to confirm they agree, the comment costs more than it gives.
+- **Compress before deleting.** A real fact buried in a block above a statement usually becomes a short trailing comment on the statement.
+- **Point, do not restate.** `(ADR-028)` beats a paragraph reproducing the decision. A contract the code states plainly — a guard that raises, a named check like `_refuse_held_out` — needs no prose.
+- **Public surface is the exception.** `lev.load`, the server, and the `/v1/systemone` types are consumed without reading the body; document their contracts.
+- Do not rewrite comments for style. When changing code, remove comments the change made obsolete, and do no unrelated cleanup.
From b8f876005ec7f10066a96d41be01d351d8cd252e Mon Sep 17 00:00:00 2001
From: Abhinavexists
Date: Thu, 15 Oct 2026 09:17:48 +0530
Subject: [PATCH 27/39] refactor: replace hand-counted padding with format
width specifiers
---
modal/app.py | 4 ++--
packages/lev/src/lev/cli.py | 4 +++-
packages/lev/src/lev/train/config.py | 16 +++++++--------
packages/lev/src/lev/train/evaluate.py | 2 +-
packages/levbench/src/levbench/runner.py | 24 +++++++++++-----------
packages/levbench/src/levbench/snake/ui.py | 2 +-
6 files changed, 27 insertions(+), 25 deletions(-)
diff --git a/modal/app.py b/modal/app.py
index 71e3d7e..7744e6a 100644
--- a/modal/app.py
+++ b/modal/app.py
@@ -336,8 +336,8 @@ def export_checkpoint(preset: str = "4b", name: str | None = None) -> dict:
print(
f"release {manifest['name']} -> {RELEASES_DIR}/{target} ({len(manifest['files'])} files)"
)
- print(f" pull it: make weights RELEASE={target}")
- print(f" publish it: make publish RELEASE={target} REPO=")
+ print(f" {'pull it:':<12}make weights RELEASE={target}")
+ print(f" {'publish it:':<12}make publish RELEASE={target} REPO=")
return manifest
diff --git a/packages/lev/src/lev/cli.py b/packages/lev/src/lev/cli.py
index 0cd616f..93b9c43 100644
--- a/packages/lev/src/lev/cli.py
+++ b/packages/lev/src/lev/cli.py
@@ -118,7 +118,9 @@ def cmd_data_build(args: argparse.Namespace) -> None:
f"(oversample {ratio}x)"
+ (" <- raise --limit-per-source to lower this" if ratio > 2 else "")
)
- print(f" sources {len(manifest['sources'])} ({', '.join(sorted(manifest['sources']))})")
+ print(
+ f" {'sources':<12} {len(manifest['sources'])} ({', '.join(sorted(manifest['sources']))})"
+ )
def cmd_data_eval(args: argparse.Namespace) -> None:
diff --git a/packages/lev/src/lev/train/config.py b/packages/lev/src/lev/train/config.py
index 2b41c8a..3363f59 100644
--- a/packages/lev/src/lev/train/config.py
+++ b/packages/lev/src/lev/train/config.py
@@ -123,18 +123,18 @@ def summary(self) -> str:
prompt = f"{self.prompt_style} prompts, Mode A to the tokenizer limit"
return "\n".join(
[
- f"model {self.model_id} ({self.params_b}B, {self.dtype})",
- f"adaptation {adaptation}",
- f"prompt {prompt}",
- f"data {self.n_examples:,} examples x "
+ f"{'model':<17}{self.model_id} ({self.params_b}B, {self.dtype})",
+ f"{'adaptation':<17}{adaptation}",
+ f"{'prompt':<17}{prompt}",
+ f"{'data':<17}{self.n_examples:,} examples x "
f"{self.avg_tokens_per_example} tok x {self.epochs} epochs"
f" = {self.total_tokens / 1e9:.2f}B tokens",
- f"steps {self.total_steps:,} "
+ f"{'steps':<17}{self.total_steps:,} "
f"({self.examples_per_step} ex/step, ~{self.tokens_per_step:,} tok/step)",
- f"compute {self.total_flops:.2e} FLOPs",
- f"H100 estimate {self.estimated_hours:.1f} hours "
+ f"{'compute':<17}{self.total_flops:.2e} FLOPs",
+ f"{'H100 estimate':<17}{self.estimated_hours:.1f} hours "
f"({self.estimated_hours / 24:.1f} days)",
- f"memory {self.optimizer_gb:.1f} GB state, "
+ f"{'memory':<17}{self.optimizer_gb:.1f} GB state, "
f"{self.headroom_gb:.1f} GB headroom of {H100_VRAM_GB:.0f} GB",
]
)
diff --git a/packages/lev/src/lev/train/evaluate.py b/packages/lev/src/lev/train/evaluate.py
index f473f01..f5d7dca 100644
--- a/packages/lev/src/lev/train/evaluate.py
+++ b/packages/lev/src/lev/train/evaluate.py
@@ -54,7 +54,7 @@ def summary(self) -> str:
weighted_accuracy = sum(s.accuracy * s.n for s in self.per_source) / total
weighted_ece = sum(s.ece * s.n for s in self.per_source) / total
rows.append(
- f" {'WEIGHTED':<18} {'':<6} {' '} n={total:<5} "
+ f" {'WEIGHTED':<18} {'':<6} {'':<1} n={total:<5} "
f"acc {weighted_accuracy:.3f} "
f"+/-{accuracy_interval(weighted_accuracy, total):.3f} "
f"ECE {weighted_ece:.4f}"
diff --git a/packages/levbench/src/levbench/runner.py b/packages/levbench/src/levbench/runner.py
index a96a302..b919b62 100644
--- a/packages/levbench/src/levbench/runner.py
+++ b/packages/levbench/src/levbench/runner.py
@@ -212,29 +212,29 @@ def _format_summary(report: EvalReport) -> list[str]:
lines.append(
f"{'wall time':<18} {report.wall_seconds:.1f}s ({n / report.wall_seconds:.2f} items/s)"
)
- lines.append(f"calls {n}")
+ lines.append(f"{'calls':<18} {n}")
if n:
- lines.append(f"latency p50 {report.pct(0.5):.3f}s")
- lines.append(f"latency mean {statistics.mean(report.latencies):.3f}s")
+ lines.append(f"{'latency p50':<18} {report.pct(0.5):.3f}s")
+ lines.append(f"{'latency mean':<18} {statistics.mean(report.latencies):.3f}s")
# Report the maximum for small samples, where p95 is dominated by connection setup.
if n >= 100:
- lines.append(f"latency p95 {report.pct(0.95):.3f}s")
+ lines.append(f"{'latency p95':<18} {report.pct(0.95):.3f}s")
else:
lines.append(
- f"latency slowest {report.latencies[-1]:.3f}s (n={n}; too few for a p95)"
+ f"{'latency slowest':<18} {report.latencies[-1]:.3f}s (n={n}; too few for a p95)"
)
- lines.append(f"input tokens {sum(call.input_tokens for call in report.calls):,}")
- lines.append(f"output tokens {sum(call.output_tokens for call in report.calls):,}")
+ lines.append(f"{'input tokens':<18} {sum(call.input_tokens for call in report.calls):,}")
+ lines.append(f"{'output tokens':<18} {sum(call.output_tokens for call in report.calls):,}")
if pricing.is_self_hosted(report.model):
- lines.append("total cost self-hosted (GPU time, not per-token)")
+ lines.append(f"{'total cost':<18} self-hosted (GPU time, not per-token)")
else:
- lines.append(f"total cost ${report.total_cost:.6f}")
- lines.append(f"schema retries {report.total_schema_retries}")
- lines.append(f"transient retries {report.total_transient_retries}")
+ lines.append(f"{'total cost':<18} ${report.total_cost:.6f}")
+ lines.append(f"{'schema retries':<18} {report.total_schema_retries}")
+ lines.append(f"{'transient retries':<18} {report.total_transient_retries}")
served = report.served_by
if served and served != {report.model} and not pricing.is_self_hosted(report.model):
lines.append(
- f"WARNING requested {report.model!r} but server served "
+ f"{'WARNING':<18} requested {report.model!r} but server served "
f"{sorted(served)} -- cost figures use the requested model's price"
)
lines.append("")
diff --git a/packages/levbench/src/levbench/snake/ui.py b/packages/levbench/src/levbench/snake/ui.py
index 741ff14..5765639 100644
--- a/packages/levbench/src/levbench/snake/ui.py
+++ b/packages/levbench/src/levbench/snake/ui.py
@@ -208,7 +208,7 @@ def compose(game: dict, decision: dict, stats: dict) -> Canvas:
c.put(height - 2, left, "SPACE pause +/- speed R reset Q quit", MUTED)
elapsed = stats.get("elapsed", 0.0)
clock = f"{int(elapsed) // 60:02d}:{int(elapsed) % 60:02d}"
- c.put(height - 2, right, f"ESTIMATES BY THE MODEL {clock}", MUTED)
+ c.put(height - 2, right, f"{'ESTIMATES BY THE MODEL':<29}{clock}", MUTED)
return c
From 7dfdd05256b5a06529a9fece40c0a13b6cec1ec2 Mon Sep 17 00:00:00 2001
From: Abhinavexists
Date: Thu, 15 Oct 2026 09:17:48 +0530
Subject: [PATCH 28/39] fix(levbench): align the jev-vs-llm comparison columns
---
packages/levbench/src/levbench/cli.py | 11 +++++++----
1 file changed, 7 insertions(+), 4 deletions(-)
diff --git a/packages/levbench/src/levbench/cli.py b/packages/levbench/src/levbench/cli.py
index 4dee4d6..264ba99 100644
--- a/packages/levbench/src/levbench/cli.py
+++ b/packages/levbench/src/levbench/cli.py
@@ -104,15 +104,18 @@ def cmd_compare(args: argparse.Namespace) -> None:
cost_ratio = llm.total_cost / jev.total_cost if jev.total_cost else 0.0
speed_ratio = llm.pct(0.5) / jev.pct(0.5) if jev.pct(0.5) else 0.0
print(
- f"cost jev ${jev.total_cost:.6f} vs llm ${llm.total_cost:.6f}"
+ f"{'cost':<18} jev ${jev.total_cost:.6f} vs llm ${llm.total_cost:.6f}"
f" ({cost_ratio:.1f}x cheaper)"
)
print(
- f"p50 jev {jev.pct(0.5):.3f}s vs llm {llm.pct(0.5):.3f}s ({speed_ratio:.1f}x faster)"
+ f"{'p50':<18} jev {jev.pct(0.5):.3f}s vs llm {llm.pct(0.5):.3f}s"
+ f" ({speed_ratio:.1f}x faster)"
)
- print(f"schema retries jev {jev.total_schema_retries} vs llm {llm.total_schema_retries}")
print(
- f"transient retries jev {jev.total_transient_retries}"
+ f"{'schema retries':<18} jev {jev.total_schema_retries} vs llm {llm.total_schema_retries}"
+ )
+ print(
+ f"{'transient retries':<18} jev {jev.total_transient_retries}"
f" vs llm {llm.total_transient_retries}"
)
From a107f63f2ccf896f1d6055bc126fa3816f124cb3 Mon Sep 17 00:00:00 2001
From: Abhinavexists
Date: Thu, 15 Oct 2026 09:17:48 +0530
Subject: [PATCH 29/39] fix(levbench): take the jev model from the sdk default
---
packages/levbench/src/levbench/cli.py | 8 +++--
packages/levbench/src/levbench/pricing.py | 9 +++---
packages/levbench/src/levbench/runner.py | 8 +++--
packages/levbench/tests/fake_transport.py | 9 +++++-
packages/levbench/tests/test_offline.py | 39 ++++++++++++++++-------
5 files changed, 51 insertions(+), 22 deletions(-)
diff --git a/packages/levbench/src/levbench/cli.py b/packages/levbench/src/levbench/cli.py
index 264ba99..e7522b1 100644
--- a/packages/levbench/src/levbench/cli.py
+++ b/packages/levbench/src/levbench/cli.py
@@ -184,11 +184,15 @@ def cmd_confidence(args: argparse.Namespace) -> None:
client, model = runner.build_client("jev", args.model, args.base_url)
answers: list = []
+ served: set[str] = set()
for item in items:
- answers.extend(runner.call_once(client, item.state, questions).answers.values())
+ result = runner.call_once(client, item.state, questions)
+ answers.extend(result.answers.values())
+ served.add(result.served_by)
samples = confidence_id.collect(answers)
- print(f"model: {model} answers with a distribution: {len(samples)}\n")
+ shown = ", ".join(sorted(served)) or model
+ print(f"model: {shown} answers with a distribution: {len(samples)}\n")
print(confidence_id.format_fits(confidence_id.identify(samples)))
diff --git a/packages/levbench/src/levbench/pricing.py b/packages/levbench/src/levbench/pricing.py
index 1d341c6..6a97304 100644
--- a/packages/levbench/src/levbench/pricing.py
+++ b/packages/levbench/src/levbench/pricing.py
@@ -11,9 +11,10 @@ class Price:
output_per_mtok: float
+# Every Jev version bills at the published Jev rate unless PRICES names it.
+JEV = Price(0.042, 0.0)
+
PRICES: dict[str, Price] = {
- "jev-latest": Price(0.042, 0.0),
- "jev-1.13": Price(0.042, 0.0),
"claude-opus-5": Price(5.00, 25.00),
"claude-sonnet-5": Price(2.00, 10.00),
"claude-haiku-4-5": Price(1.00, 5.00),
@@ -21,7 +22,7 @@ class Price:
}
-# Unknown jev names use jev-latest rates; other unknown models have zero token cost.
+# Unknown non-Jev models are self-hosted and have zero token cost.
SELF_HOSTED = Price(0.0, 0.0)
@@ -33,7 +34,7 @@ def lookup(model: str) -> Price:
if model in PRICES:
return PRICES[model]
if model.startswith("jev"):
- return PRICES["jev-latest"]
+ return JEV
return SELF_HOSTED
diff --git a/packages/levbench/src/levbench/runner.py b/packages/levbench/src/levbench/runner.py
index b919b62..f4507c1 100644
--- a/packages/levbench/src/levbench/runner.py
+++ b/packages/levbench/src/levbench/runner.py
@@ -69,6 +69,8 @@ def served_by(self) -> set[str]:
DEFAULT_LOCAL_BASE_URL = "http://localhost:8000"
+# Label for Jev runs left on the SDK's default model; reports name the version served instead.
+JEV_DEFAULT = "jev"
# Modal's 4B cold start measured 20-55 s, against the SDK's 10 s default (ADR-022).
@@ -87,8 +89,8 @@ def build_client(
base_url = base_url or DEFAULT_LOCAL_BASE_URL
from typesafe_sdk import TypeSafeClient
- resolved = model or ("jev-latest" if backend == "jev" else "local")
- kwargs: dict[str, Any] = {"model": resolved}
+ resolved = model or (JEV_DEFAULT if backend == "jev" else "local")
+ kwargs: dict[str, Any] = {"model": None if resolved == JEV_DEFAULT else resolved}
if timeout is not None:
kwargs["timeout"] = timeout
elif backend == "lev":
@@ -191,7 +193,7 @@ def run_eval(
)
served = report.served_by
- if backend == "lev" and len(served) == 1:
+ if (backend == "lev" or model == JEV_DEFAULT) and len(served) == 1:
# The label was a placeholder; the server knows what it loaded.
report.model = served.pop()
diff --git a/packages/levbench/tests/fake_transport.py b/packages/levbench/tests/fake_transport.py
index 6882ab7..7a204db 100644
--- a/packages/levbench/tests/fake_transport.py
+++ b/packages/levbench/tests/fake_transport.py
@@ -6,6 +6,9 @@
import json
import httpx2
+from typesafe_sdk import constants
+
+JEV_VERSION = "jev-1.13.0"
def _rand(*parts: str) -> float:
@@ -53,6 +56,10 @@ def handler(request: httpx2.Request) -> httpx2.Response:
state_text = state if isinstance(state, str) else json.dumps(state, sort_keys=True)
state_sig = hashlib.sha256(state_text.encode()).hexdigest()[:16]
+ requested = body.get("model")
+ # Like the hosted API, serve a concrete version when asked for the SDK's default alias.
+ served = JEV_VERSION if requested in (None, constants.DEFAULT_MODEL) else requested
+
questions = body["questions"]
answers = {k: _answer(k, q, state_sig) for k, q in questions.items()}
@@ -63,7 +70,7 @@ def handler(request: httpx2.Request) -> httpx2.Response:
return httpx2.Response(
200,
json={
- "model": body.get("model", "jev-latest"),
+ "model": served,
"answers": answers,
"usage": {"input_tokens": input_tokens, "output_tokens": output_tokens},
},
diff --git a/packages/levbench/tests/test_offline.py b/packages/levbench/tests/test_offline.py
index cb01ac4..fb35dc0 100644
--- a/packages/levbench/tests/test_offline.py
+++ b/packages/levbench/tests/test_offline.py
@@ -12,7 +12,7 @@
import httpx2
import pytest
-from fake_transport import transport
+from fake_transport import JEV_VERSION, transport
from levbench import batching, confidence_id, metrics, runner
from levbench.pricing import cost_usd
from levbench.tasks import FileItem, dataset, detectable_difference, load_task_file
@@ -87,7 +87,7 @@ def system_one(self, state, questions):
def test_eval_runs_end_to_end_over_the_built_in_fixture() -> None:
items, questions = dataset()
- report = runner.run_eval(fake_client(), "jev", "jev-latest", items, questions)
+ report = runner.run_eval(fake_client(), "jev", runner.JEV_DEFAULT, items, questions)
assert len(report.calls) == len(items)
assert set(report.per_question) == set(questions)
assert report.total_cost > 0
@@ -98,7 +98,7 @@ def test_eval_runs_end_to_end_over_the_built_in_fixture() -> None:
def test_sweep_arithmetic_over_a_fixed_billing_shape() -> None:
"""Synthetic billing verifies sweep arithmetic; live billing requires a live sweep."""
state = (ROOT / "data" / "sample_policy.md").read_text()
- rows = batching.sweep(fake_client(), "jev-latest", state, [1, 2, 4, 8, 13])
+ rows = batching.sweep(fake_client(), runner.JEV_DEFAULT, state, [1, 2, 4, 8, 13])
assert rows[0].cost_ratio == pytest.approx(1.0, abs=0.01, rel=0), (
"at N=1 batched and split are the same call; ratio must be ~1"
@@ -108,18 +108,18 @@ def test_sweep_arithmetic_over_a_fixed_billing_shape() -> None:
assert rows[-1].cost_ratio > 5, (
f"at N=13 expected a large saving, got {rows[-1].cost_ratio:.1f}x"
)
- assert "batching sweep" in batching.format_sweep(rows, "jev-latest", len(state))
+ assert "batching sweep" in batching.format_sweep(rows, runner.JEV_DEFAULT, len(state))
@pytest.mark.parametrize("count", [0, -1])
def test_sweep_rejects_non_positive_question_counts(count) -> None:
with pytest.raises(ValueError, match="must be positive"):
- batching.sweep(None, "jev-latest", "state", [count])
+ batching.sweep(None, runner.JEV_DEFAULT, "state", [count])
def test_sweep_respects_an_explicitly_empty_bank() -> None:
with pytest.raises(ValueError, match="bank holds 0"):
- batching.sweep(None, "jev-latest", "state", [1], bank=[])
+ batching.sweep(None, runner.JEV_DEFAULT, "state", [1], bank=[])
@pytest.mark.parametrize("planted", confidence_id.CANDIDATES)
@@ -193,11 +193,26 @@ def test_local_base_url_never_forwards_the_real_key(monkeypatch) -> None:
assert "REAL-SECRET" not in str(local._config.api_key), "leaked the real key"
assert local._config.api_key == "local"
- hosted, _ = runner.build_client("jev", "jev-latest")
+ hosted, _ = runner.build_client("jev")
assert isinstance(hosted, TypeSafeClient)
assert hosted._config.api_key == "sk-REAL-SECRET-KEY", "hosted path lost its key"
+def test_an_unpinned_jev_client_takes_the_sdk_default(monkeypatch) -> None:
+ monkeypatch.setenv("TYPESAFE_API_KEY", "sk-test")
+ monkeypatch.setenv("TYPESAFE_DEFAULT_MODEL", "jev-1.12")
+ client, label = runner.build_client("jev")
+ assert label == runner.JEV_DEFAULT
+ assert client._config.default_model == "jev-1.12"
+
+
+def test_an_unpinned_jev_run_reports_the_version_served() -> None:
+ items, questions = dataset()
+ report = runner.run_eval(fake_client(), "jev", runner.JEV_DEFAULT, items, questions)
+ assert report.model == JEV_VERSION
+ assert "WARNING" not in runner.format_report(report)
+
+
def test_empty_local_key_env_var_falls_back(monkeypatch) -> None:
"""An empty key produces an invalid Bearer header; use the local fallback."""
monkeypatch.setenv("LEVBENCH_LOCAL_API_KEY", "")
@@ -235,7 +250,7 @@ def test_eval_runs_over_a_generated_task_file(tmp_path, question_type) -> None:
"""Exercise the task-file format produced by `lev data eval` for every primitive."""
items, questions = dataset(_write_task_file(tmp_path / "gen.json", question_type))
assert questions["q"].type == question_type
- report = runner.run_eval(fake_client(), "jev", "jev-latest", items, questions)
+ report = runner.run_eval(fake_client(), "jev", runner.JEV_DEFAULT, items, questions)
assert len(report.calls) == 5
assert set(report.per_question) == {"q"}
assert report.per_question["q"] is not None
@@ -361,7 +376,7 @@ def system_one(self, state, questions):
class TestCostAccounting:
def test_cost_is_per_million_tokens_at_list_price(self) -> None:
- assert cost_usd("jev-latest", 1_000_000, 0) == pytest.approx(0.042)
+ assert cost_usd(JEV_VERSION, 1_000_000, 0) == pytest.approx(0.042)
assert cost_usd("claude-opus-5", 1_000_000, 1_000_000) == pytest.approx(30.0)
assert cost_usd("Qwen/Qwen3.5-4B", 1_000_000, 1_000_000) == 0.0, "self-hosted is unmetered"
@@ -378,8 +393,8 @@ def test_missing_token_counts_raise_rather_than_count_as_free(self) -> None:
def test_a_substituted_model_is_flagged_in_the_report(self) -> None:
"""Report model substitution because pricing uses the requested model."""
- report = runner.EvalReport(backend="jev", model="jev-latest")
- report.calls.append(runner.CallResult({}, 0.3, "jev-1.13.0", 10, 1))
- assert "requested 'jev-latest' but server served ['jev-1.13.0']" in runner.format_report(
+ report = runner.EvalReport(backend="jev", model="jev-1.12")
+ report.calls.append(runner.CallResult({}, 0.3, JEV_VERSION, 10, 1))
+ assert "requested 'jev-1.12' but server served ['jev-1.13.0']" in runner.format_report(
report
)
From 32bb003701e195b9d12d70ed3c3b86a928d9ef5e Mon Sep 17 00:00:00 2001
From: Abhinavexists
Date: Thu, 15 Oct 2026 09:17:49 +0530
Subject: [PATCH 30/39] docs: move internal docstrings to why-comments, keep
public api docs
---
docs/charts/build.py | 3 ---
modal/app.py | 22 ++++------------
packages/lev/src/lev/batcher.py | 2 --
packages/lev/src/lev/calibrate.py | 8 ------
packages/lev/src/lev/cli.py | 1 -
packages/lev/src/lev/data/build.py | 2 --
packages/lev/src/lev/data/contamination.py | 3 ---
packages/lev/src/lev/data/export_eval.py | 2 --
packages/lev/src/lev/data/mixture.py | 4 +--
packages/lev/src/lev/data/s1bench.py | 8 ------
packages/lev/src/lev/data/sources.py | 21 ++++-----------
packages/lev/src/lev/data/splits.py | 3 +--
packages/lev/src/lev/labels.py | 2 --
packages/lev/src/lev/metrics.py | 2 +-
packages/lev/src/lev/model.py | 26 +++++++------------
packages/lev/src/lev/presentation.py | 9 -------
packages/lev/src/lev/prompt.py | 19 +++++---------
packages/lev/src/lev/readout/mode_b.py | 1 -
packages/lev/src/lev/release.py | 3 ---
packages/lev/src/lev/server.py | 1 -
packages/lev/src/lev/train/calibration_run.py | 5 ----
packages/lev/src/lev/train/checkpoints.py | 10 +++----
packages/lev/src/lev/train/collate.py | 12 ++-------
packages/lev/src/lev/train/evaluate.py | 6 +----
packages/lev/src/lev/train/loop.py | 23 ++++------------
packages/lev/src/lev/train/progress.py | 3 +--
packages/lev/tests/conftest.py | 13 ++++------
packages/lev/tests/test_adapter_scale.py | 1 -
packages/lev/tests/test_batcher.py | 13 +++-------
packages/lev/tests/test_cache_fork.py | 4 +--
packages/lev/tests/test_calibration.py | 3 +--
packages/lev/tests/test_collate.py | 4 +--
packages/lev/tests/test_contamination.py | 2 +-
packages/lev/tests/test_core.py | 10 +++----
packages/lev/tests/test_data_pipeline.py | 21 +++++++--------
packages/lev/tests/test_engine_forward.py | 5 ++--
packages/lev/tests/test_eval_export.py | 5 ++--
packages/lev/tests/test_evaluate.py | 2 +-
packages/lev/tests/test_loss.py | 2 +-
packages/lev/tests/test_packaging.py | 3 ---
packages/lev/tests/test_presentation.py | 15 +++++------
packages/lev/tests/test_progress.py | 7 +++--
packages/lev/tests/test_release.py | 3 +--
packages/lev/tests/test_s1bench.py | 4 +--
packages/lev/tests/test_training_loop.py | 17 +++++-------
packages/levbench/src/levbench/batching.py | 1 -
packages/levbench/src/levbench/cli.py | 2 --
.../levbench/src/levbench/confidence_id.py | 11 +-------
packages/levbench/src/levbench/metrics.py | 10 +++----
packages/levbench/src/levbench/runner.py | 4 +--
.../levbench/src/levbench/snake/policy.py | 1 -
packages/levbench/src/levbench/snake/ui.py | 4 +--
packages/levbench/src/levbench/tasks.py | 2 +-
packages/levbench/tests/test_offline.py | 21 +++++----------
packages/levbench/tests/test_snake.py | 5 ++--
55 files changed, 110 insertions(+), 286 deletions(-)
diff --git a/docs/charts/build.py b/docs/charts/build.py
index bee494b..78d68a2 100644
--- a/docs/charts/build.py
+++ b/docs/charts/build.py
@@ -30,7 +30,6 @@ def pct(v: float) -> str:
def bar_path(x0: float, x1: float, y: float, h: float, r: float = 4) -> str:
- """Square at the baseline, 4px rounded at the data end."""
if x1 - x0 <= 0.5:
return ""
r = min(r, (x1 - x0) / 2, h / 2)
@@ -62,7 +61,6 @@ def table(headers, rows) -> str:
def leaderboard(rows, vmin=0.2, vmax=0.8) -> str:
- """Ranked horizontal bars. Rows: dicts with name, value, var, meta, bold."""
label_w, meta_w, row_h, bar_h, top = 196, 150, 23, 13, 6
plot_w = W - label_w - meta_w - 40
x = lambda v: label_w + (v - vmin) / (vmax - vmin) * plot_w # noqa: E731
@@ -275,7 +273,6 @@ def mean(scores, key, subsets=SUBSETS):
def majority_rate(subset: str) -> float:
- """Accuracy of always answering the most common label."""
labels = definition(subset)["labels"]
return max(labels.values()) / sum(labels.values())
diff --git a/modal/app.py b/modal/app.py
index 7744e6a..f25bad0 100644
--- a/modal/app.py
+++ b/modal/app.py
@@ -309,9 +309,9 @@ def serve():
)
+# Default entrypoint: print the budget without spending it.
@app.local_entrypoint()
def main(preset: str = "4b", dry_run: bool = True):
- """Default entrypoint: print the budget without spending it."""
result = train.remote(preset=preset, dry_run=dry_run)
print(result)
@@ -394,6 +394,8 @@ def check_release(name: str = "4b") -> dict:
return {"health": health, "response": served}
+# `split` is in-distribution and favours full strength, so the scale is chosen by
+# leave-one-family-out over S1Bench (FINDINGS §19). `scales` only reports; it never selects.
@app.function(gpu="H100", volumes=VOLUMES, secrets=SECRETS, timeout=4 * 60 * 60)
def fit_at_adapter_scale(
name: str = "lev",
@@ -404,18 +406,6 @@ def fit_at_adapter_scale(
search_rows: int = 6000,
apply: bool = False,
) -> dict:
- """Refit the temperature profile with the adapter scaled to `scale`.
-
- Shrinking the adapter trades in-distribution accuracy for out-of-distribution
- robustness, so `split` cannot choose the scale: it is drawn from the training
- mixture, and accuracy there rises monotonically to scale 1.0 (0.6233 at 0.125,
- 0.8133 at 1.0). The scale is chosen instead by leave-one-family-out over S1Bench,
- which holds each family out of its own selection; that put it at 0.25 for 6 of 7
- families. Temperatures still belong on `split`, which is what this fits.
-
- Passing `scales` sweeps them on `search_rows` rows and reports in-distribution
- sensitivity, without selecting. `apply` writes the scale and profile into the release.
- """
import json
import random
@@ -503,8 +493,8 @@ def collect(scale: float, rows):
return out
+# Mirrors levbench's label rule so a sweep is comparable with a `levbench eval` run.
def _predicted_label(answer):
- """levbench's label rule, so a sweep is comparable with a `levbench eval` run."""
if answer.type == "noul":
return float(answer.noul) >= 0.5
if answer.type == "choice":
@@ -512,13 +502,13 @@ def _predicted_label(answer):
return max(answer.probabilities.items(), key=lambda kv: kv[1])[0]
+# Scale zero is the frozen-backbone control.
@app.function(gpu="H100", volumes=VOLUMES, secrets=SECRETS, timeout=2 * 60 * 60)
def diagnose_ordinal(
name: str = "lev",
scales: str = "0,0.25,0.5,0.75,1.0",
subsets: str = "summeval-consistency,summeval-relevance,helpsteer2",
) -> dict:
- """Sweep adapter scales on rating tasks; use scale zero as the frozen-backbone control."""
import collections
import json
@@ -562,7 +552,6 @@ def diagnose_ordinal(
@app.function(gpu="H100", volumes=VOLUMES, secrets=SECRETS, timeout=30 * 60)
def diagnose_candidates(model_id: str = "Qwen/Qwen3.5-4B-Base") -> dict:
- """Compare candidate embeddings alone, batched, reversed, and left-padded."""
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
@@ -666,7 +655,6 @@ def _probabilities(answer: Answer) -> dict[Any, float]:
@app.function(gpu="H100", volumes=VOLUMES, secrets=SECRETS, timeout=30 * 60)
def profile_engine(preset: str = "4b", rounds: int = 20) -> dict:
- """Time CUDA-synchronized engine stages after warmup, excluding network time."""
import statistics
import time
diff --git a/packages/lev/src/lev/batcher.py b/packages/lev/src/lev/batcher.py
index 5c2c207..c8eb954 100644
--- a/packages/lev/src/lev/batcher.py
+++ b/packages/lev/src/lev/batcher.py
@@ -174,8 +174,6 @@ def _fill(self, batch: list[_Job]) -> None:
rows, width = grown_rows, grown_width
def _take(self, block: bool) -> _Job | None:
- """Return the next request worth computing."""
-
while True:
if self._held is not None:
job, self._held = self._held, None
diff --git a/packages/lev/src/lev/calibrate.py b/packages/lev/src/lev/calibrate.py
index 3d8074a..5b85e20 100644
--- a/packages/lev/src/lev/calibrate.py
+++ b/packages/lev/src/lev/calibrate.py
@@ -83,8 +83,6 @@ def fit_temperature(
def family_weights(families: Sequence[str]) -> list[float]:
- """Give each family equal total weight."""
-
counts = Counter(families)
return [1.0 / counts[family] for family in families]
@@ -101,8 +99,6 @@ def leave_one_family_out_ece(
families: Sequence[str],
method: str,
) -> float:
- """Measure transfer ECE by holding out each family once."""
-
eces = []
for held_out in sorted(set(families)):
@@ -131,8 +127,6 @@ def fit_for_transfer(
split_name: str,
min_samples: int = 50,
) -> tuple[CalibrationProfile, dict[str, dict]]:
- """Choose the row- or family-weighted fit that transfers better per bucket."""
-
_refuse_held_out(split_name)
profile = CalibrationProfile(fitted_on=f"{split_name} (transfer-selected)")
@@ -288,8 +282,6 @@ def fit(
split_name: str,
min_samples: int = 50,
) -> CalibrationProfile:
- """Fit one temperature per sufficiently large bucket."""
-
_refuse_held_out(split_name)
profile = CalibrationProfile(fitted_on=split_name)
diff --git a/packages/lev/src/lev/cli.py b/packages/lev/src/lev/cli.py
index 93b9c43..f420054 100644
--- a/packages/lev/src/lev/cli.py
+++ b/packages/lev/src/lev/cli.py
@@ -247,7 +247,6 @@ def _weights_dtype(engine) -> str | None:
def cmd_presentation_checks(args: argparse.Namespace) -> None:
- """Run presentation checks; exit 2 for settings mismatches and 1 for failed checks."""
import time
from . import presentation
diff --git a/packages/lev/src/lev/data/build.py b/packages/lev/src/lev/data/build.py
index 2612786..f742175 100644
--- a/packages/lev/src/lev/data/build.py
+++ b/packages/lev/src/lev/data/build.py
@@ -74,7 +74,6 @@ def read_jsonl(path: Path) -> list[Example]:
def verify_round_trip(path: Path, sample: int = 1000) -> int:
- """Check sampled rows round-trip exactly, including option order; return the count."""
rows = read_jsonl(path)
step = max(1, len(rows) // sample)
with path.open(encoding="utf-8") as handle:
@@ -104,7 +103,6 @@ def build_dataset(
cache_dir: str | None = None,
loader: Callable | None = None,
) -> dict:
- """Write each split's JSONL and the manifest to `out_dir`; return the manifest."""
out = Path(out_dir)
out.mkdir(parents=True, exist_ok=True)
weights = sources or default_weights()
diff --git a/packages/lev/src/lev/data/contamination.py b/packages/lev/src/lev/data/contamination.py
index f49236a..6e9e09b 100644
--- a/packages/lev/src/lev/data/contamination.py
+++ b/packages/lev/src/lev/data/contamination.py
@@ -70,7 +70,6 @@ class ContaminationError(RuntimeError):
def normalise(name: str) -> str:
- """Canonicalise a dataset name for lookup."""
cleaned = name.strip().lower()
cleaned = re.sub(r"[\s_]+", "-", cleaned)
return re.sub(r":(train|validation|dev|test)$", "", cleaned)
@@ -87,7 +86,6 @@ def normalise(name: str) -> str:
def resolve(name: str) -> str | None:
- """Map a dataset name to the blocked subset it belongs to, if any."""
normalised = normalise(name)
if normalised in _BLOCKED_BY_NORMALISED:
@@ -118,7 +116,6 @@ def check_mixture(dataset_names: Iterable[str]) -> dict[str, str]:
def assert_eval_only(dataset_name: str) -> str:
- """Resolve an evaluation subset or raise ContaminationError."""
subset = resolve(dataset_name)
if subset is None:
raise ContaminationError(
diff --git a/packages/lev/src/lev/data/export_eval.py b/packages/lev/src/lev/data/export_eval.py
index a8fc3af..1cc15f0 100644
--- a/packages/lev/src/lev/data/export_eval.py
+++ b/packages/lev/src/lev/data/export_eval.py
@@ -15,7 +15,6 @@
def truth_for(question: Question, target: int) -> bool | int | str:
- """Convert a target to a Choice key, Score index, or Noul bool."""
if isinstance(question, Noul):
# The target is a rating index; collapse it as the readout does.
return noul_probability({target: 1.0}) >= 0.5
@@ -30,7 +29,6 @@ def export(
split: Split = Split.TEST,
limit_per_source: int | None = None,
) -> dict:
- """Write `.json` per source, plus an `index.json` describing them."""
out = Path(out_dir)
out.mkdir(parents=True, exist_ok=True)
diff --git a/packages/lev/src/lev/data/mixture.py b/packages/lev/src/lev/data/mixture.py
index a985fda..a97655c 100644
--- a/packages/lev/src/lev/data/mixture.py
+++ b/packages/lev/src/lev/data/mixture.py
@@ -26,10 +26,9 @@ class Example:
soft_target: list[float] | None = None
+# Negated wordings flip the Noul target.
@dataclass(frozen=True)
class Augment:
- """Per-source wording and option variation; negations flip Noul targets."""
-
paraphrases: tuple[str, ...] = ()
negations: tuple[str, ...] = ()
min_options: int = 2
@@ -67,7 +66,6 @@ def validate(self) -> None:
def build_mixture(spec: MixtureSpec, loaders: Mapping[str, Loader]) -> Iterator[Example]:
- """Sample examples from validated sources using the configured weights and augmentations."""
spec.validate()
missing = set(spec.sources) - set(loaders)
if missing:
diff --git a/packages/lev/src/lev/data/s1bench.py b/packages/lev/src/lev/data/s1bench.py
index e81ee79..e4e77ed 100644
--- a/packages/lev/src/lev/data/s1bench.py
+++ b/packages/lev/src/lev/data/s1bench.py
@@ -24,8 +24,6 @@
@dataclass(frozen=True)
class EvalItem:
- """An evaluation record with a Choice key, Score index, or Noul bool as truth."""
-
id: str
state: dict | str
truth: bool | int | str
@@ -33,8 +31,6 @@ class EvalItem:
@dataclass(frozen=True)
class EvalSubset:
- """A subset with published Jev 1.13 and measured s1-fast scores."""
-
name: str
hf_id: str
jev_published: float
@@ -66,12 +62,10 @@ def subset_names() -> list[str]:
def get_subset(name: str) -> EvalSubset:
- """Resolve a subset by name or alias. Raises unless it is S1Bench evaluation data."""
return EVAL_SUBSETS[assert_eval_only(name)]
def definition(name: str) -> dict:
- """The vendored definition: `instructions`, `criteria`, `ids`, `labels`, provenance."""
return json.loads((SUBSET_DIR / f"{name}.json").read_text(encoding="utf-8"))
@@ -254,7 +248,6 @@ def _read_pubmedqa() -> Iterator[EvalItem]:
def load_eval_subset(name: str) -> tuple[Question, list[EvalItem]]:
- """Load a S1Bench subset, validating its records against pinned ids and labels."""
subset = get_subset(name)
spec = definition(subset.name)
by_id = {item.id: item for item in _READERS[subset.name]()}
@@ -273,7 +266,6 @@ def load_eval_subset(name: str) -> tuple[Question, list[EvalItem]]:
def export(out_dir: str | Path, subsets: list[str] | None = None) -> dict:
- """Write `.json` levbench task files, plus an `index.json`."""
out = Path(out_dir)
out.mkdir(parents=True, exist_ok=True)
diff --git a/packages/lev/src/lev/data/sources.py b/packages/lev/src/lev/data/sources.py
index 25b0606..982ae7a 100644
--- a/packages/lev/src/lev/data/sources.py
+++ b/packages/lev/src/lev/data/sources.py
@@ -34,8 +34,6 @@
@dataclass(frozen=True)
class SourceSpec:
- """A dataset and its question, label mapping and optional row reader."""
-
name: str
hf_id: str
primitive: Primitive
@@ -67,11 +65,10 @@ def _read_race(row: dict, rng: random.Random) -> ChoiceRow | None:
return {"passage": row["article"], "question": row["question"]}, options, index
+# Includes ARC's numeric choice labels, not only letters.
def _read_keyed_choices(
question_field: str, context_field: str | None = None
) -> Callable[[dict, random.Random], ChoiceRow | None]:
- """Match answerKey to choice labels, including numeric ARC labels."""
-
def reader(row: dict, rng: random.Random) -> ChoiceRow | None:
labels = [str(x) for x in row["choices"]["label"]]
texts = [str(x) for x in row["choices"]["text"]]
@@ -119,7 +116,6 @@ def _read_nli(row: dict, rng: random.Random) -> Row | None:
def swap_two_words(text: str, rng: random.Random) -> str | None:
- """Swap distinct content words, or return None if no pair is found in eight draws."""
words = text.split()
slots = [i for i, w in enumerate(words) if len(w) > 3 and w.isalpha()]
if len(slots) < 2:
@@ -136,8 +132,6 @@ def swap_two_words(text: str, rng: random.Random) -> str | None:
def _read_pair(
first: str, second: str, adversarial: float = 0.0
) -> Callable[[dict, random.Random], list[PairRow]]:
- """Read sentence pairs, optionally adding word-swapped negatives to positive rows."""
-
def reader(row: dict, rng: random.Random) -> list[PairRow]:
a, b = row[first], row[second]
label = int(row["label"])
@@ -179,8 +173,6 @@ def _read_strategyqa(row: dict, rng: random.Random) -> PairRow:
def yes_no_from_choices(reader: Reader) -> Callable[[dict, random.Random], list[PairRow] | None]:
- """Turn each multiple-choice row into two Noul rows: gold and wrong proposals."""
-
def wrapped(row: dict, rng: random.Random) -> list[PairRow] | None:
produced = reader(row, rng)
if produced is None:
@@ -610,7 +602,6 @@ def _read_snips(row: dict, rng: random.Random) -> Row | None:
def family_of(source: str) -> str:
- """Return the adjacent-source family used for calibration transfer (ADR-028)."""
for group in ADJACENT:
if source in group:
return "+".join(sorted(group))
@@ -618,20 +609,19 @@ def family_of(source: str) -> str:
def adjacency_map() -> dict[str, frozenset[str]]:
- """Map sources to those excluded as abstain donors."""
return {
name: frozenset().union(*(group for group in ADJACENT if name in group)) - {name}
for name in REGISTRY
}
+# Rating 1 would mean P(yes)=0.125, so binary labels map to ratings 0 and 8.
def _noul_rating(label: int) -> int:
- """Map binary labels to ratings 0 and 8; rating 1 would mean P(yes)=0.125."""
return 0 if int(label) == 0 else len(NOUL_RATING_TOKENS) - 1
+# Leave names such as Sci/Tech intact.
def humanise(label: str) -> str:
- """Expand snake_case and camelCase labels; leave names such as Sci/Tech intact."""
spaced = re.sub(r"(?<=[a-z])(?=[A-Z])", " ", label)
return re.sub(r"[_\-]+", " ", spaced).strip()
@@ -665,6 +655,7 @@ def build_question(spec: SourceSpec, names: list[str]) -> Question:
)
+# Shuffle before the limit: corpora often ship sorted by label, so a head slice can hold one class.
def load_source(
spec: SourceSpec,
limit: int | None = None,
@@ -673,7 +664,6 @@ def load_source(
seed: int = 17,
load_dataset: Callable | None = None,
) -> Iterator[Example]:
- """Yield examples through an injectable loader, shuffling before applying the limit."""
if load_dataset is None:
from datasets import load_dataset
@@ -746,8 +736,8 @@ def _blank(state) -> bool:
LARGE_SET_MIN_OPTIONS = 15
+# Large taxonomies keep their full option set half the time (ADR-026).
def augmentation_map() -> dict[str, Augment]:
- """Configure option variation; large taxonomies retain full sets half the time (ADR-026)."""
return {
name: Augment(
paraphrases=spec.paraphrases,
@@ -760,7 +750,6 @@ def augmentation_map() -> dict[str, Augment]:
def default_weights() -> dict[str, float]:
- """Reserve MODE_B_SHARE for large taxonomies; balance the remainder across primitives."""
mode_b_sources = sorted(MODE_B_SOURCES)
sources_by_primitive: dict[str, list[str]] = {}
for name, spec in REGISTRY.items():
diff --git a/packages/lev/src/lev/data/splits.py b/packages/lev/src/lev/data/splits.py
index 2d09c2a..0bfa500 100644
--- a/packages/lev/src/lev/data/splits.py
+++ b/packages/lev/src/lev/data/splits.py
@@ -32,8 +32,8 @@ def row_key(source: str, index: int, text: str) -> str:
return f"{source}|{index}|{text[:512]}"
+# Python's hash() is salted per process; splits must stay stable across runs.
def hash_position(key: str, salt: str = SPLIT_SALT) -> float:
- """Map a key to a process-stable float in [0, 1)."""
digest = hashlib.blake2b(f"{salt}|{key}".encode(), digest_size=8).digest()
return int.from_bytes(digest, "big") / float(1 << 64)
@@ -84,7 +84,6 @@ def split_examples(
def check_coverage(splits: dict[Split, list[Example]], strict: bool = True) -> SplitReport:
- """Check observed labels appear in train and offered labels appear in the data."""
seen: dict[str, set[int]] = defaultdict(set)
in_train: dict[str, set[int]] = defaultdict(set)
offered: dict[str, int] = {}
diff --git a/packages/lev/src/lev/labels.py b/packages/lev/src/lev/labels.py
index 8439649..eed2bdb 100644
--- a/packages/lev/src/lev/labels.py
+++ b/packages/lev/src/lev/labels.py
@@ -27,7 +27,6 @@ def label_codes(n: int) -> list[str]:
def single_token_codes(
tokenizer, n: int, prefix: str = " ", skip_multi_token: bool = False
) -> list[str] | None:
- """Find n single-token codes using the prompt boundary prefix; optionally skip split codes."""
if not skip_multi_token:
try:
codes = label_codes(n)
@@ -60,6 +59,5 @@ def _is_single_token(tokenizer, text: str) -> bool:
def noul_probability(level_probs: dict[int, float]) -> float:
- """Collapse a 9-level rating distribution to p(yes) = sum(i/8 * p_i)."""
top = len(NOUL_RATING_TOKENS) - 1
return sum((level / top) * p for level, p in level_probs.items())
diff --git a/packages/lev/src/lev/metrics.py b/packages/lev/src/lev/metrics.py
index 3a43203..b48bfe7 100644
--- a/packages/lev/src/lev/metrics.py
+++ b/packages/lev/src/lev/metrics.py
@@ -6,8 +6,8 @@
from collections.abc import Sequence
+# Multiclass Brier: squared error summed over the whole vector.
def brier(probs: Sequence[float], truth: int) -> float:
- """Multiclass Brier score: summed squared error over the whole vector."""
return sum((p - (1.0 if i == truth else 0.0)) ** 2 for i, p in enumerate(probs))
diff --git a/packages/lev/src/lev/model.py b/packages/lev/src/lev/model.py
index 145e4a2..39602d0 100644
--- a/packages/lev/src/lev/model.py
+++ b/packages/lev/src/lev/model.py
@@ -62,8 +62,8 @@ class EngineConfig:
pad_to: int = 32
+# Rounding up to a multiple gives graph capture a stable set of shapes.
def bucket(n: int, multiple: int) -> int:
- """`n` rounded up to a multiple; a stable set of shapes for graph capture."""
return max(multiple, ((n + multiple - 1) // multiple) * multiple)
@@ -73,8 +73,6 @@ def pow2(n: int) -> int:
@dataclass(frozen=True)
class Variant:
- """One rendering of one question: the candidate order shown, and its suffix."""
-
name: str
order: list[int] | None
suffix_ids: list[int]
@@ -82,8 +80,6 @@ class Variant:
@dataclass(frozen=True)
class Prepared:
- """One request validated, routed and tokenised, ready to share a forward."""
-
questions: dict[str, Question]
routes: dict[str, Route]
prefix_ids: list[int]
@@ -102,8 +98,8 @@ def want_hidden(self) -> bool:
return any(r.mode is Mode.CANDIDATE_PATH for r in self.routes.values())
+# Average probabilities, not logits.
def average_orders(prob_rows: list[list[float]], orders: list[list[int] | None]) -> list[float]:
- """Restore canonical candidate order and average probabilities, not logits."""
if len(prob_rows) != len(orders):
raise ValueError("one order per probability row")
k = len(prob_rows[0])
@@ -114,10 +110,10 @@ def average_orders(prob_rows: list[list[float]], orders: list[list[int] | None])
return [t / len(prob_rows) for t in total]
+# Shared with `lev route`, so its preview matches serving.
def serving_routes(
questions: dict[str, Question], tokenizer, config: EngineConfig
) -> dict[str, Route]:
- """Resolve the serving routes, also exposed by `lev route`."""
return route_all(
questions,
tokenizer,
@@ -288,8 +284,8 @@ def _respond(self, request: Prepared, logits, hidden) -> SystemOneResponse:
),
)
+ # Score levels stay canonical unless Score averaging is enabled (ADR-029).
def _orders(self, question: Question, route: Route) -> list[list[int] | None]:
- """Choose candidate orders; keep scales canonical unless Score averaging is enabled."""
if (
not self.config.order_average
or route.mode is not Mode.LABEL_TOKEN
@@ -344,8 +340,8 @@ def _render(
for name, order, r in rendered
]
+ # Last-token logits (N, V) and optional hidden states (N, H), in request order.
def _forward(self, requests: list[Prepared]):
- """Return last-token logits (N, V) and optional hidden states (N, H) in request order."""
import torch
want_hidden = any(r.want_hidden for r in requests)
@@ -364,7 +360,6 @@ def _forward(self, requests: list[Prepared]):
return logits, hidden
def _forward_single(self, rows: list[list[int]], want_hidden: bool):
- """Run right-padded prefix/suffix rows, using shape buckets when compiled."""
import torch
device = self._device
@@ -386,7 +381,6 @@ def _forward_single(self, rows: list[list[int]], want_hidden: bool):
return self._project(states, last_positions, want_hidden)
def _forward_forked(self, prefix_ids: list[int], suffixes: list[list[int]], want_hidden: bool):
- """Prefill the prefix once, then run all suffixes against forked caches."""
import torch
device = self._device
@@ -417,15 +411,14 @@ def _forward_forked(self, prefix_ids: list[int], suffixes: list[list[int]], want
return self._project(states, [len(s) - 1 for s in suffixes], want_hidden)
def _project(self, states, last_positions: list[int], want_hidden: bool):
- """Gather each row's last real token and project only those to the vocabulary."""
import torch
rows = torch.arange(states.size(0), device=states.device)
last = states[rows, torch.tensor(last_positions, device=states.device)] # (N, H)
return self._lm_head(last), (last if want_hidden else None)
+ # `logits` is (V,); `hidden` is (H,) or None.
def _scores(self, logits, hidden, route: Route, question: Question) -> list[float]:
- """Scores for one variant row: `logits` is `(V,)`, `hidden` `(H,)` or None."""
if route.mode is Mode.LABEL_TOKEN:
return self._label_token_scores(logits, route)
return self._candidate_path_scores(hidden, question)
@@ -458,8 +451,8 @@ def _candidate_path_scores(self, hidden, question: Question) -> list[float]:
)
return scores[0].float().tolist()
+ # Unbounded cache: meant for fixed schemas, whose candidate set does not grow.
def _candidate_reprs(self, texts: list[str]):
- """Cache candidate embeddings without a size bound; intended for fixed schemas."""
import torch
key = tuple(texts)
@@ -622,8 +615,8 @@ def load(
return engine
+# Scale before any merge, which bakes the current scale into the weights.
def _set_adapter_scale(model, scale: float) -> None:
- """Scale every LoRA layer relative to its trained strength, before any merge bakes it in."""
from peft.tuners.lora import LoraLayer
layers = [m for m in model.modules() if isinstance(m, LoraLayer)]
@@ -635,7 +628,6 @@ def _set_adapter_scale(model, scale: float) -> None:
def _load_head(path: Path, model):
- """Load the saved Mode B head, or return None for Mode A-only serving."""
import torch
from .readout.mode_b import CandidatePathReadout
@@ -651,8 +643,8 @@ def _load_head(path: Path, model):
return head.to(device=next(model.parameters()).device, dtype=torch.float32).eval()
+# Binary Noul calibration is kept separate from the trained rating scale.
def _bucket_type(question: Question, route: Route) -> str:
- """Keep binary Noul calibration separate from the trained rating scale."""
return "noul_binary" if route.reason == BINARY_NOUL else question.type
diff --git a/packages/lev/src/lev/presentation.py b/packages/lev/src/lev/presentation.py
index 34b9020..b033e96 100644
--- a/packages/lev/src/lev/presentation.py
+++ b/packages/lev/src/lev/presentation.py
@@ -65,7 +65,6 @@
def identical_questions(kind: str, text: str, k: int, instructions: str) -> dict[str, dict]:
- """The identical-option questions of one configuration, keyed by key assignment."""
if kind == "score":
return {"": {"type": "score", "instructions": instructions, "criteria": [text] * k}}
return {
@@ -79,7 +78,6 @@ def identical_questions(kind: str, text: str, k: int, instructions: str) -> dict
def probe_questions(lang: str = "en", kinds: Sequence[str] = KINDS) -> dict[str, dict]:
- """Build identical-option and permutation probes with structured question ids."""
questions: dict[str, dict] = {}
for kind in kinds:
spec = TEXT[lang][kind]
@@ -105,7 +103,6 @@ def is_identical(qid: str) -> bool:
@contextmanager
def configured(engine, **changes) -> Generator[None, None, None]:
- """Run the block with `engine.config` changed as given, then restore it."""
config = engine.config
engine.config = replace(config, **changes)
try:
@@ -119,7 +116,6 @@ def single_order(engine):
def call_groups(engine, questions: dict[str, dict], max_rows: int) -> list[dict[str, dict]]:
- """Group questions into calls within the engine's rendered-row budget."""
rows = engine.question_rows(questions)
groups: list[dict[str, dict]] = []
current: dict[str, dict] = {}
@@ -142,7 +138,6 @@ def ask(
kinds: Sequence[str] = KINDS,
max_rows: int | None = None,
) -> list[dict[str, Any]]:
- """Run single-order identical probes and configured permutation probes within max_rows."""
max_rows = max_rows or MAX_ROWS
questions = probe_questions(lang, kinds)
identical = {q: v for q, v in questions.items() if is_identical(q)}
@@ -165,7 +160,6 @@ def ask(
def slot_probs(answer: Any, question: dict) -> list[float]:
- """An answer's probabilities in the order the options were shown."""
probs = answer.probabilities
if question["type"] == "score":
return [float(probs[i]) for i in range(len(question["criteria"]))]
@@ -193,7 +187,6 @@ def report(
kinds: Sequence[str] = KINDS,
slot0_min: float = SLOT0_MIN,
) -> dict[str, dict]:
- """Both checks for each kind from `ask`'s answers."""
questions = probe_questions(lang, kinds)
out: dict[str, dict] = {}
for kind in kinds:
@@ -255,7 +248,6 @@ def packed_consistency(
lang: str = "en",
kinds: Sequence[str] = KINDS,
) -> float:
- """Return the largest probability difference between packed and individual probes."""
questions = probe_questions(lang, kinds)
worst = 0.0
for state, packed in zip(states, answers, strict=False):
@@ -282,7 +274,6 @@ def settings_mismatch(saved: dict, run: dict, keys: Sequence[str] = SETTINGS) ->
def drift(report: dict, snapshot: dict, tolerance: float) -> list[str]:
- """Every metric that moved more than `tolerance` from the committed report."""
moved = []
for kind, checks in snapshot.items():
for check, committed in checks.items():
diff --git a/packages/lev/src/lev/prompt.py b/packages/lev/src/lev/prompt.py
index 6e03c0e..e7aba43 100644
--- a/packages/lev/src/lev/prompt.py
+++ b/packages/lev/src/lev/prompt.py
@@ -18,15 +18,13 @@ class Layout(StrEnum):
SCHEMA_FIRST = "schema_first"
+# The prompt format must match training (ADR-027).
class Style(StrEnum):
- """Plain or ChatML prompts; the format must match training (ADR-027)."""
-
PLAIN = "plain"
CHAT = "chat"
def label_prefix(style: Style) -> str:
- """Use a space after plain Answer: and no prefix at the start of a chat answer."""
return " " if style is Style.PLAIN else ""
@@ -43,10 +41,9 @@ def label_prefix(style: Style) -> str:
}
+# The prefix is reusable and the suffix varies; the suffix's final token is the one scored.
@dataclass(frozen=True)
class Rendered:
- """Reusable prefix and varying suffix; score the suffix's final token."""
-
prefix: str
suffix: str
@@ -69,6 +66,7 @@ def _ordered(items: list, order: list[int] | None) -> list:
return items if order is None else [items[i] for i in order]
+# Two Noul codes select the binary readout.
def render_question(
name: str,
question: Question,
@@ -76,7 +74,6 @@ def render_question(
order: list[int] | None = None,
style: Style = Style.PLAIN,
) -> str:
- """Render positional label codes or Mode B text; two Noul codes select binary readout."""
if style is Style.CHAT:
return _render_question_chat(name, question, codes, order)
lines = [f"Question: {question.instructions or name}"]
@@ -119,7 +116,6 @@ def render_question(
def _render_question_chat(
name: str, question: Question, codes: list[str] | None, order: list[int] | None
) -> str:
- """Render ChatML criterion and option sections with positional codes."""
lines = [f"# Criterion\n{question.instructions or name}\n"]
if isinstance(question, Choice):
@@ -164,13 +160,13 @@ def _render_question_chat(
return "\n".join(lines)
+# Sorted object keys keep the rendering deterministic.
def render_content(value: JSONContent) -> str:
- """Render JSON content deterministically, sorting object keys."""
return value if isinstance(value, str) else json.dumps(value, sort_keys=True)
+# Shared by Mode B serving and training, so both see the same candidate text.
def candidate_texts(question: Question) -> list[str]:
- """Candidate text shared by the Mode B serving and training paths."""
if isinstance(question, Noul):
return list(NOUL_RATING_TOKENS)
if isinstance(question, Score):
@@ -188,7 +184,6 @@ def build(
order: list[int] | None = None,
style: Style = Style.PLAIN,
) -> Rendered:
- """Render one question against one state, cut for the chosen layout."""
if style is Style.CHAT:
return _build_chat(state, name, question, codes, layout, cached_schema, order)
state_block = f"Context:\n{render_state(state)}\n\n"
@@ -202,8 +197,8 @@ def build(
return Rendered(prefix=prefix, suffix=f"{state_block}{ANSWER_CUE}")
+# Score the boundary after the empty think block.
def _build_chat(state, name, question, codes, layout, cached_schema, order) -> Rendered:
- """Render ChatML evidence and criterion; score the boundary after the empty think block."""
head = f"<|im_start|>system\n{CHAT_SYSTEM_PROMPT}<|im_end|>\n<|im_start|>user\n"
evidence = f"# Evidence\n{render_state(state)}\n\n"
criterion = render_question(name, question, codes, order, Style.CHAT)
@@ -213,12 +208,12 @@ def _build_chat(state, name, question, codes, layout, cached_schema, order) -> R
return Rendered(prefix=f"{head}{catalogue}\n\n", suffix=f"{evidence.rstrip()}\n{CHAT_TAIL}")
+# The full question catalogue, so schema-first caching can share it across states.
def schema_block(
questions: Mapping[str, Question],
codes: Mapping[str, list[str] | None],
style: Style = Style.PLAIN,
) -> str:
- """The full question catalogue, for schema-first caching across states."""
return "\n\n".join(
render_question(n, q, codes.get(n), style=style) for n, q in questions.items()
)
diff --git a/packages/lev/src/lev/readout/mode_b.py b/packages/lev/src/lev/readout/mode_b.py
index 54410ef..46429d5 100644
--- a/packages/lev/src/lev/readout/mode_b.py
+++ b/packages/lev/src/lev/readout/mode_b.py
@@ -39,7 +39,6 @@ def forward(
def ordinal_penalty(scores: torch.Tensor, target_level: torch.Tensor) -> torch.Tensor:
- """Expected squared distance from the true level under the predicted distribution."""
n_levels = scores.size(-1)
levels = torch.arange(n_levels, device=scores.device, dtype=scores.dtype)
distance = (levels.unsqueeze(0) - target_level.unsqueeze(1).to(scores.dtype)) ** 2
diff --git a/packages/lev/src/lev/release.py b/packages/lev/src/lev/release.py
index e57d53e..dfbdcd8 100644
--- a/packages/lev/src/lev/release.py
+++ b/packages/lev/src/lev/release.py
@@ -23,7 +23,6 @@ def build_release(
metrics: dict | None = None,
prompt_style: str = "plain",
) -> dict:
- """Package a checkpoint; default to parent calibration, otherwise serve raw softmax."""
step_dir = resolve_checkpoint(checkpoint)
target = Path(out)
target.mkdir(parents=True, exist_ok=True)
@@ -63,7 +62,6 @@ def build_release(
def model_card(manifest: dict) -> str:
- """Build Hub metadata and usage instructions for a release."""
metrics = manifest.get("metrics") or {}
metric_lines = "\n".join(f"| {k} | {v} |" for k, v in metrics.items()) or "| — | not recorded |"
calibration = (
@@ -127,7 +125,6 @@ def model_card(manifest: dict) -> str:
def publish(release_dir: str | Path, repo_id: str, *, private: bool = False) -> str:
- """Upload a release using HF authentication; create the repo if needed and return its URL."""
from huggingface_hub import HfApi
folder = Path(release_dir)
diff --git a/packages/lev/src/lev/server.py b/packages/lev/src/lev/server.py
index 08236f3..807ea02 100644
--- a/packages/lev/src/lev/server.py
+++ b/packages/lev/src/lev/server.py
@@ -111,7 +111,6 @@ async def system_one(body: SystemOneRequest, request: Request) -> Any:
async def _disconnected(request: Request) -> None:
- """Wait for an ASGI disconnect; if none is reported, wait without polling."""
if (await request.receive())["type"] == "http.disconnect":
return
await asyncio.Event().wait()
diff --git a/packages/lev/src/lev/train/calibration_run.py b/packages/lev/src/lev/train/calibration_run.py
index 79a5137..43b5233 100644
--- a/packages/lev/src/lev/train/calibration_run.py
+++ b/packages/lev/src/lev/train/calibration_run.py
@@ -19,9 +19,6 @@ def fit_profile(
config=None,
method: str = "transfer",
) -> dict:
- """Fit bucket temperatures on calibration data; optionally select by family transfer.
-
- Save transfer comparisons and preserve the previous profile before replacing it."""
rows = collect_logits(checkpoint_dir, data_dir, split, config=config)
out = Path(checkpoint_dir) / CALIBRATION
if out.is_file():
@@ -68,8 +65,6 @@ def collect_logits(
batch_size: int = 16,
limit: int | None = None,
) -> dict[str, list[tuple[list[float], int, str]]]:
- """Collect raw logits and task families by calibration bucket, skipping abstain rows."""
-
from ..data.build import load_split
from ..data.sources import family_of
from ..data.splits import Split
diff --git a/packages/lev/src/lev/train/checkpoints.py b/packages/lev/src/lev/train/checkpoints.py
index 6325b77..fe1be76 100644
--- a/packages/lev/src/lev/train/checkpoints.py
+++ b/packages/lev/src/lev/train/checkpoints.py
@@ -10,8 +10,8 @@
CALIBRATION = "calibration.json"
+# Highest step number, not newest mtime.
def latest_checkpoint(output: str | Path) -> Path | None:
- """Return the highest numbered step with adapter weights, regardless of mtime."""
source = Path(output)
steps = sorted(
(d for d in source.glob("step-*") if (d / ADAPTER_WEIGHTS).is_file()),
@@ -21,7 +21,6 @@ def latest_checkpoint(output: str | Path) -> Path | None:
def fetch_checkpoint(spec: str | Path, cache_dir: str | None = None) -> Path:
- """Use an existing local path or download a Hub id; report ambiguous missing paths clearly."""
local = Path(spec)
if local.exists():
return local
@@ -40,7 +39,6 @@ def fetch_checkpoint(spec: str | Path, cache_dir: str | None = None) -> Path:
def resolve_checkpoint(path: str | Path, cache_dir: str | None = None) -> Path:
- """Resolve a step directory, training output, flat release, or Hub id."""
source = fetch_checkpoint(path, cache_dir)
if (source / ADAPTER_WEIGHTS).is_file():
return source
@@ -54,7 +52,6 @@ def resolve_checkpoint(path: str | Path, cache_dir: str | None = None) -> Path:
def load_training_state(path: str | Path) -> dict | None:
- """Load resume state, or return None for a weights-only checkpoint."""
import torch
file = resolve_checkpoint(path) / TRAINING_STATE
@@ -64,8 +61,9 @@ def load_training_state(path: str | Path) -> dict | None:
return torch.load(file, map_location="cpu", weights_only=True)
+# Restores in place so the optimizer keeps valid parameter references; head mismatches are
+# rejected.
def load_checkpoint(model, head, path: str | Path) -> None:
- """Restore weights in place to preserve optimizer references; reject head mismatches."""
import torch
from peft import set_peft_model_state_dict
from safetensors.torch import load_file
@@ -91,6 +89,7 @@ def load_checkpoint(model, head, path: str | Path) -> None:
)
+# An interrupted save may leave the checkpoint incomplete.
def save_checkpoint(
model,
head,
@@ -100,7 +99,6 @@ def save_checkpoint(
on_checkpoint=None,
state: dict | None = None,
) -> Path:
- """Write weights, tokenizer, and optional resume state; interrupted saves may be incomplete."""
import torch
path = output / f"step-{step}"
diff --git a/packages/lev/src/lev/train/collate.py b/packages/lev/src/lev/train/collate.py
index cf103bd..b04fe6b 100644
--- a/packages/lev/src/lev/train/collate.py
+++ b/packages/lev/src/lev/train/collate.py
@@ -18,9 +18,8 @@
IGNORE_INDEX = -100
+# Shared by the batcher and collator so each example is tokenised once.
class RouteCache:
- """Cache routes shared by the batcher and collator to avoid repeated tokenization."""
-
def __init__(self, tokenizer, max_label_options: int | None = None, style: Style = Style.PLAIN):
self.tokenizer = tokenizer
self.max_label_options = max_label_options
@@ -42,8 +41,6 @@ def route_for(self, example: Example) -> Route:
@dataclass
class Batch:
- """A training batch with one readout mode."""
-
mode: Mode
input_ids: Tensor # (B, T)
attention_mask: Tensor # (B, T)
@@ -78,8 +75,6 @@ def render(example: Example, codes: list[str] | None, style: Style = Style.PLAIN
class ModeBatcher:
- """Group by mode, sort within length windows, and shuffle batches without dropping examples."""
-
def __init__(
self,
tokenizer,
@@ -99,9 +94,9 @@ def __init__(
def route_for(self, example: Example) -> Route:
return self.routes.route_for(example)
+ # Character count stands in for length, to avoid tokenising twice.
@staticmethod
def length_of(example: Example) -> int:
- """Use character count as a length proxy to avoid tokenizing twice."""
return len(str(example.state))
def __call__(self, examples: Iterable[Example], epoch: int = 0) -> Iterator[list[Example]]:
@@ -131,8 +126,6 @@ def __call__(self, examples: Iterable[Example], epoch: int = 0) -> Iterator[list
class DecisionCollator:
- """Collate rows sharing one readout mode."""
-
def __init__(
self,
tokenizer,
@@ -231,7 +224,6 @@ def _label_token_ids(self, routes, k_max: int):
return ids
def _attach_candidate_pool(self, batch: Batch, examples: list[Example], k_max: int) -> None:
- """Encode each distinct candidate string once and index into the pool."""
import torch
pool: dict[str, int] = {}
diff --git a/packages/lev/src/lev/train/evaluate.py b/packages/lev/src/lev/train/evaluate.py
index f5d7dca..482ec94 100644
--- a/packages/lev/src/lev/train/evaluate.py
+++ b/packages/lev/src/lev/train/evaluate.py
@@ -11,7 +11,6 @@
def accuracy_interval(accuracy: float, n: int, z: float = 1.96) -> float:
- """Normal-approximation confidence interval half-width; 95% at the default z."""
if n <= 0:
return 1.0
return min(1.0, z * math.sqrt(max(accuracy * (1 - accuracy), 1e-9) / n))
@@ -69,14 +68,12 @@ def summary(self) -> str:
def score(
raw_logits: Sequence[tuple[list[float], int]], temperature: float
) -> tuple[float, float, float, float]:
- """Accuracy, ECE, mean Brier and mean log loss at one temperature."""
return score_mixed([(logits, truth, temperature) for logits, truth in raw_logits])
def score_mixed(
rows: Sequence[tuple[list[float], int, float]],
) -> tuple[float, float, float, float]:
- """`score` for rows that each carry their own temperature."""
probs = [softmax(logits, t) for logits, _, t in rows]
truths = [truth for _, truth, _ in rows]
correct = sum(
@@ -90,6 +87,7 @@ def score_mixed(
)
+# Raw and calibrated reports share the stored logits, so they differ only in temperature.
def evaluate_split(
checkpoint_dir: str,
data_dir: str,
@@ -99,8 +97,6 @@ def evaluate_split(
limit_per_source: int | None = None,
batch_size: int = 16,
) -> tuple[EvalReport, EvalReport]:
- """Return raw and calibrated reports from the same stored logits."""
-
from ..data.build import load_split
from ..data.splits import Split
from .checkpoints import resolve_checkpoint
diff --git a/packages/lev/src/lev/train/loop.py b/packages/lev/src/lev/train/loop.py
index 5abbd38..bcb9678 100644
--- a/packages/lev/src/lev/train/loop.py
+++ b/packages/lev/src/lev/train/loop.py
@@ -29,7 +29,6 @@
def build_model(config: TrainConfig, model_cache: str | None = None):
- """Load the tokenizer and backbone, wrapped in LoRA when `config.use_lora`."""
import torch
from peft import LoraConfig, get_peft_model
from transformers import AutoModelForCausalLM, AutoTokenizer
@@ -80,6 +79,8 @@ def build_model(config: TrainConfig, model_cache: str | None = None):
return model, tokenizer
+# logits (B, K), -inf at padded slots; targets (B,) gold indices, IGNORE_INDEX for abstain rows;
+# ordinal (B,) bool or a scalar bool; soft_targets (B, K) distributions for abstain rows.
def decision_loss(
logits,
targets,
@@ -87,14 +88,6 @@ def decision_loss(
ordinal: bool | torch.Tensor = False,
soft_targets=None,
):
- """Cross-entropy, Brier and ordinal loss over padded candidate sets.
-
- Args:
- logits: `(B, K)`, with `-inf` at padded slots.
- targets: `(B,)` gold indices; `IGNORE_INDEX` for abstain rows.
- ordinal: `(B,)` bool for ordered targets, or a scalar bool.
- soft_targets: `(B, K)` distributions for abstain rows.
- """
import torch
import torch.nn.functional as F
@@ -126,8 +119,8 @@ def decision_loss(
return loss
+# (B, K) scores: label logits in Mode A, pooled candidate embeddings in Mode B.
def candidate_logits(model, batch, head=None):
- """Compute (B, K) scores using label logits or pooled candidate embeddings."""
import torch
from ..router import Mode
@@ -173,8 +166,8 @@ def candidate_logits(model, batch, head=None):
return logits.float().masked_fill(batch.candidate_mask.to(logits.device), float("-inf"))
+# The head stays fp32 beside the bf16 backbone.
def build_head(config: TrainConfig, model=None):
- """The Mode B matching head, in fp32 beside the bf16 backbone."""
import torch
from ..readout.mode_b import CandidatePathReadout
@@ -189,7 +182,6 @@ def build_head(config: TrainConfig, model=None):
def load_for_eval(config: TrainConfig, checkpoint_dir, model_cache: str | None = None):
- """Build the model and head, restore a checkpoint into them, and put both in eval mode."""
model, tokenizer = build_model(config, model_cache)
head = build_head(config, model) if config.train_mode_b_head else None
load_checkpoint(model, head, checkpoint_dir)
@@ -200,7 +192,6 @@ def load_for_eval(config: TrainConfig, checkpoint_dir, model_cache: str | None =
def scored_rows(model, tokenizer, head, config: TrainConfig, rows, batch_size):
- """Yield `(example, logits, mode, width)` per row, batched as training batches."""
import torch
routes = RouteCache(tokenizer, config.max_label_options, Style(config.prompt_style))
@@ -232,8 +223,8 @@ def to_device(batch, device):
return batch
+# Validate before loading the model, so bad data fails fast.
def prepare_data(config: TrainConfig, data_dir: str):
- """Validate data and contamination before loading the model."""
from ..data.build import MANIFEST, load_split
from ..data.splits import Split
@@ -269,9 +260,6 @@ def run_training(
max_steps: int | None = None,
fresh: bool = False,
) -> dict:
- """Train with periodic checkpoints; auto-resume the newest step unless fresh=True.
-
- Resume optimizer, schedule, and data order when saved; otherwise restore weights only."""
import torch
from torch.optim import AdamW
from transformers import get_cosine_schedule_with_warmup
@@ -445,7 +433,6 @@ def training_state() -> dict:
def set_aside_previous_run(output: Path) -> Path | None:
- """Move previous run artifacts into a timestamped directory, or return None if empty."""
stale = list(output.glob("step-*")) + [
output / name for name in (HISTORY, CALIBRATION) if (output / name).exists()
]
diff --git a/packages/lev/src/lev/train/progress.py b/packages/lev/src/lev/train/progress.py
index 8277e2c..fc0e612 100644
--- a/packages/lev/src/lev/train/progress.py
+++ b/packages/lev/src/lev/train/progress.py
@@ -5,9 +5,8 @@
import time
+# Throughput counts real tokens, not padding.
class ProgressLog:
- """Flush periodic per-mode losses and throughput measured from real tokens."""
-
def __init__(self, total_steps: int, log_every: int):
self.total = total_steps
self.every = max(1, log_every)
diff --git a/packages/lev/tests/conftest.py b/packages/lev/tests/conftest.py
index 834b950..8fcb061 100644
--- a/packages/lev/tests/conftest.py
+++ b/packages/lev/tests/conftest.py
@@ -10,8 +10,6 @@
class FakeTokenizer:
- """Control which strings encode as a single token."""
-
def __init__(self, single_tokens: set[str]):
self.single_tokens = single_tokens
@@ -22,17 +20,17 @@ def encode(self, text: str, add_special_tokens: bool = True) -> list[int]:
return [1] * (len(text) + 1)
+# Space-prefixed A-Z and 0-8 are single tokens. Two-letter codes are not.
@pytest.fixture
def rich_tokenizer() -> FakeTokenizer:
- """Space-prefixed A-Z and 0-8 are single tokens. Two-letter codes are not."""
from string import ascii_uppercase
return FakeTokenizer({f" {c}" for c in ascii_uppercase} | {f" {i}" for i in range(9)})
+# Bare A-Z and 0-8 are single tokens too -- what follows an assistant turn.
@pytest.fixture
def chat_tokenizer() -> FakeTokenizer:
- """Bare A-Z and 0-8 are single tokens too -- what follows an assistant turn."""
from string import ascii_uppercase
letters = set(ascii_uppercase) | {f" {c}" for c in ascii_uppercase}
@@ -40,15 +38,14 @@ def chat_tokenizer() -> FakeTokenizer:
return FakeTokenizer(letters | digits)
+# Only the first four letters are single tokens, which forces Mode B quickly.
@pytest.fixture
def poor_tokenizer() -> FakeTokenizer:
- """Only the first four letters are single tokens — forces Mode B quickly."""
return FakeTokenizer({" A", " B", " C", " D"} | {f" {i}" for i in range(9)})
+# One id per character, to expose padding and last-token errors.
class BatchingTokenizer(FakeTokenizer):
- """Encode one id per character to expose padding and last-token errors."""
-
pad_token_id = 0
padding_side = "right"
@@ -88,9 +85,9 @@ def batching_tokenizer() -> BatchingTokenizer:
return BatchingTokenizer({f" {c}" for c in ascii_uppercase} | {f" {i}" for i in range(9)})
+# A throwaway copy of the smoke preset, safe for a test to mutate.
@pytest.fixture
def train_config():
- """A throwaway copy of the smoke preset, safe for a test to mutate."""
from dataclasses import replace
return replace(PRESETS["smoke"])
diff --git a/packages/lev/tests/test_adapter_scale.py b/packages/lev/tests/test_adapter_scale.py
index ca08469..6b9a6d6 100644
--- a/packages/lev/tests/test_adapter_scale.py
+++ b/packages/lev/tests/test_adapter_scale.py
@@ -33,7 +33,6 @@ def tiny_base():
def merged_delta(scale: float | None):
- """Weight the merge adds to `TARGET`, with the adapter scaled by `scale` first."""
base = tiny_base()
before = base.get_submodule(TARGET).weight.detach().clone()
diff --git a/packages/lev/tests/test_batcher.py b/packages/lev/tests/test_batcher.py
index de8a811..0206737 100644
--- a/packages/lev/tests/test_batcher.py
+++ b/packages/lev/tests/test_batcher.py
@@ -14,9 +14,8 @@ def request(name: str, rows: int = 1, width: int = 10):
return SimpleNamespace(name=name, rows=rows, width=width)
+# Holds the first forward so later requests queue behind it.
class GatedEngine:
- """Hold the first forward so later requests queue behind it."""
-
def __init__(self):
self.forwards: list[list[str]] = []
self.started = threading.Event()
@@ -33,7 +32,6 @@ def answer(self, requests):
async def submit_while_busy(batcher, engine, first, queued):
- """Occupy the worker with `first` and queue `queued` behind it."""
head = asyncio.ensure_future(batcher.submit(first))
while not engine.started.is_set():
await asyncio.sleep(0.001)
@@ -148,9 +146,8 @@ async def run():
assert after == "answer to c", "the worker survives a failed forward"
+# Every forward waits for one `step()`, so a test controls when each runs.
class SteppedEngine:
- """Every forward waits for one `step()`, so a test controls when each runs."""
-
def __init__(self, fail_with: BaseException | None = None, short_by: int = 0):
self.forwards: list[list[str]] = []
self.calls = 0
@@ -228,9 +225,8 @@ async def run():
assert asyncio.run(run()).forwards == [["a"], ["b"]]
+# Disconnecting does not release GPU capacity already in use.
def test_an_abandoned_running_request_still_counts_as_pending():
- """Disconnecting does not release GPU capacity already in use."""
-
async def run():
engine = SteppedEngine()
batcher = Batcher(engine)
@@ -264,9 +260,8 @@ async def run():
assert asyncio.run(run()) == "answer to a"
+# A short response list trips zip(strict=True) outside the forward's error handling.
def test_an_error_outside_the_forward_fails_its_batch_and_the_worker_keeps_serving():
- """A short response list triggers zip(strict=True) outside forward error handling."""
-
async def run():
engine = SteppedEngine(short_by=1)
batcher = Batcher(engine)
diff --git a/packages/lev/tests/test_cache_fork.py b/packages/lev/tests/test_cache_fork.py
index 999c64a..efa1a6f 100644
--- a/packages/lev/tests/test_cache_fork.py
+++ b/packages/lev/tests/test_cache_fork.py
@@ -12,8 +12,8 @@
BATCH, HEADS, SEQ, DIM = 1, 2, 5, 4
+# Shaped like Qwen3.5's cache: full-attention and linear layers mixed.
def hybrid_cache():
- """A cache shaped like Qwen3.5's: full-attention and linear layers mixed."""
full = cache_utils.DynamicLayer()
full.update(torch.randn(BATCH, HEADS, SEQ, DIM), torch.randn(BATCH, HEADS, SEQ, DIM))
@@ -62,8 +62,8 @@ def test_fork_does_not_mutate_the_source():
assert torch.equal(original.layers[1].conv_states[0], before_conv)
+# Hybrid caches do not support batch_repeat_interleave.
def test_the_old_approach_still_fails():
- """Hybrid caches do not support batch_repeat_interleave."""
linear = cache_utils.LinearAttentionLayer()
assert not hasattr(linear, "batch_repeat_interleave"), (
"LinearAttentionLayer now implements batch_repeat_interleave; "
diff --git a/packages/lev/tests/test_calibration.py b/packages/lev/tests/test_calibration.py
index 197a085..b79a80b 100644
--- a/packages/lev/tests/test_calibration.py
+++ b/packages/lev/tests/test_calibration.py
@@ -17,7 +17,6 @@
def overconfident_samples(n: int = 600, seed: int = 7, sharpness: float = 4.0):
- """Generate roughly 70% accuracy at 96% confidence."""
rng = random.Random(seed)
samples = []
for _ in range(n):
@@ -137,8 +136,8 @@ def test_banded_temperature_falls_back_to_the_unbanded_bucket(self):
class TestTransferSelectedCalibration:
+ # Large easy families outnumber a small hard family at equal confidence.
def family_rows(self):
- """Large easy families outnumber a small hard family at equal confidence."""
rows = []
for fam, n, acc in (
("easy1", 400, 0.97),
diff --git a/packages/lev/tests/test_collate.py b/packages/lev/tests/test_collate.py
index cc3912f..8ed1296 100644
--- a/packages/lev/tests/test_collate.py
+++ b/packages/lev/tests/test_collate.py
@@ -19,8 +19,8 @@ class TestCollator:
def collator(self, batching_tokenizer):
return DecisionCollator(batching_tokenizer, max_seq_len=512)
+ # Right padding makes seq - 1 point at a pad token.
def test_last_position_is_the_final_real_token_not_seq_minus_one(self, collator):
- """Right padding makes seq - 1 point to a pad token."""
batch = collator(
[an_example(a_choice(), state="short"), an_example(a_choice(), state="x" * 200)]
)
@@ -117,8 +117,6 @@ def test_batcher_and_collator_can_share_one_route_cache(self, batching_tokenizer
class TestLengthBucketing:
- """Test padding savings without global length ordering."""
-
def rows(self, n=512):
rng = random.Random(0)
# Bimodal, like the real mixture: short intents and long reviews.
diff --git a/packages/lev/tests/test_contamination.py b/packages/lev/tests/test_contamination.py
index 6c3185b..8e56a60 100644
--- a/packages/lev/tests/test_contamination.py
+++ b/packages/lev/tests/test_contamination.py
@@ -72,8 +72,8 @@ def test_aliases_and_variants_resolve(self, alias, expected):
def test_innocent_datasets_pass(self, innocent):
assert resolve(innocent) is None
+ # Includes subsets absent from the completed runs.
def test_subsets_that_did_not_run_are_blocked_too(self):
- """All evaluation subsets are blocked, including those absent from completed runs."""
unrun = [
"massive-de-DE",
"squad2",
diff --git a/packages/lev/tests/test_core.py b/packages/lev/tests/test_core.py
index 08c9afa..ebc042c 100644
--- a/packages/lev/tests/test_core.py
+++ b/packages/lev/tests/test_core.py
@@ -34,8 +34,8 @@
from lev.types import Choice, Noul, Score
+# Every space-prefixed letter and letter pair is one token, except `missing`.
def two_letter_tokenizer(missing: str | None = None) -> FakeTokenizer:
- """Every space-prefixed letter and letter pair is one token, except `missing`."""
pairs = {f" {a}{b}" for a in ascii_uppercase for b in ascii_uppercase}
if missing is not None:
pairs.discard(f" {missing}")
@@ -267,8 +267,8 @@ def test_binary_noul_is_read_both_ways(self, rich_tokenizer):
[1, 0],
]
+ # Training presents ordered levels from low to high.
def test_score_keeps_its_level_order(self, rich_tokenizer):
- """Training presents ordered levels from low to high."""
assert self.orders(Score(criteria=["low", "mid", "high"]), rich_tokenizer) == [None]
def test_rating_noul_keeps_its_scale(self, rich_tokenizer):
@@ -381,7 +381,6 @@ def test_route_uses_mode_a_past_the_split_only_when_asked(self):
class TestRoutePreviewMatchesServing:
def preview(self, tokenizer, request, tmp_path, monkeypatch, capsys) -> str:
- """Run the route CLI with an injected tokenizer."""
from lev.cli import main
stub = types.ModuleType("transformers")
@@ -431,16 +430,15 @@ def test_a_dict_question_is_parsed_and_routed(self, rich_tokenizer):
with pytest.raises(RuntimeError, match="need Mode B"):
engine.system_one("state", questions)
+ # The server maps ValueError to HTTP 422.
def test_a_malformed_dict_is_a_value_error(self, rich_tokenizer):
- """The server maps ValueError to HTTP 422."""
engine = self.engine(rich_tokenizer)
with pytest.raises(ValueError, match="at least 2 options"):
engine.system_one("state", {"pick": {"type": "choice", "criteria": {"a": None}}})
+# Concurrent access to a fast tokenizer can corrupt settings or raise Already borrowed.
class TestTokenizerIsUsedOneCallAtATime:
- """Concurrent access to a fast tokenizer can corrupt settings or raise Already borrowed."""
-
class OverlapDetector:
def __init__(self, inner):
self.inner = inner
diff --git a/packages/lev/tests/test_data_pipeline.py b/packages/lev/tests/test_data_pipeline.py
index 88576e1..86573af 100644
--- a/packages/lev/tests/test_data_pipeline.py
+++ b/packages/lev/tests/test_data_pipeline.py
@@ -53,8 +53,8 @@ def select(self, indices):
return FakeDataset([self.rows[i] for i in indices], self.features)
+# Grouped by label, which is how most real corpora ship.
def label_sorted_loader(n_labels: int, per_label: int, text_field="text", label_field="label"):
- """A corpus grouped by label -- which is how most of the real ones ship."""
rows = [
{text_field: f"row {label}-{i}", label_field: label}
for label in range(n_labels)
@@ -120,8 +120,8 @@ def test_choice_options_are_humanised(self):
class TestSampling:
+ # A head slice of a label-sorted corpus may contain only one class.
def test_limit_samples_rather_than_truncates(self):
- """A head slice of a label-sorted corpus may contain only one class."""
spec = REGISTRY["dbpedia_14"]
loader = label_sorted_loader(14, 100, text_field="content")
rows = list(load_source(spec, limit=140, load_dataset=loader))
@@ -133,8 +133,8 @@ def test_sampling_is_reproducible(self):
b = [e.state for e in load_source(REGISTRY["dbpedia_14"], 50, load_dataset=loader)]
assert a == b
+ # A binary label of 1 must become rating 8, not P(yes) = 0.125.
def test_noul_labels_land_on_the_ends_of_the_rating_scale(self):
- """A binary label of 1 must become rating 8, not P(yes) = 0.125."""
loader = label_sorted_loader(2, 20)
rows = list(load_source(REGISTRY["imdb"], load_dataset=loader))
assert {e.target for e in rows} == {0, len(NOUL_RATING_TOKENS) - 1}
@@ -155,8 +155,8 @@ def test_unnamed_label_feature_raises_rather_than_guessing(self):
list(load_source(REGISTRY["ag_news"], load_dataset=loader))
+# The state names its own source, so donor tests can trace it.
def traceable_example(source, target, index, n_options=4):
- """One row whose state names its own source, so donor tests can trace it."""
return an_example(
a_choice(n_options), target=target, source=source, state=f"{source} text {index}"
)
@@ -180,8 +180,8 @@ def test_splits_are_disjoint(self, examples):
assert not states[Split.TRAIN] & states[Split.CALIBRATION]
assert not states[Split.CALIBRATION] & states[Split.TEST]
+ # Python's hash() is salted per process.
def test_assignment_is_stable_across_processes(self):
- """Python hash() is salted per process."""
import os
import subprocess
import sys
@@ -212,8 +212,8 @@ def test_label_only_outside_train_raises(self):
with pytest.raises(ValueError, match="never in it"):
check_coverage(splits)
+ # Train coverage alone misses offered labels absent from every split.
def test_option_never_observed_at_all_raises(self):
- """Train coverage alone misses offered labels absent from every split."""
splits = {
Split.TRAIN: [traceable_example("s", 0, 1), traceable_example("s", 1, 2)],
Split.CALIBRATION: [],
@@ -222,8 +222,8 @@ def test_option_never_observed_at_all_raises(self):
with pytest.raises(ValueError, match="never show some of the options"):
check_coverage(splits)
+ # Binary data observes only the endpoints of the nine-level scale.
def test_noul_is_exempt_from_the_offered_option_check(self):
- """Binary data observes only the endpoints of the nine-level scale."""
rows = [
Example("t", "n", Noul(instructions="q"), t, Layout.STATE_FIRST, "imdb")
for t in (0, 8, 0, 8)
@@ -269,9 +269,8 @@ def test_the_threshold_is_the_single_letter_alphabet(self):
assert LABEL_OPTION_CAP == 26
+# Adjacent sources may still answer the question, so they are unsuitable abstain donors.
class TestAbstainDonors:
- """Adjacent sources may still answer the question and are unsuitable abstain donors."""
-
def test_adjacency_is_symmetric_and_excludes_self(self):
mapping = adjacency_map()
for source, related in mapping.items():
@@ -496,8 +495,8 @@ def test_noul_reader_targets_land_on_the_rating_ends(self):
assert [e.target for e in examples] == [len(NOUL_RATING_TOKENS) - 1, 0]
assert examples[0].state == {"sentence1": "a", "sentence2": "b"}
+ # Both polarities stop yes from always meaning a good outcome.
def test_every_noul_source_carries_a_negation(self):
- """Both polarities prevent yes from always meaning a good outcome."""
for name, spec in REGISTRY.items():
if spec.primitive == "noul":
assert spec.negations, f"{name} has no negated question"
@@ -721,8 +720,8 @@ def build(self, out):
loader=self.loader,
)
+ # Late-bound loader closures can cross source contents while keeping source labels.
def test_every_split_draws_from_every_source(self, tmp_path):
- """Late-bound loader closures can cross source contents while preserving source labels."""
from lev.data.build import SPLIT_FILES, read_jsonl
manifest = self.build(tmp_path)
diff --git a/packages/lev/tests/test_engine_forward.py b/packages/lev/tests/test_engine_forward.py
index e362ec7..4bfa231 100644
--- a/packages/lev/tests/test_engine_forward.py
+++ b/packages/lev/tests/test_engine_forward.py
@@ -49,9 +49,8 @@ def test_single_forward_matches_the_full_logits_at_each_last_token(tiny_model, w
assert (hidden is not None) == want_hidden
+# Ids inside the tiny vocabulary; space-prefixed A-Z are single tokens, as label codes need.
class TinyTokenizer:
- """Ids inside the tiny vocabulary; space-prefixed A-Z are single tokens, as label codes need."""
-
pad_token_id = 0
single = {f" {c}": 400 + i for i, c in enumerate("ABCDEFGHIJKLMNOPQRSTUVWXYZ")}
@@ -61,8 +60,8 @@ def encode(self, text, add_special_tokens=True):
)
+# The token limit never touches a request Score averaging did not grow (ADR-029).
def test_without_score_averaging_a_large_request_answers_as_with_no_limit(tiny_model):
- """The token limit never touches a request Score averaging did not grow (ADR-029)."""
from lev.types import Choice, Score
questions = {
diff --git a/packages/lev/tests/test_eval_export.py b/packages/lev/tests/test_eval_export.py
index 867be83..74f4caf 100644
--- a/packages/lev/tests/test_eval_export.py
+++ b/packages/lev/tests/test_eval_export.py
@@ -35,8 +35,8 @@ def write_test_split(directory, rows):
class TestRoundTrip:
+ # Sorting cache keys changes what positional targets refer to (ADR-024).
def test_same_options_in_a_different_order_are_different_questions(self, tmp_path):
- """Sorting cache keys changes what positional targets refer to (ADR-024)."""
forward = Choice(instructions="q", criteria={"a": None, "b": None, "c": None})
backward = Choice(instructions="q", criteria={"c": None, "b": None, "a": None})
rows = [an_example(forward, target=0, state="x"), an_example(backward, target=0, state="y")]
@@ -51,7 +51,6 @@ def test_same_options_in_a_different_order_are_different_questions(self, tmp_pat
assert back[0].question is not back[1].question
def test_round_trip_guard_catches_an_order_blind_reader(self, tmp_path, monkeypatch):
- """Inject an order-blind cache and require the guard to reject it."""
import lev.data.build as build
from lev.data.build import verify_round_trip
@@ -152,8 +151,8 @@ def test_an_eval_too_small_to_measure_anything_raises(self, tmp_path):
class TestVariableQuestions:
+ # One task file cannot represent different option sets per row.
def test_sources_whose_question_varies_per_row_are_skipped_and_listed(self, tmp_path):
- """One task file cannot represent different option sets per row."""
rows = [example(CHOICE, i % 2, i, source="a") for i in range(MIN_USEFUL_ITEMS + 10)]
rows += [
example(
diff --git a/packages/lev/tests/test_evaluate.py b/packages/lev/tests/test_evaluate.py
index 0d65d51..1ecd353 100644
--- a/packages/lev/tests/test_evaluate.py
+++ b/packages/lev/tests/test_evaluate.py
@@ -35,8 +35,8 @@ def test_interval_shrinks_with_n(self):
def test_matches_the_normal_approximation(self):
assert accuracy_interval(0.86, 200) == pytest.approx(0.048, abs=0.001)
+ # Perfect observed accuracy does not imply zero uncertainty.
def test_a_certain_estimate_still_reports_a_finite_interval(self):
- """Perfect observed accuracy does not imply zero uncertainty."""
assert 0.0 <= accuracy_interval(1.0, 200) < 0.01
def test_no_items_means_no_information(self):
diff --git a/packages/lev/tests/test_loss.py b/packages/lev/tests/test_loss.py
index d0669c8..01906cc 100644
--- a/packages/lev/tests/test_loss.py
+++ b/packages/lev/tests/test_loss.py
@@ -13,8 +13,8 @@
class TestDecisionLoss:
+ # Padded slots must avoid 0 * -inf, which produces NaN.
def test_padded_slots_do_not_produce_nan(self, train_config):
- """Padded slots must avoid 0 * -inf, which produces NaN."""
logits = torch.tensor(
[[1.0, 2.0, 0.5, 0.1], [0.3, 1.2, float("-inf"), float("-inf")]],
requires_grad=True,
diff --git a/packages/lev/tests/test_packaging.py b/packages/lev/tests/test_packaging.py
index b926f52..2048e2d 100644
--- a/packages/lev/tests/test_packaging.py
+++ b/packages/lev/tests/test_packaging.py
@@ -26,7 +26,6 @@ def _doc_sources() -> list[Path]:
def _documented_values(pattern: str) -> dict[str, set[str]]:
- """Map each value `pattern` captures to the documentation files that mention it."""
found: dict[str, set[str]] = {}
for src in _doc_sources():
for value in re.findall(pattern, src.read_text()):
@@ -64,7 +63,6 @@ def test_known_extras_are_present(expected):
def _inspect_cli(tool: str) -> tuple[set[str], dict[str, set[str]]]:
- """Inspect the live parser to collect accepted commands and flags."""
import argparse
import importlib
@@ -111,7 +109,6 @@ def test_every_documented_backend_exists():
def _documented_commands(tool: str) -> dict[str, set[str]]:
- """Extract commands from fenced blocks and inline code, excluding prose."""
found: dict[str, set[str]] = {}
pattern = re.compile(rf"(?:^|\s)(?:uv run )?{tool} ([a-z][a-z0-9-]*)", re.M)
for src in _doc_sources():
diff --git a/packages/lev/tests/test_presentation.py b/packages/lev/tests/test_presentation.py
index 830f091..35bd7d1 100644
--- a/packages/lev/tests/test_presentation.py
+++ b/packages/lev/tests/test_presentation.py
@@ -40,18 +40,16 @@ def softmax(z: list[float]) -> list[float]:
return [v / sum(e) for v in e]
+# Space-prefixed A-Z and 0-8 are single tokens, as in `rich_tokenizer`.
class Tokenizer:
- """Space-prefixed A-Z and 0-8 are single tokens, as `rich_tokenizer`."""
-
single = {f" {c}" for c in ascii_uppercase} | {f" {i}" for i in range(9)}
def encode(self, text: str, add_special_tokens: bool = True) -> list[int]:
return [1] if text in self.single else [1] * (len(text) + 1)
+# Real routing and orders, with scripted content, key, slot and packing effects.
class FakeEngine:
- """Use real routing and orders with scripted content, key, slot, and packing effects."""
-
def __init__(self, slot0=0.0, content=None, packing=0.0, key_pull=None, **config):
self.real = DecisionEngine(model=None, tokenizer=Tokenizer(), config=EngineConfig(**config))
self.slot0, self.content, self.packing = slot0, content or {}, packing
@@ -178,8 +176,8 @@ def test_a_planted_slot_zero_penalty_is_measured_exactly(self):
assert r[kind]["first_slot"]["argmax_by_slot"][0] == 0
assert r[kind]["first_slot"]["decisions"] == 18
+ # Argmax would credit slot zero for every flat distribution.
def test_ties_are_split_between_the_tied_slots(self):
- """Argmax would incorrectly credit slot zero for every flat distribution."""
r = run(FakeEngine(), ["a", "b"])
for kind in KINDS:
assert r[kind]["first_slot"]["metric"] == pytest.approx(1 / 3)
@@ -194,9 +192,9 @@ def test_a_choice_key_pull_cancels_and_a_position_effect_does_not(self):
alone["choice"]["identical"]["metric"]
)
+ # Order averaging hides the raw position bias this probe measures.
@pytest.mark.parametrize("mode", ["reversed", "cyclic"])
def test_the_identical_check_reads_one_order_whatever_the_engine_averages(self, mode):
- """Order averaging hides the raw position bias this probe measures."""
single = run(FakeEngine(slot0=-2.0, order_average=False), ["a"])
averaged = run(FakeEngine(slot0=-2.0, score_order_average=mode), ["a"])
for kind in KINDS:
@@ -409,8 +407,8 @@ def test_a_long_state_with_few_averaged_rows_is_refused(self, rich_tokenizer):
assert engine.prepare("short", question).rows == 3
assert self.engine(rich_tokenizer).prepare(LONG, question).rows == 1
+ # The batcher runs oversized requests alone.
def test_without_score_averaging_a_large_request_is_never_refused(self, rich_tokenizer):
- """The batcher runs oversized requests alone."""
prepared = self.engine(rich_tokenizer).prepare("x" * 400, BIG)
assert prepared.rows == 80 and prepared.rows * prepared.width > MAX_BATCH_TOKENS
assert [v.order for v in prepared.variants] == [None, [3, 2, 1, 0]] * 20 + [None] * 40
@@ -485,9 +483,8 @@ def test_it_parses_and_carries_every_metric(self):
assert drift(committed["checks"], committed["checks"], 0.0) == []
+# Settings mismatches exit 2; metric failures exit 1.
class TestCompareGuardsItsSettings:
- """Settings mismatches exit 2; metric failures exit 1."""
-
def cli(self, monkeypatch, tmp_path, engine, *extra):
from lev.cli import main
diff --git a/packages/lev/tests/test_progress.py b/packages/lev/tests/test_progress.py
index e57879b..efaf1c5 100644
--- a/packages/lev/tests/test_progress.py
+++ b/packages/lev/tests/test_progress.py
@@ -30,8 +30,8 @@ def test_always_logs_the_final_step(self, capsys):
log.record(step, 1.0, "A", 1e-4, tokens=10)
assert "step 7/7" in capsys.readouterr().out
+ # Mode B starts near ln(K), so averaging the modes hides their separate losses.
def test_modes_are_reported_separately(self, capsys):
- """Mode B starts near ln(K), so averaging modes hides their separate losses."""
log = self.make(total=2, every=2)
log.record(0, 1.0, "A", 1e-4, tokens=10)
log.record(1, 5.0, "B", 1e-4, tokens=10)
@@ -68,12 +68,11 @@ def test_hms_formats_hours(self):
assert hms(3661) == "1:01:01"
+# Windowed rates keep startup delay out of steady-state throughput and ETA.
class TestWindowedRate:
- """Windowed rates exclude startup delay from steady-state throughput and ETA."""
-
+ # Set `clock["t"]` to move the time `ProgressLog` reads.
@pytest.fixture
def clock(self, monkeypatch):
- """Set `clock["t"]` to move the time `ProgressLog` reads."""
now = {"t": 0.0}
monkeypatch.setattr(progress.time, "monotonic", lambda: now["t"])
return now
diff --git a/packages/lev/tests/test_release.py b/packages/lev/tests/test_release.py
index 2b890dc..3aa5f19 100644
--- a/packages/lev/tests/test_release.py
+++ b/packages/lev/tests/test_release.py
@@ -10,7 +10,6 @@
def fake_checkpoint(root, step=18750, calibrated=True):
- """Create placeholder files matching a saved checkpoint and calibration."""
out = root / "4b"
step_dir = out / f"step-{step}"
step_dir.mkdir(parents=True)
@@ -96,8 +95,8 @@ def test_a_missing_path_that_is_not_a_hub_id_fails_clearly(self, spec):
with pytest.raises(FileNotFoundError, match="neither a local path nor a Hub id"):
fetch_checkpoint(spec)
+ # A missing relative path may also parse as a Hub id.
def test_a_mistyped_local_path_that_looks_like_a_hub_id_fails_clearly(self, monkeypatch):
- """A missing relative path may also parse as a Hub id."""
# Both ship with the `train` extra, absent on a bare `uv sync`.
huggingface_hub = pytest.importorskip("huggingface_hub")
httpx = pytest.importorskip("httpx")
diff --git a/packages/lev/tests/test_s1bench.py b/packages/lev/tests/test_s1bench.py
index bd7eb1b..4355338 100644
--- a/packages/lev/tests/test_s1bench.py
+++ b/packages/lev/tests/test_s1bench.py
@@ -45,8 +45,8 @@ def test_training_sources_are_refused(self, name: str) -> None:
class TestStructuralSeparation:
+ # A fresh interpreter, so imports already made by the test suite cannot mask dependencies.
def test_s1bench_cannot_reach_the_mixture_builder(self) -> None:
- """Use a fresh interpreter so existing test-suite imports do not mask dependencies."""
probe = (
"import sys; import lev.data.s1bench; "
"leaked = sorted(m for m in sys.modules "
@@ -98,9 +98,9 @@ def test_definition_is_self_consistent_and_names_its_source(self, name: str) ->
assert spec["source"].startswith("https://github.com/bespokelabsai/nimble/blob/")
assert s1bench.question_for(spec).type == EXPECTED_TYPES[name]
+ # The pinned helpsteer2 filter yields 249 records, one fewer than the board.
@pytest.mark.parametrize("name", sorted(EVAL_SUBSETS))
def test_record_counts_match_the_board(self, name: str) -> None:
- """The pinned helpsteer2 filter yields 249 records, one fewer than the board."""
board = jev_board(snapshot())[name]
expected = board["total"] - (1 if name == "helpsteer2" else 0)
assert definition(name)["count"] == expected
diff --git a/packages/lev/tests/test_training_loop.py b/packages/lev/tests/test_training_loop.py
index f64b09a..4820982 100644
--- a/packages/lev/tests/test_training_loop.py
+++ b/packages/lev/tests/test_training_loop.py
@@ -36,8 +36,8 @@ def make_batching_tokenizer():
)
+# The 64 rows are 4-option Choices, so every batch takes Mode A.
def stub_backbone(monkeypatch, loop):
- """The 64 rows are 4-option Choices, so every batch takes Mode A."""
rows = [an_example(a_choice(4), target=i % 4, state=f"row {i}") for i in range(64)]
monkeypatch.setattr(loop, "prepare_data", lambda c, d: {"train": rows, "calibration": []})
monkeypatch.setattr(
@@ -72,7 +72,6 @@ def smoke_config(tmp_path, *, epochs, checkpoint_every):
def run_resumable(
tmp_path, monkeypatch, *, max_steps, checkpoint_every=3, fresh=False, persist=True
):
- """Run with on-disk checkpoints; return saved steps, observed batches, and summary."""
import lev.train.loop as loop
from lev.train.checkpoints import TRAINING_STATE
from lev.train.collate import DecisionCollator
@@ -104,8 +103,6 @@ def fake_save(m, h, tk, out, step, cb=None, state=None):
class TestCheckpointCadence:
- """Exercise real batching and checkpoint decisions with model and storage stubs."""
-
def run(self, tmp_path, monkeypatch, *, max_steps, checkpoint_every, epochs=1):
import lev.train.loop as loop
@@ -165,15 +162,15 @@ def test_fresh_ignores_the_checkpoint(self, tmp_path, monkeypatch):
assert saved == [3, 6]
assert summary["history"][0]["step"] == 0
+ # Weight loading is stubbed; this checks the schedule restart only.
def test_weights_only_checkpoint_restarts_the_schedule(self, tmp_path, monkeypatch):
- """Weight loading is stubbed; this checks schedule restart only."""
run_resumable(tmp_path, monkeypatch, max_steps=6, persist=False)
saved, _, summary = run_resumable(tmp_path, monkeypatch, max_steps=6)
assert saved == [3, 6]
assert summary["history"][0]["step"] == 0
+ # An off-interval final checkpoint must carry resume state.
def test_rerunning_a_finished_run_trains_nothing(self, tmp_path, monkeypatch):
- """An off-interval final checkpoint must carry resume state."""
run_resumable(tmp_path, monkeypatch, max_steps=7)
saved, seen, summary = run_resumable(tmp_path, monkeypatch, max_steps=7)
assert seen == [], "a finished run must not train again"
@@ -188,8 +185,8 @@ def test_a_final_checkpoint_between_saves_resumes_where_it_stopped(self, tmp_pat
assert saved == [9, 10]
assert [h["step"] for h in summary["history"]] == list(range(10))
+ # At batch size two, step 33 is the first step of the second 64-row epoch.
def test_resume_across_an_epoch_boundary(self, tmp_path, monkeypatch):
- """At batch size two, step 33 is the first step of the second 64-row epoch."""
_, full, _ = run_resumable(tmp_path / "ref", monkeypatch, max_steps=36)
run_resumable(tmp_path, monkeypatch, max_steps=33)
_, seen, summary = run_resumable(tmp_path, monkeypatch, max_steps=36)
@@ -217,8 +214,8 @@ def test_stale_checkpoints_and_calibration_are_moved_not_deleted(self, tmp_path)
assert (moved / "calibration.json").is_file()
assert set_aside_previous_run(out) is None
+ # Stale higher-step checkpoints must not survive in the active run directory.
def test_fresh_run_does_not_resume_from_a_higher_stale_step(self, tmp_path, monkeypatch):
- """Stale higher-step checkpoints must not survive in the active run directory."""
from lev.train.checkpoints import latest_checkpoint
stale = tmp_path / "out" / "step-18750"
@@ -233,7 +230,6 @@ def test_fresh_run_does_not_resume_from_a_higher_stale_step(self, tmp_path, monk
class TestModeAProjectsOnlyTheAnswerPositions:
def test_logits_to_keep_selects_distinct_positions_and_rows_read_their_own(self):
- """Project only answer positions and select the correct output column per row."""
from types import SimpleNamespace
from lev.router import Mode
@@ -266,9 +262,8 @@ def __call__(self, input_ids, attention_mask, logits_to_keep):
assert out.tolist() == [[71.0, 72.0], [141.0, 142.0], [273.0, 274.0]]
+# `scored_rows` is the shared forward that calibration and evaluation both read.
class TestScoredRows:
- """`scored_rows` is the shared forward that calibration and evaluation both read."""
-
def rows_through(self, monkeypatch, n_rows):
from lev.train import loop
from lev.train.config import PRESETS
diff --git a/packages/levbench/src/levbench/batching.py b/packages/levbench/src/levbench/batching.py
index e763022..2023521 100644
--- a/packages/levbench/src/levbench/batching.py
+++ b/packages/levbench/src/levbench/batching.py
@@ -88,7 +88,6 @@ def sweep(
bank: list[tuple[str, Any]] | None = None,
errors: list[tuple[int, str]] | None = None,
) -> list[SweepRow]:
- """Run the sweep, appending any per-count failure to `errors`."""
bank = QUESTION_BANK if bank is None else bank
rows: list[SweepRow] = []
errors = errors if errors is not None else []
diff --git a/packages/levbench/src/levbench/cli.py b/packages/levbench/src/levbench/cli.py
index e7522b1..60ad20f 100644
--- a/packages/levbench/src/levbench/cli.py
+++ b/packages/levbench/src/levbench/cli.py
@@ -21,7 +21,6 @@
def repo_data_dir() -> Path:
- """Find the repository data directory relative to this module."""
for parent in Path(__file__).resolve().parents:
if (candidate := parent / "data").is_dir():
return candidate
@@ -42,7 +41,6 @@ def _require_key(backend: str, base_url: str | None = None) -> None:
def _task_files(path: str | None) -> Sequence[Path | None]:
- """Return task files, or None to use the built-in fixture."""
if path is None:
return [None]
target = Path(path)
diff --git a/packages/levbench/src/levbench/confidence_id.py b/packages/levbench/src/levbench/confidence_id.py
index 69ebba3..92fe9f5 100644
--- a/packages/levbench/src/levbench/confidence_id.py
+++ b/packages/levbench/src/levbench/confidence_id.py
@@ -11,7 +11,6 @@
def gini(p: Distribution) -> float:
- """Normalized Gini concentration: (K*sum(p^2) - 1) / (K - 1)."""
k = len(p)
if k <= 1:
return 1.0
@@ -38,7 +37,6 @@ def top_two_margin(p: Distribution) -> float:
def normalized_max_prob(p: Distribution) -> float:
- """Normalize maximum probability relative to uniform chance."""
k = len(p)
if k <= 1:
return 1.0
@@ -63,7 +61,6 @@ def top_two_ratio(p: Distribution) -> float:
def detect_precision(values: list[float], max_decimals: int = 6) -> float:
- """Find a shared quantization step, or return zero if none matches."""
for decimals in range(1, max_decimals + 1):
unit = 10.0**-decimals
if all(abs(v / unit - round(v / unit)) < 1e-6 for v in values):
@@ -72,7 +69,6 @@ def detect_precision(values: list[float], max_decimals: int = 6) -> float:
def rounding_sensitivity(fn: Callable[[Distribution], float], p: Distribution, eps: float) -> float:
- """Bound how probability rounding affects a confidence formula."""
base = fn(p)
raised = fn([min(1.0, x + eps) for x in p])
lowered = fn([max(0.0, x - eps) for x in p])
@@ -87,14 +83,13 @@ class Fit:
max_abs_error: float
tolerance: float = 5e-3
+ # Tolerance allows for quantization error, as amplified by the candidate formula.
@property
def matches(self) -> bool:
- """Allow quantization error amplified by the candidate formula."""
return self.max_abs_error <= self.tolerance
def collect(answers: Iterable[Any]) -> list[tuple[Distribution, float]]:
- """Pull (distribution, reported confidence) from answers that have both."""
samples: list[tuple[Distribution, float]] = []
for answer in answers:
reported = getattr(answer, "confidence", None)
@@ -106,7 +101,6 @@ def collect(answers: Iterable[Any]) -> list[tuple[Distribution, float]]:
def identify(samples: list[tuple[Distribution, float]]) -> list[Fit]:
- """Rank candidate formulas by how closely they reproduce `confidence`."""
if not samples:
return []
everything = [v for dist, reported in samples for v in (*dist, reported)]
@@ -129,7 +123,6 @@ def identify(samples: list[tuple[Distribution, float]]) -> list[Fit]:
def identify_by_size(samples: list[tuple[Distribution, float]]) -> dict[int, list[Fit]]:
- """Fit confidence formulas separately for each candidate-set size."""
by_size: dict[int, list[tuple[Distribution, float]]] = {}
for dist, reported in samples:
by_size.setdefault(len(dist), []).append((dist, reported))
@@ -139,7 +132,6 @@ def identify_by_size(samples: list[tuple[Distribution, float]]) -> dict[int, lis
def worst_residuals(
samples: list[tuple[Distribution, float]], name: str, limit: int = 3
) -> list[tuple[Distribution, float, float]]:
- """Return the worst fits as (distribution, reported, predicted)."""
fn = CANDIDATES[name]
scored = [(dist, reported, fn(dist)) for dist, reported in samples]
return sorted(scored, key=lambda row: abs(row[2] - row[1]), reverse=True)[:limit]
@@ -172,7 +164,6 @@ def format_fits(fits: list[Fit]) -> str:
def format_diagnosis(samples: list[tuple[Distribution, float]]) -> str:
- """Format per-size fits and worst residuals."""
out: list[str] = []
off = [abs(sum(dist) - 1.0) for dist, _ in samples]
diff --git a/packages/levbench/src/levbench/metrics.py b/packages/levbench/src/levbench/metrics.py
index 372051e..f96a838 100644
--- a/packages/levbench/src/levbench/metrics.py
+++ b/packages/levbench/src/levbench/metrics.py
@@ -22,8 +22,8 @@ def to_distribution(answer: Any) -> dict[Any, float]:
raise ValueError(f"Unknown answer type {kind!r}")
+# Choice uses the model's own pick rather than the argmax.
def predicted_label(answer: Any) -> Any:
- """The argmax label, except for Choice, which uses the model's own pick."""
kind = answer.type
if kind == "noul":
return float(answer.noul) >= 0.5
@@ -36,7 +36,6 @@ def predicted_label(answer: Any) -> Any:
def top_probability(answer: Any) -> float:
- """Return the top-label probability used for ECE."""
return max(to_distribution(answer).values())
@@ -44,8 +43,8 @@ def log_loss(dist: dict[Any, float], truth: Any) -> float:
return -math.log(max(dist.get(truth, 0.0), _EPS))
+# Multiclass Brier: squared error summed over the whole vector.
def brier(dist: dict[Any, float], truth: Any) -> float:
- """Multiclass Brier score: summed squared error over the whole vector."""
labels = set(dist) | {truth}
return sum((dist.get(k, 0.0) - (1.0 if k == truth else 0.0)) ** 2 for k in labels)
@@ -66,9 +65,9 @@ def accuracy(self) -> float:
def mean_confidence(self) -> float:
return self.conf_sum / self.n if self.n else 0.0
+ # Signed: positive means overconfident.
@property
def gap(self) -> float:
- """Signed: positive means overconfident."""
return self.mean_confidence - self.accuracy
@@ -94,7 +93,6 @@ def mean_brier(self) -> float:
@property
def ece(self) -> float:
- """Expected Calibration Error: sample-weighted mean absolute bin gap."""
if not self.n:
return 0.0
return sum(b.n * abs(b.gap) for b in self.bins) / self.n
@@ -103,7 +101,6 @@ def ece(self) -> float:
def calibration(
records: list[tuple[dict[Any, float], Any, Any, float]], n_bins: int = 10
) -> Calibration:
- """Aggregate metrics over (distribution, predicted, truth, confidence) rows."""
edges = [i / n_bins for i in range(n_bins + 1)]
cal = Calibration(bins=[Bin(edges[i], edges[i + 1]) for i in range(n_bins)])
@@ -127,7 +124,6 @@ def calibration(
def selective_accuracy(
records: list[tuple[dict[Any, float], Any, Any, float]], threshold: float
) -> tuple[float, float]:
- """Return accuracy and coverage above a confidence threshold."""
kept = [r for r in records if r[3] >= threshold]
if not kept:
return 0.0, 0.0
diff --git a/packages/levbench/src/levbench/runner.py b/packages/levbench/src/levbench/runner.py
index f4507c1..a751e54 100644
--- a/packages/levbench/src/levbench/runner.py
+++ b/packages/levbench/src/levbench/runner.py
@@ -83,7 +83,6 @@ def build_client(
base_url: str | None = None,
timeout: float | None = None,
):
- """Build a Jev, local lev, or Anthropic client and return its model name."""
if backend in ("jev", "lev"):
if backend == "lev":
base_url = base_url or DEFAULT_LOCAL_BASE_URL
@@ -118,8 +117,8 @@ def build_client(
raise ValueError(f"Unknown backend {backend!r} (expected 'jev', 'lev' or 'anthropic')")
+# Billed retry totals win; zero counts are accepted, missing counts raise.
def _token_count(usage: Any, total_field: str, base_field: str) -> int:
- """Prefer billed retry totals; accept zero counts and reject missing counts."""
for field_name in (total_field, base_field):
value = getattr(usage, field_name, None)
if value is not None:
@@ -163,7 +162,6 @@ def run_eval(
questions: dict[str, Any],
concurrency: int = 1,
) -> EvalReport:
- """Evaluate concurrently, returning results in item order and per-call latencies."""
from concurrent.futures import ThreadPoolExecutor
report = EvalReport(backend=backend, model=model)
diff --git a/packages/levbench/src/levbench/snake/policy.py b/packages/levbench/src/levbench/snake/policy.py
index 04b5fe4..376d179 100644
--- a/packages/levbench/src/levbench/snake/policy.py
+++ b/packages/levbench/src/levbench/snake/policy.py
@@ -43,7 +43,6 @@ def to_dict(self) -> dict:
def describe(game: SnakeGame, prompt: str) -> tuple[str, dict[str, Any], dict]:
- """Build prompts and planner truth without calling the server."""
if prompt not in PROMPTS:
raise ValueError(f"prompt must be one of {PROMPTS}, got {prompt!r}")
moves = game.moves()
diff --git a/packages/levbench/src/levbench/snake/ui.py b/packages/levbench/src/levbench/snake/ui.py
index 5765639..e8d97ef 100644
--- a/packages/levbench/src/levbench/snake/ui.py
+++ b/packages/levbench/src/levbench/snake/ui.py
@@ -90,8 +90,8 @@ def network_label(backend: str, base_url: str | None) -> str:
return "localhost"
+# Probabilities describe the displayed board, before its announced next step.
def compose(game: dict, decision: dict, stats: dict) -> Canvas:
- """Probabilities describe the displayed board, before its announced next step."""
width, height = layout_size(game["width"], game["height"])
c = Canvas(width, height)
left, right, top = 3, max(58, game["width"] * 2 + 10), 6
@@ -227,8 +227,6 @@ def status_line(game: dict, decision: dict, stats: dict) -> str:
class Keyboard:
- """Non-blocking single-key reads from a TTY; a no-op anywhere else."""
-
def __enter__(self):
self.saved = None
if sys.stdin.isatty():
diff --git a/packages/levbench/src/levbench/tasks.py b/packages/levbench/src/levbench/tasks.py
index af8a75a..d824c97 100644
--- a/packages/levbench/src/levbench/tasks.py
+++ b/packages/levbench/src/levbench/tasks.py
@@ -227,8 +227,8 @@ def dataset(path: str | Path | None = None) -> tuple[Sequence[Item | FileItem],
return ITEMS, questions()
+# Comparing two runs needs more data than this single-run margin.
def detectable_difference(n: int, baseline: float = 0.8, z: float = 1.96) -> float:
- """Estimate the accuracy margin (95% at default z); comparing runs needs more data."""
if n <= 0:
return 1.0
return min(1.0, z * math.sqrt(baseline * (1 - baseline) / n))
diff --git a/packages/levbench/tests/test_offline.py b/packages/levbench/tests/test_offline.py
index fb35dc0..fce0812 100644
--- a/packages/levbench/tests/test_offline.py
+++ b/packages/levbench/tests/test_offline.py
@@ -64,9 +64,8 @@ def test_every_primitive_flattens_to_a_distribution() -> None:
assert score.score == pytest.approx(expected, abs=1e-2, rel=0)
+# Vendor confidence formulas differ, so ECE must use the answer's top probability.
def test_ece_bins_on_the_top_probability_not_the_vendor_confidence_field() -> None:
- """Vendor confidence formulas differ; ECE must use the answer's top probability."""
-
class Server:
def system_one(self, state, questions):
answer = SimpleNamespace(
@@ -95,8 +94,8 @@ def test_eval_runs_end_to_end_over_the_built_in_fixture() -> None:
assert "accuracy" in text and "ECE" in text
+# Synthetic billing checks the sweep arithmetic; live billing needs a live sweep.
def test_sweep_arithmetic_over_a_fixed_billing_shape() -> None:
- """Synthetic billing verifies sweep arithmetic; live billing requires a live sweep."""
state = (ROOT / "data" / "sample_policy.md").read_text()
rows = batching.sweep(fake_client(), runner.JEV_DEFAULT, state, [1, 2, 4, 8, 13])
@@ -122,10 +121,9 @@ def test_sweep_respects_an_explicitly_empty_bank() -> None:
batching.sweep(None, runner.JEV_DEFAULT, "state", [1], bank=[])
+# A match only means something if the identifier can tell candidate formulas apart.
@pytest.mark.parametrize("planted", confidence_id.CANDIDATES)
def test_confidence_identifier_recovers_a_planted_formula(planted) -> None:
- """A match is useful only if the identifier can distinguish candidate formulas."""
-
def handle(request):
body = json.loads(request.content)
answers = {}
@@ -185,7 +183,6 @@ def handle(request):
def test_local_base_url_never_forwards_the_real_key(monkeypatch) -> None:
- """Custom endpoints use a local key; hosted requests retain the real credential."""
monkeypatch.setenv("TYPESAFE_API_KEY", "sk-REAL-SECRET-KEY")
monkeypatch.delenv("LEVBENCH_LOCAL_API_KEY", raising=False)
local, _ = runner.build_client("jev", "litjev", "http://127.0.0.1:8000")
@@ -214,7 +211,6 @@ def test_an_unpinned_jev_run_reports_the_version_served() -> None:
def test_empty_local_key_env_var_falls_back(monkeypatch) -> None:
- """An empty key produces an invalid Bearer header; use the local fallback."""
monkeypatch.setenv("LEVBENCH_LOCAL_API_KEY", "")
client, _ = runner.build_client("lev")
assert isinstance(client, TypeSafeClient)
@@ -247,7 +243,6 @@ def _write_task_file(path, question_type="choice", n=5):
@pytest.mark.parametrize("question_type", ["choice", "score", "noul"])
def test_eval_runs_over_a_generated_task_file(tmp_path, question_type) -> None:
- """Exercise the task-file format produced by `lev data eval` for every primitive."""
items, questions = dataset(_write_task_file(tmp_path / "gen.json", question_type))
assert questions["q"].type == question_type
report = runner.run_eval(fake_client(), "jev", runner.JEV_DEFAULT, items, questions)
@@ -258,7 +253,6 @@ def test_eval_runs_over_a_generated_task_file(tmp_path, question_type) -> None:
def test_a_task_file_labelling_an_undefined_question_raises(tmp_path) -> None:
- """Reject undefined questions so their labels cannot be silently excluded from scoring."""
path = tmp_path / "bad.json"
path.write_text(
json.dumps(
@@ -283,8 +277,8 @@ def test_detectable_difference_shrinks_with_n() -> None:
assert detectable_difference(0) == 1.0
+# Grouping by candidate-set size can expose a formula that pooled samples hide.
def test_pooling_can_hide_a_formula_that_holds_per_candidate_set_size() -> None:
- """Grouping by candidate-set size can expose formulas hidden by pooled samples."""
four = [0.7, 0.1, 0.1, 0.1]
three = [0.5, 0.3, 0.2]
samples = [(four, confidence_id.max_prob(four))] * 6
@@ -311,14 +305,14 @@ def test_worst_residuals_surfaces_the_samples_that_break_a_candidate() -> None:
assert predicted == pytest.approx(0.5, abs=1e-9, rel=0)
+# Incomplete distributions can distort confidence formulas independently of fit.
def test_diagnosis_reports_whether_distributions_are_complete() -> None:
- """Incomplete distributions can distort confidence formulas independently of fit."""
truncated = [([0.6, 0.3], 0.6)]
assert "sum to 1 within 0.1" in confidence_id.format_diagnosis(truncated)
+# Normalized max probability amplifies rounding by K/(K-1).
def test_tolerance_accounts_for_how_much_a_statistic_amplifies_rounding() -> None:
- """Normalized max probability amplifies rounding by K/(K-1)."""
rng = random.Random(0)
samples = []
for size in (4, 3):
@@ -381,7 +375,6 @@ def test_cost_is_per_million_tokens_at_list_price(self) -> None:
assert cost_usd("Qwen/Qwen3.5-4B", 1_000_000, 1_000_000) == 0.0, "self-hosted is unmetered"
def test_billed_totals_win_over_base_counts(self) -> None:
- """Retry totals reflect billed usage."""
usage = SimpleNamespace(
input_tokens=10, input_tokens_total=25, output_tokens=2, output_tokens_total=4
)
@@ -391,8 +384,8 @@ def test_missing_token_counts_raise_rather_than_count_as_free(self) -> None:
with pytest.raises(ValueError, match="refusing to assume 0 tokens"):
runner._usage_ints(SimpleNamespace(input_tokens=None, output_tokens=None))
+ # Pricing uses the requested model, so a substitution has to be reported.
def test_a_substituted_model_is_flagged_in_the_report(self) -> None:
- """Report model substitution because pricing uses the requested model."""
report = runner.EvalReport(backend="jev", model="jev-1.12")
report.calls.append(runner.CallResult({}, 0.3, JEV_VERSION, 10, 1))
assert "requested 'jev-1.12' but server served ['jev-1.13.0']" in runner.format_report(
diff --git a/packages/levbench/tests/test_snake.py b/packages/levbench/tests/test_snake.py
index f730753..c02ddea 100644
--- a/packages/levbench/tests/test_snake.py
+++ b/packages/levbench/tests/test_snake.py
@@ -40,9 +40,8 @@ def unsafe_direction(game: SnakeGame) -> str:
return next(m.direction for m in game.moves() if not m.safe)
+# Answers from a chosen direction, with two Noul probabilities.
class StubServer:
- """Answers from a chosen direction and two Noul probabilities"""
-
def __init__(self, pick=None, route=0.9, food=0.9, follow_planner=False, invert=False):
self.pick, self.route, self.food = pick, route, food
self.follow_planner, self.invert = follow_planner, invert
@@ -248,8 +247,8 @@ def test_planner_never_needs_the_shield_and_reads_its_state(self):
class TestRounds:
+ # A small board clears within the step budget, forcing a new round.
def test_guarded_run_continues_past_a_cleared_board(self):
- """A small board clears within the step budget, forcing a new round."""
summary = play(
PlannerClient(), "planner", "planner", width=4, height=4, initial_length=2, steps=60
)
From 15bc6ffbc57b20413a51fbfac546edfdd6a08da2 Mon Sep 17 00:00:00 2001
From: Abhinavexist
Date: Thu, 15 Oct 2026 09:21:40 +0530
Subject: [PATCH 31/39] docs(claude): adapt repo references for the public
repository
---
CLAUDE.md | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/CLAUDE.md b/CLAUDE.md
index e09cc73..dcc546b 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -23,7 +23,7 @@ Read the relevant doc before changing anything it covers; most "obvious" improve
- **Readout.** Mode A reads label-token logits at the answer boundary whenever every label code is a single token — 68 options for this tokenizer, since the 69th code, `BQ`, is two. Above that, Mode B scores candidate text with the trained head. Training and serving route the same way (ADR-025, ADR-026); serving also skips split codes (ADR-028). Routing is in `router.py`.
- **Calibration** is per bucket, keyed `{type}:{mode}[:{band}]`, selected by leave-one-family-out transfer ECE. `calibrate.fit` refuses `test`, `eval` and `holdout` splits — fit only on the dedicated `calibration` split.
- **The release** (`lev-4b`) is preset `4b-instruct`, step 18750, chat prompt style. Its `lev_release.json` manifest sets the base model, prompt style and optional `adapter_scale`, and `lev.load` honours all three.
-- **The released weights were trained at commit `c6ccad7`.** Changes to data, sources, prompts or the training loop after it are not in the weights until a retrain; do not attribute a benchmark move to them.
+- **The released weights were trained on 2026-09-23 code, before this repository's first commit.** Changes to data, sources, prompts or the training loop since are not in the weights until a retrain; do not attribute a benchmark move to them.
## Running things
@@ -43,7 +43,7 @@ Read the relevant doc before changing anything it covers; most "obvious" improve
- **Ask before any paid call**: `levbench` against Jev, the Anthropic backend, or a sweep. They bill per token.
- `levbench` never sends `TYPESAFE_API_KEY` to a `--base-url` host; a custom endpoint gets `LEVBENCH_LOCAL_API_KEY` or `local`. Keep it that way.
- Do not read `.env`.
-- This repository is mirrored by hand to the public [`InterfazeAI/lev`](https://github.com/InterfazeAI/lev); the two diverge, so a change meant for release must be ported, not assumed.
+- This is the public repository. Development happens in [`Abhinavexists/lev`](https://github.com/Abhinavexists/lev) and is ported here by hand; the two diverge, so a change made here must be ported back, not assumed.
## Comments
From 7220285a72c47170c7146a1e072a7d112293e2f3 Mon Sep 17 00:00:00 2001
From: Abhinavexists
Date: Thu, 15 Oct 2026 10:57:53 +0530
Subject: [PATCH 32/39] feat(modal): serve a packaged release with
LEV_SERVE_RELEASE
---
Makefile | 4 ++--
modal/app.py | 8 ++++++++
2 files changed, 10 insertions(+), 2 deletions(-)
diff --git a/Makefile b/Makefile
index 808d89a..5ba5612 100644
--- a/Makefile
+++ b/Makefile
@@ -96,10 +96,10 @@ calibrate:
.PHONY: serve deploy release weights publish
serve:
- LEV_SERVE_PRESET=$(PRESET) modal serve modal/app.py
+ LEV_SERVE_PRESET=$(PRESET) $(if $(NAME),LEV_SERVE_RELEASE=$(NAME)) modal serve modal/app.py
deploy:
- LEV_SERVE_PRESET=$(PRESET) modal deploy modal/app.py
+ LEV_SERVE_PRESET=$(PRESET) $(if $(NAME),LEV_SERVE_RELEASE=$(NAME)) modal deploy modal/app.py
release:
modal run modal/app.py::export_checkpoint --preset $(PRESET) $(if $(NAME),--name $(NAME),)
diff --git a/modal/app.py b/modal/app.py
index f25bad0..573d7ca 100644
--- a/modal/app.py
+++ b/modal/app.py
@@ -101,6 +101,7 @@ def _serve_overrides() -> list:
for key in (
"LEV_SERVE_PRESET",
"LEV_SERVE_MODEL",
+ "LEV_SERVE_RELEASE",
"LEV_SERVE_COMPILE",
"LEV_SERVE_MAX_LABEL_OPTIONS",
"LEV_SERVE_PROMPT",
@@ -285,9 +286,16 @@ def serve():
skip_codes = os.environ.get("LEV_SERVE_SKIP_CODES", "1") not in ("0", "false", "no")
# A frozen-model override must bypass the adapter checkpoint.
frozen = os.environ.get("LEV_SERVE_MODEL")
+ release = os.environ.get("LEV_SERVE_RELEASE")
if frozen:
print(f"serving {frozen} frozen: no adapter, binary Noul, raw softmax")
checkpoint_dir, model_id = None, frozen
+ elif release:
+ # A release is what was published: its manifest's adapter scale and refit temperatures.
+ checkpoint_dir, model_id = f"{RELEASES_DIR}/{release}", config.model_id
+ if not Path(checkpoint_dir, "lev_release.json").is_file():
+ raise FileNotFoundError(f"LEV_SERVE_RELEASE={release!r}: nothing at {checkpoint_dir}")
+ print(f"serving release {release}")
else:
output = Path(f"{CKPT_DIR}/{preset}")
# `latest_checkpoint` skips a step directory without weights (an interrupted save).
From 7d0fcdacc34da5b5fa2fd6c0a64c9124df6dfbde Mon Sep 17 00:00:00 2001
From: Abhinavexists
Date: Thu, 15 Oct 2026 10:57:53 +0530
Subject: [PATCH 33/39] docs(setup): document serving a packaged release
---
docs/SETUP.md | 1 +
docs/TRAINING.md | 2 +-
2 files changed, 2 insertions(+), 1 deletion(-)
diff --git a/docs/SETUP.md b/docs/SETUP.md
index cf0aaab..e51fe8d 100644
--- a/docs/SETUP.md
+++ b/docs/SETUP.md
@@ -78,6 +78,7 @@ Set on the machine where `modal deploy` runs, from the environment or `.env`; th
| --- | --- | --- |
| `LEV_SERVE_PRESET` | `4b` | Which preset's newest checkpoint to serve. `make deploy PRESET=...` sets it |
| `LEV_SERVE_MODEL` | unset | Serve this model frozen instead — no adapter, binary Noul, the zero-shot baseline |
+| `LEV_SERVE_RELEASE` | unset | Serve `releases/` instead of the training output, with its manifest's adapter scale and calibration. `make deploy NAME=...` sets it |
| `LEV_SERVE_COMPILE` | `0` | `torch.compile`. Off unless asked: measured slower than eager here (ADR-023) |
| `LEV_SERVE_MAX_LABEL_OPTIONS` | unset | Cap on label options. Unset means Mode A up to the tokenizer limit (ADR-025) |
| `LEV_SERVE_PROMPT` | the preset's | Override the prompt style, mainly for a frozen model |
diff --git a/docs/TRAINING.md b/docs/TRAINING.md
index d4fd4d2..4070cbf 100644
--- a/docs/TRAINING.md
+++ b/docs/TRAINING.md
@@ -260,7 +260,7 @@ make publish RELEASE=4b-instruct REPO=org/name # -> huggingface.co/org/name (HF
### On Modal: which checkpoint, and `serve` vs `deploy`
-`make deploy PRESET=4b-instruct` serves that preset's newest checkpoint; the preset travels to the container as `LEV_SERVE_PRESET`. `LEV_SERVE_MODEL=Qwen/Qwen3.5-4B make deploy` serves that model frozen — no adapter, binary Noul — the zero-shot baseline.
+`make deploy PRESET=4b-instruct` serves that preset's newest checkpoint; the preset travels to the container as `LEV_SERVE_PRESET`. `make deploy NAME=lev` serves the packaged release instead — what the Hub ships, including an `adapter_scale` or calibration refit after export. `LEV_SERVE_MODEL=Qwen/Qwen3.5-4B make deploy` serves that model frozen — no adapter, binary Noul — the zero-shot baseline.
`modal serve` is an ephemeral dev server. So is every `modal run` of this app file, and each one registers the `serve` web function under the same `-dev` label — so running an export or a diagnostic while the dev server is up steals its URL, and the URL returns 404 when the run exits. `make deploy` gives the server a stable URL that runs cannot take.
From 0ff1e8385d7da371e9d8d6c5b677c343df764da5 Mon Sep 17 00:00:00 2001
From: Abhinavexists
Date: Thu, 15 Oct 2026 10:57:53 +0530
Subject: [PATCH 34/39] docs(readme): note score order averaging and the
presentation checks
---
README.md | 6 +++---
hf/README.md | 2 +-
2 files changed, 4 insertions(+), 4 deletions(-)
diff --git a/README.md b/README.md
index b7d1488..6eeb07a 100644
--- a/README.md
+++ b/README.md
@@ -166,7 +166,7 @@ flowchart LR
style O fill:#2a78d6,stroke:#2a78d6,color:#ffffff
```
-Each option gets a short code, and the answer is read from the next-token logits over those codes. Choices are read in two option orders and averaged, which cancels position bias. Codes that would split into two tokens are skipped, so this reaches several hundred options.
+Each option gets a short code, and the answer is read from the next-token logits over those codes. Choices are read in two option orders and averaged, which cancels position bias. Score levels stay in their trained order unless the server runs with `--score-order-average` (off by default, [ADR-029](docs/DECISIONS.md#adr-029--score-is-read-in-one-order-measure-the-position-effect-and-offer-averaging-off-by-default)). Codes that would split into two tokens are skipped, so this reaches several hundred options.
**The router is the part nobody else had.** Every other implementation picks one readout family and hits a wall when options stop fitting in single tokens: it caps the option count or rejects the request. lev routes those questions to a small learned head that matches the state against each option's text, so there is no fixed option ceiling. [ADR-005 →](docs/DECISIONS.md#adr-005--dual-mode-readout-the-differentiator)
@@ -265,7 +265,7 @@ The data build refuses any source that resolves to one of the 13 S1Bench subsets
```text
packages/
- lev/ the model — schema, prompt layouts, router, readouts, calibration, loader, server, S1Bench export
+ lev/ the model — schema, prompt layouts, router, readouts, calibration, loader, server, presentation checks, S1Bench export
levbench/ the harness — accuracy, ECE, cost, latency, batching economics, Snake
modal/ app.py — train / calibrate / serve / check a release on one H100
hf/ README.md — the Hub model card, and the charts it shows
@@ -276,7 +276,7 @@ docs/ ARCHITECTURE — the design, and every model it was derived fro
STATUS — what is built and how far it has been run
SETUP, TRAINING — local and Modal, the pipeline step by step
charts/ — the lev-vs-Jev comparison page, built from the logs
-data/ sample state + the S1Bench snapshot the analysis rests on
+data/ sample state, the S1Bench snapshot the analysis rests on, the presentation report
```
## Prior work
diff --git a/hf/README.md b/hf/README.md
index afc1f69..44edae4 100644
--- a/hf/README.md
+++ b/hf/README.md
@@ -187,7 +187,7 @@ The 69 ms is engine compute for a short request (a three-sentence state), measur
-- **Label-token readout.** Each option gets a short code, and the answer is read from the next-token logits over those codes. Codes that would split into two tokens are skipped, so every option stays one token. This handles yes/no, scores, and choice lists up to several hundred options. Choices are read in two option orders and averaged, which cancels position bias.
+- **Label-token readout.** Each option gets a short code, and the answer is read from the next-token logits over those codes. Codes that would split into two tokens are skipped, so every option stays one token. This handles yes/no, scores, and choice lists up to several hundred options. Choices are read in two option orders and averaged, which cancels position bias. Score levels stay in their trained order unless `lev serve` runs with `--score-order-average`, which is off by default.
- **Candidate-path head.** Past that point, a small learned head matches the state against each option's text, so the number of options has no fixed ceiling.
- **Calibration.** Temperatures fitted after training are applied at load time. There is one per question type, readout mode, and choice option-count band. They were selected by how well they carry over to task families left out of the fit, not only by how well they fit held-out rows.
From 25590d465d206ac789db49f4cdf78765b76da4fe Mon Sep 17 00:00:00 2001
From: Abhinavexists
Date: Thu, 15 Oct 2026 12:26:37 +0530
Subject: [PATCH 35/39] feat(release): read the hub write key from .env when
publishing
---
packages/lev/pyproject.toml | 1 +
packages/lev/src/lev/cli.py | 4 ++++
uv.lock | 3 +++
3 files changed, 8 insertions(+)
diff --git a/packages/lev/pyproject.toml b/packages/lev/pyproject.toml
index 9330159..ed18196 100644
--- a/packages/lev/pyproject.toml
+++ b/packages/lev/pyproject.toml
@@ -28,6 +28,7 @@ train = [
"accelerate>=1.2",
"datasets>=3.2",
"huggingface-hub>=1.0",
+ "python-dotenv>=1.2.3",
"safetensors>=0.4",
]
serve = [
diff --git a/packages/lev/src/lev/cli.py b/packages/lev/src/lev/cli.py
index f420054..bf4e1b6 100644
--- a/packages/lev/src/lev/cli.py
+++ b/packages/lev/src/lev/cli.py
@@ -193,8 +193,12 @@ def cmd_release_build(args: argparse.Namespace) -> None:
def cmd_release_publish(args: argparse.Namespace) -> None:
+ from dotenv import find_dotenv, load_dotenv
+
from .release import publish
+ # The Hub write key lives in .env; `hf auth login` may hold a token without org write access.
+ load_dotenv(find_dotenv(usecwd=True))
url = publish(args.release, args.repo, private=args.private)
print(f"published -> {url}")
print(f"\n uv run lev serve --checkpoint {args.repo}")
diff --git a/uv.lock b/uv.lock
index 0c74dea..2c299e6 100644
--- a/uv.lock
+++ b/uv.lock
@@ -966,6 +966,7 @@ serve = [
{ name = "fastapi" },
{ name = "huggingface-hub" },
{ name = "peft" },
+ { name = "python-dotenv" },
{ name = "safetensors" },
{ name = "torch" },
{ name = "transformers" },
@@ -976,6 +977,7 @@ train = [
{ name = "datasets" },
{ name = "huggingface-hub" },
{ name = "peft" },
+ { name = "python-dotenv" },
{ name = "safetensors" },
{ name = "torch" },
{ name = "transformers" },
@@ -991,6 +993,7 @@ requires-dist = [
{ name = "modal", marker = "extra == 'modal'", specifier = ">=1.0" },
{ name = "peft", marker = "extra == 'train'", specifier = ">=0.14" },
{ name = "pydantic", specifier = ">=2.12" },
+ { name = "python-dotenv", marker = "extra == 'train'", specifier = ">=1.2.3" },
{ name = "safetensors", marker = "extra == 'train'", specifier = ">=0.4" },
{ name = "torch", marker = "extra == 'train'", specifier = ">=2.6" },
{ name = "transformers", marker = "extra == 'train'", specifier = ">=5.0" },
From b24de30e7f72cc3b12efc073142514a6835e7e32 Mon Sep 17 00:00:00 2001
From: Abhinavexists
Date: Thu, 15 Oct 2026 12:26:38 +0530
Subject: [PATCH 36/39] test(release): pin the hub key precedence for publish
---
packages/lev/tests/test_cli.py | 37 ++++++++++++++++++++++++++++++++++
1 file changed, 37 insertions(+)
diff --git a/packages/lev/tests/test_cli.py b/packages/lev/tests/test_cli.py
index af22e87..10ac262 100644
--- a/packages/lev/tests/test_cli.py
+++ b/packages/lev/tests/test_cli.py
@@ -1,3 +1,4 @@
+import os
from dataclasses import replace
from lev import cli
@@ -42,3 +43,39 @@ def run_training(config, data_dir, **kwargs):
assert outputs == [override, default_output]
assert preset.output_dir == default_output
+
+
+def published_token(monkeypatch, tmp_path, env_file: str | None, exported: str | None):
+ from lev import release
+
+ monkeypatch.chdir(tmp_path)
+ if env_file is not None:
+ (tmp_path / ".env").write_text(env_file)
+ if exported is None:
+ monkeypatch.delenv("HF_TOKEN", raising=False)
+ else:
+ monkeypatch.setenv("HF_TOKEN", exported)
+ monkeypatch.setattr(os, "environ", dict(os.environ)) # load_dotenv writes here
+ seen = {}
+
+ def fake_publish(*args, **kwargs):
+ seen["token"] = os.environ.get("HF_TOKEN") or None
+ return "url"
+
+ monkeypatch.setattr(release, "publish", fake_publish)
+ cli.main(["release", "publish", "weights/lev", "--repo", "org/name"])
+ return seen["token"]
+
+
+def test_publish_takes_the_write_key_from_dot_env(monkeypatch, tmp_path, capsys):
+ env = 'TYPESAFE_API_KEY=x\nexport HF_TOKEN="hf_write"\n'
+ assert published_token(monkeypatch, tmp_path, env, None) == "hf_write"
+
+
+def test_an_exported_token_wins_over_dot_env(monkeypatch, tmp_path, capsys):
+ assert published_token(monkeypatch, tmp_path, "HF_TOKEN=hf_file\n", "hf_shell") == "hf_shell"
+
+
+def test_without_either_publish_falls_back_to_the_hub_login(monkeypatch, tmp_path, capsys):
+ assert published_token(monkeypatch, tmp_path, None, None) is None
+ assert published_token(monkeypatch, tmp_path, "HF_TOKEN=\n", None) is None
From 136725ccddf845daace5354f3d73d55a581c4a00 Mon Sep 17 00:00:00 2001
From: Abhinavexists
Date: Thu, 15 Oct 2026 12:26:38 +0530
Subject: [PATCH 37/39] docs(readme): report the adapter-scaled release on
s1bench
---
README.md | 44 ++++++++++++-------------
hf/README.md | 46 +++++++++++++--------------
hf/assets/accuracy-by-subset.png | Bin 175950 -> 176474 bytes
hf/assets/accuracy-per-parameter.png | Bin 102550 -> 103213 bytes
hf/assets/leaderboard.png | Bin 240773 -> 245613 bytes
5 files changed, 45 insertions(+), 45 deletions(-)
diff --git a/README.md b/README.md
index 6eeb07a..31ae9c2 100644
--- a/README.md
+++ b/README.md
@@ -10,9 +10,9 @@ lev is an open System One decision model and the harness that measures it. Give
Read full blog here: https://interfaze.ai/blog/jev-now-open-source-lev
-
68.9% on all 13 S1Bench subsets. 4B parameters. Zero output tokens.
+
74.3% on all 13 S1Bench subsets. 4B parameters. Zero output tokens.
-
Qwen3.5-4B + LoRA on one H100. On the six subsets the public S1Bench board completed, level with reflex-4b and behind only Jev and three open models of 26B–35B. Accuracy receipt · Speed receipt
+
Qwen3.5-4B + LoRA on one H100. On the six subsets the public S1Bench board completed, behind only Jev and two open models of 26B and up, ahead of reflex-4b and a 35B model. Accuracy receipt · Speed receipt
@@ -22,27 +22,27 @@ lev is an open System One decision model and the harness that measures it. Give
| subset | task | **lev** | Jev | always the most common label |
| -------------------- | ------------------------------ | --------: | --------: | ---------------------------: |
-| vitaminc-dev | claim verification | 0.668 | **0.801** | 0.503 |
-| massive-en-US | intent routing, 18 scenarios | 0.857 | **0.874** | 0.163 |
-| massive-de-DE | intent routing, German | 0.823 | **0.871** | 0.163 |
-| boolq | yes/no reading comprehension | 0.827 | **0.893** | 0.580 |
-| squad2 | answerability | 0.813 | **0.836** | 0.502 |
-| paws | adversarial paraphrase | 0.776 | **0.900** | 0.516 |
-| multinli | natural language inference | **0.890** | 0.836 | 0.361 |
-| civil_comments | toxicity | 0.760 | **0.803** | 0.893 |
-| aegis2 | safety moderation | 0.800 | **0.804** | 0.568 |
-| helpsteer2 | helpfulness, 5 levels | **0.386** | 0.341 | 0.422 |
-| summeval-relevance | summary relevance, 5 levels | 0.358 | 0.358 | 0.458 |
-| summeval-consistency | summary faithfulness, 5 levels | 0.271 | **0.812** | 0.840 |
-| pubmedqa | biomedical yes/no/maybe | 0.732 | **0.764** | 0.532 |
-| **macro** | | 0.689 | **0.761** | |
-
-lev and Jev ran through the same harness on all 3,880 items S1Bench scores, pinned by [Nimble](https://github.com/bespokelabsai/nimble)'s manifests. The Jev run lands within 0.8 points of TypeSafe's published figure on every subset, so the harness is not the gap.
-
-- **Noise:** at these sizes a per-subset difference needs roughly 5–9 points to be real. lev's leads on multinli and helpsteer2 are inside that.
-- **Where Jev is clearly ahead:** the minimal-edit pairs (paws −12.4, vitaminc −13.3) and summeval-consistency (−54.1), where lev rates most fully faithful summaries one level low. Fine-tuning introduced that: the untuned backbone scores 0.826.
+| vitaminc-dev | claim verification | 0.726 | **0.801** | 0.503 |
+| massive-en-US | intent routing, 18 scenarios | 0.869 | **0.874** | 0.163 |
+| massive-de-DE | intent routing, German | 0.829 | **0.871** | 0.163 |
+| boolq | yes/no reading comprehension | 0.853 | **0.893** | 0.580 |
+| squad2 | answerability | 0.819 | **0.836** | 0.502 |
+| paws | adversarial paraphrase | 0.844 | **0.900** | 0.516 |
+| multinli | natural language inference | **0.849** | 0.836 | 0.361 |
+| civil_comments | toxicity | 0.783 | **0.803** | 0.893 |
+| aegis2 | safety moderation | **0.836** | 0.804 | 0.568 |
+| helpsteer2 | helpfulness, 5 levels | **0.361** | 0.341 | 0.422 |
+| summeval-relevance | summary relevance, 5 levels | **0.371** | 0.358 | 0.458 |
+| summeval-consistency | summary faithfulness, 5 levels | 0.806 | **0.812** | 0.840 |
+| pubmedqa | biomedical yes/no/maybe | 0.716 | **0.764** | 0.532 |
+| **macro** | | 0.743 | **0.761** | |
+
+lev and Jev ran through the same harness on all 3,880 items S1Bench scores, pinned by [Nimble](https://github.com/bespokelabsai/nimble)'s manifests. The Jev run lands within 0.8 points of TypeSafe's published figure on every subset, so the harness is not the gap. The adapter scale was chosen by leave-one-family-out over these subsets; scored only on families held out of that choice, it estimates 0.728 ([FINDINGS §19](docs/FINDINGS.md#19-the-adapter-scale-is-an-out-of-distribution-knob-and-it-is-worth-39-macro-points)).
+
+- **Noise:** at these sizes a per-subset difference needs roughly 5–9 points to be real. lev's leads on aegis2, helpsteer2, multinli and summeval-relevance are all inside that.
+- **Where Jev is clearly ahead:** the minimal-edit pairs (vitaminc −7.5, paws −5.6). The adapter runs at a quarter of its trained strength, which narrowed those gaps from −13.3 and −12.4 and closed summeval-consistency's from −54.1 to −0.6: at full strength it rated most fully faithful summaries one level low.
- **Constant baselines:** on civil_comments (89% not toxic) and the three 5-level rating subsets, always answering the most common label beats both models.
-- **Calibration:** mean ECE 0.115 for lev, 0.091 for Jev; lev is better calibrated on 5 of 13.
+- **Calibration:** mean ECE 0.097 for lev, 0.091 for Jev; lev is better calibrated on 6 of 13. summeval-consistency is the outlier, under-confident at 0.303.
- **Speed:** 69 ms is lev's engine compute for a short request inside the container. End to end from a laptop, Jev answered in 335–346 ms median and lev on one Modal H100 in 414–654 ms across two runs.
[Every run, per-subset breakdowns and what each run changed →](docs/FINDINGS.md)
diff --git a/hf/README.md b/hf/README.md
index 44edae4..37b96af 100644
--- a/hf/README.md
+++ b/hf/README.md
@@ -26,9 +26,9 @@ lev answers typed questions about a piece of context in a single forward pass. Y
-
68.9% on all 13 S1Bench subsets. 4B parameters. Zero output tokens.
+
74.3% on all 13 S1Bench subsets. 4B parameters. Zero output tokens.
-
Qwen3.5-4B + LoRA on one H100. On the six subsets the public S1Bench board completed, level with reflex-4b and behind only Jev and three open models of 26B–35B.
+
Qwen3.5-4B + LoRA on one H100. On the six subsets the public S1Bench board completed, behind only Jev and two open models of 26B and up, ahead of reflex-4b and a 35B model.
@@ -88,7 +88,7 @@ print(result.usage.output_tokens) # 0
These are real outputs from this checkpoint. The answers are objects, and `result.model_dump()` gives the same JSON the HTTP server returns.
-All the questions share one forward pass, so asking three questions costs about the same as asking one. `lev.load` reads `lev_release.json` from this repository to find the base model and the prompt format the adapter was trained with. It also applies the shipped calibration and loads the matching head, so there is nothing to configure.
+All the questions share one forward pass, so asking three questions costs about the same as asking one. `lev.load` reads `lev_release.json` from this repository to find the base model, the prompt format the adapter was trained with, and the strength to apply the adapter at. It also applies the shipped calibration and loads the matching head, so there is nothing to configure.
Because the probabilities are calibrated, you can gate on them. For example, act automatically above 0.9 and send anything lower to a person.
@@ -135,27 +135,27 @@ print(response.answers["team"].choice, response.answers["bug"].noul) # technica
| subset | task | **lev** | Jev | always the most common label |
| --- | --- | --: | --: | --: |
-| vitaminc-dev | claim verification | 0.668 | **0.801** | 0.503 |
-| massive-en-US | intent routing, 18 scenarios | 0.857 | **0.874** | 0.163 |
-| massive-de-DE | intent routing, German | 0.823 | **0.871** | 0.163 |
-| boolq | yes/no reading comprehension | 0.827 | **0.893** | 0.580 |
-| squad2 | answerability | 0.813 | **0.836** | 0.502 |
-| paws | adversarial paraphrase | 0.776 | **0.900** | 0.516 |
-| multinli | natural language inference | **0.890** | 0.836 | 0.361 |
-| civil_comments | toxicity | 0.760 | **0.803** | 0.893 |
-| aegis2 | safety moderation | 0.800 | **0.804** | 0.568 |
-| helpsteer2 | helpfulness, 5 levels | **0.386** | 0.341 | 0.422 |
-| summeval-relevance | summary relevance, 5 levels | 0.358 | 0.358 | 0.458 |
-| summeval-consistency | summary faithfulness, 5 levels | 0.271 | **0.812** | 0.840 |
-| pubmedqa | biomedical yes/no/maybe | 0.732 | **0.764** | 0.532 |
-| **macro** | | 0.689 | **0.761** | |
-
-lev and TypeSafe Jev ran through the same harness on all 3,880 items S1Bench scores, pinned by [Nimble](https://github.com/bespokelabsai/nimble)'s manifests. The Jev run lands within 0.8 points of TypeSafe's published figure on every subset, so the harness is not the gap.
-
-- **Noise:** at these sizes a per-subset difference needs roughly 5–9 points to be real. lev's leads on multinli and helpsteer2 are inside that.
-- **Where Jev is clearly ahead:** the minimal-edit pairs (paws −12.4, vitaminc −13.3) and summeval-consistency (−54.1), where lev rates most fully faithful summaries one level low. Fine-tuning introduced that: the untuned backbone scores 0.826.
+| vitaminc-dev | claim verification | 0.726 | **0.801** | 0.503 |
+| massive-en-US | intent routing, 18 scenarios | 0.869 | **0.874** | 0.163 |
+| massive-de-DE | intent routing, German | 0.829 | **0.871** | 0.163 |
+| boolq | yes/no reading comprehension | 0.853 | **0.893** | 0.580 |
+| squad2 | answerability | 0.819 | **0.836** | 0.502 |
+| paws | adversarial paraphrase | 0.844 | **0.900** | 0.516 |
+| multinli | natural language inference | **0.849** | 0.836 | 0.361 |
+| civil_comments | toxicity | 0.783 | **0.803** | 0.893 |
+| aegis2 | safety moderation | **0.836** | 0.804 | 0.568 |
+| helpsteer2 | helpfulness, 5 levels | **0.361** | 0.341 | 0.422 |
+| summeval-relevance | summary relevance, 5 levels | **0.371** | 0.358 | 0.458 |
+| summeval-consistency | summary faithfulness, 5 levels | 0.806 | **0.812** | 0.840 |
+| pubmedqa | biomedical yes/no/maybe | 0.716 | **0.764** | 0.532 |
+| **macro** | | 0.743 | **0.761** | |
+
+lev and TypeSafe Jev ran through the same harness on all 3,880 items S1Bench scores, pinned by [Nimble](https://github.com/bespokelabsai/nimble)'s manifests. The Jev run lands within 0.8 points of TypeSafe's published figure on every subset, so the harness is not the gap. The adapter scale was chosen by leave-one-family-out over these subsets; scored only on families held out of that choice, it estimates 0.728 ([FINDINGS §19](https://github.com/Abhinavexists/lev/blob/main/docs/FINDINGS.md#19-the-adapter-scale-is-an-out-of-distribution-knob-and-it-is-worth-39-macro-points)).
+
+- **Noise:** at these sizes a per-subset difference needs roughly 5–9 points to be real. lev's leads on aegis2, helpsteer2, multinli and summeval-relevance are all inside that.
+- **Where Jev is clearly ahead:** the minimal-edit pairs (vitaminc −7.5, paws −5.6). The adapter runs at a quarter of its trained strength, which narrowed those gaps from −13.3 and −12.4 and closed summeval-consistency's from −54.1 to −0.6: at full strength it rated most fully faithful summaries one level low.
- **Constant baselines:** on civil_comments (89% not toxic) and the three 5-level rating subsets, always answering the most common label beats both models.
-- **Calibration:** mean ECE 0.115 for lev, 0.091 for Jev; lev is better calibrated on 5 of 13.
+- **Calibration:** mean ECE 0.097 for lev, 0.091 for Jev; lev is better calibrated on 6 of 13. summeval-consistency is the outlier, under-confident at 0.303.