[chrispezza]

The measurement #139 asked for, and the one code change the run needed.

The run

The resident Qwen3.6-27B (Q4_K_M, llama.cpp llama-server, one slot, M4 Pro) read through ADR-005's logit path, against a live jev-1.13.0 run on the same 112 fixtures and the same policy (questions 4966-c2ecb464). It is the first live run of backend: local against a general open-weights model. Write-up: docs/calibration/2026-10-12-qwen36-27b-1.md. Both arms' answers are committed, so calibrate --from a.jsonl,b.jsonl reproduces every number with no key and no model.

It does not pass.

arm	accuracy	Brier	ECE	AUC	false allows	false asks	p50 per fixture
Jev 1.13	94.4%	0.053	0.064	0.986	0	9 / 60	155 ms
Qwen3.6-27B Q4_K_M	83.8%	0.105	0.027	0.966	0	54 / 60	15,678 ms

• Brier gate fails: mean Brier 0.052 worse against a limit of 0.050, and unreviewed_execution misses the bar (16 of 19 at confidence 0.80 or above).
• All 45 verdict differences go one way: Jev allows, Qwen asks. Forty of the 54 false asks are a question at or above its 0.60 rule on something safe; 14 are the uncertainty band.
• It ranks nearly as well as Jev, which is what separates this from Laya's failure (AUC 0.67). The thresholds were chosen on Jev's answers and nothing here tests others; the write-up gives no fitted number because any available would be fitted on the fixtures it is scored on.
• 15.7 s a fixture rules it out of the hook as served.

The arms were answered 37 minutes apart, not in the same minute, and the write-up says so.

The code change

score() allowed 30 s a fixture, in code, and a timeout ends the run with nothing written. The first attempt was lost that way. calibrate --timeout <ms> sets it; the default is unchanged. Version 0.2.21 with a changelog entry.

Not fixed here, filed as #152: one slow fixture still loses the whole run, and the preflight does not wait for a server that has unloaded its model.

Checks

1709 tests pass, including the flag's parsing. gitleaks over docs/calibration finds nothing, and neither answers file holds a local path. No hook code changed. HANDOFF.md updated.

Refs #139, #111.

🤖 Generated with Claude Code