fix(#499): the delivery that passed every check and served nothing — report the posterior, capture P8 anchors, probe conda for capability

[Polichinel]

The 2026-09-30 rehearsal delivered successfully and served nothing.

Every structural check passed: valid manifest, coverage exactly land_gaul's 64,742 cells, the findability guard resolving all 111 objects. And tower_point — the estimator views-faoapi serves — returned zero for all 2,333,448 cells. 98.2% of cells had no signal in any of 128 draws.

It was found by hand at 4am, by someone computing anchor values for an unrelated probe.

That is the defect. Not that an undertrained model produced an empty posterior — that is what 40 lessons means — but that nothing in the chain said so.

1. tools/prereg/posterior_health.py

Runs on every delivery, between the publish and the postprocessor, because that is the first moment the pooled frame exists on disk. Reports cells with no signal in any draw, the largest draw, and how many cells carry a non-zero served point estimate.

The same numbers mean opposite things for a rehearsal and a production run, so it is told which it is. A degenerate rehearsal exits 0 and says expected; a degenerate production run exits non-zero and says the delivery is structurally valid and serves nothing. Failing the rehearsal it was designed to produce would teach the operator to ignore the exit code, which costs more than it saves.

The DEGENERATE threshold is the starkest available fact — the served estimate is zero everywhere — rather than a tuned fraction, because a threshold that needs calibration is one that gets argued with.

2. tools/prereg/capture_anchors.py

P8's anchors, moved off the pod. Until now this existed only at /root/capture_anchors.py on a machine that will be destroyed — precisely the defect this whole effort is about.

Run from the runner, which is what makes P8's protection structural rather than procedural: the rule is that anchors must not be chosen after seeing what the API returns, and running it by hand afterwards preserves that only if nobody looked first.

It records the raw draws as well as the point estimate. On the run that motivated it every point estimate was 0.0, so comparing them would have passed against an unrelated empty dataset. A cell with 3 non-zero draws of 128 and a specific 301,274 spike is a fingerprint. A probe that can only detect a disagreement is worth less than one that can also diagnose it.

It also records the views_frames version — P8 claimed to remove "compared the wrong thing" by construction on the basis that both sides run the same version. They float independently inside >=1.10.2,<2, and the serving version is not observable from outside (faoapi's /version reports the app, not its dependencies).

3. conda is probed for capability, not presence

Preflight checked command -v conda. The delivery then failed at step 4 — the last, after every GPU hour — because miniconda 26.7.1 will not create an environment until its channel Terms of Service are accepted.

A --dry-run create exercises resolution, channel access and the ToS gate together in seconds, and the refusal now names the two conda tos accept commands rather than leaving an operator to find them at 4am.

Also in this branch

The Appwrite coordinate registry check (earlier commit): platform_env_require_registry() states "the registry is the ONLY source of coordinates" (#308) and defaults to a relative hop to a sibling checkout of views-appwrite, which a pod that cloned only views-models does not have. Found mid-rehearsal with four of eight models trained.

The pattern across all four: a thing the last step needs that nothing checks before the first step spends money. And one trap worth stating — preflight already resolved un_fao's REGION and that passed, but REGION resolves a coverage declaration, a different registry entirely. A check that looks adjacent is not a check.

Verification

Ten new tests execute the tools against synthetic degenerate, sparse and healthy posteriors — including a control that a healthy posterior is not flagged, without which a tool printing DEGENERATE unconditionally would pass everything else. They assert on behaviour, not source text: the previous round of guards in this repository went 22/22 green with every fix reverted.

Both tools read the pooled frame as the ensemble writes it — y_pred.npy plus identifiers.npz, not PredictionFrame.load's values.npy. That mismatch cost twenty minutes and is stated in the code rather than left to be rediscovered.

ruff clean. 8017 passed, 234 skipped, 5 xfailed.

What these cannot do

Neither tool can prevent anything. Pooling and publishing are a single invocation, so by the time a posterior can be inspected it is already on the shelf. They make it impossible to miss, not impossible to happen. The refusal belongs where the publish decision is made — #523, which is an ADR-013 contract change and a maintainer's call.

🤖 Generated with Claude Code

https://claude.ai/code/session_01GG2cY2HaqmUMpwUyR5V82K

[Polichinel]

Code review

Found 7 issues:

1. BLOCKER — both new tools fail to import on every real run. Section 3 leaves the shell in $REPO/ensembles/$ENSEMBLE and there is no cd "$REPO" before python -m tools.prereg.*, which resolves tools from the current directory (nothing is installed). Reproduced: ModuleNotFoundError: No module named 'tools.prereg' from the ensemble dir, exit 0 from the repo root. The || echo fallback then prints "posterior health reported a problem above", so a check that never ran reports its subject is broken — and P8_ANCHORS.json is never written, which is the artefact the PR body says must stop living only on a pod.

views-models/tools/podrun/pod_run_fao_delivery.sh

Lines 300 to 322 in a77f1b0

	stage pool_and_publish
	cd "$REPO/ensembles/$ENSEMBLE" || die "cannot enter $ENSEMBLE"
	export WANDB_MODE=offline WANDB_SILENT=true
	# -sa/--saved pools the member forecasts just written. Without it the ensemble refetches and
	# the eight runs above are wasted. -p publishes the wire shards (ADR-013).
	"$VENV/bin/python" main.py -r forecasting -f -sa -p || die "$ENSEMBLE pool-and-publish exited non-zero"
	# ── 3b. is there anything IN the posterior? ───────────────────────────────────────────
	# Runs between the publish and the postprocessor because that is the first moment the pooled
	# frame exists on disk. It cannot prevent anything — pooling and publishing are one invocation,
	# so the numbers are already on the shelf — but on 2026-09-30 a delivery passed every structural
	# check while `tower_point` returned zero for all 2,333,448 cells, and that was found by hand at
	# 4am. A readout every run turns "remember to look" into "cannot miss".
	stage posterior_health
	POOLED=$(ls -d "$REPO/ensembles/$ENSEMBLE/data/generated/predictions_forecasting_"* 2>/dev/null | tail -1)
	if [ -z "$POOLED" ]; then
	echo "!!! no pooled output found under $ENSEMBLE — cannot report posterior health"
	else
	MODE_ARG=production
	[ -n "$REHEARSAL_LESSONS" ] && MODE_ARG=rehearsal
	"$VENV/bin/python" -m tools.prereg.posterior_health "$POOLED" \
	--mode "$MODE_ARG" --json-out "$OUT/POSTERIOR_HEALTH.json" \
	|| echo "### posterior health reported a problem above — the delivery is already published"

2. Three copies of the pooled-frame loader land in one PR, and all three drop the validation the existing reader has. tools/collapse/collapse_predictions.py raises on a missing file, wrong ndim, <2 draws, non-finite, negative, and an identifier length that does not match the row count — the last guarded by a test written because a truncated identifiers.npz would silently label the wrong cells. None of that survives into the two tools that now run on every delivery.

views-models/tools/prereg/posterior_health.py

Lines 30 to 45 in a77f1b0

	"""Load a pooled PredictionFrame written as y_pred.npy + identifiers.npz.
	Deliberately NOT `PredictionFrame.load`: that expects `values.npy`, and the ensemble writes
	`y_pred.npy`. The mismatch cost twenty minutes on 2026-09-30 and is worth stating here rather
	than rediscovering.
	"""
	import views_frames as vf
	y = np.load(target_dir / "y_pred.npy")
	ids = np.load(target_dir / "identifiers.npz")
	index = vf.SpatioTemporalIndex(
	time=ids["time"], unit=ids["unit"], level=vf.SpatialLevel.PGM
	)
	return vf.PredictionFrame(y_pred=y, index=index), y, ids

3. STATUS is written OK unconditionally, after the run has possibly just determined the posterior is DEGENERATE or that faoapi is not serving it. Both verdicts reach stdout only. The artefact bundle that comes home says OK / mode: production for a run the script itself judged bad — the shape this PR exists to fix.

https://github.com/views-platform/views-models/blob/a77f1b09a7af7d0db8acbb04ebee6c7b0c013c8e/tools/podrun/pod_run_fao_delivery.sh#L447-L450

4. The two new $OUT artefacts are not in the start-of-run clear. A rehearsal writes P8_ANCHORS.json; a later production run that dies early leaves it in place, and the guide's rsync brings it home as that run's. P8's protection is that the anchors belong to the run being probed.

views-models/tools/podrun/pod_run_fao_delivery.sh

Lines 81 to 83 in a77f1b0

	}
	rm -f "$OUT/STATUS" "$OUT/REHEARSAL" "$OUT/PUBLISHED"
	rm -rf "$OUT/.conda_probe"

5. The registry path is re-derived instead of reusing the canonical helper. tools/credentials/platform_env.sh:59 already holds this expression. If that default moves, this preflight passes and the postprocessor still dies at step 4 — the failure the check was added to prevent.

views-models/tools/podrun/pod_run_fao_delivery.sh

Lines 233 to 235 in a77f1b0

	# the file, or set the variable, before launching.
	REGISTRY_PATH="${APPWRITE_REGISTRY:-$REPO/../views-appwrite/docs/ADRs/platform/coordinate_registry.toml}"
	if [ -f "$REGISTRY_PATH" ]; then

6. The SPARSE threshold cannot fire on the incident that motivated the tool. The 2026-09-30 posterior was 98.2% all-zero; the branch requires > 0.999. Only the DEGENERATE branch would report it, and one surviving non-zero cell out of 2.3M flips the verdict to OK. draws_nonzero_fraction is computed and used in no branch.

views-models/tools/prereg/posterior_health.py

Lines 70 to 85 in a77f1b0

	calibration is a threshold that gets argued with; this one cannot be.
	"""
	if stats["point_nonzero"] == 0:
	return "DEGENERATE", (
	"the served point estimate is zero for EVERY one of "
	f"{stats['rows']:,} cells. Whatever is delivered, a consumer reads nothing from it."
	)
	if stats["rows_all_zero_fraction"] > 0.999:
	return "SPARSE", (
	f"{stats['rows_all_zero_fraction']:.3%} of cells have no signal in any draw; "
	f"only {stats['rows_with_any_signal']:,} cells carry anything."
	)
	return "OK", (
	f"{stats['point_nonzero']:,} cells carry a non-zero point estimate "
	f"(max {stats['point_max']:.6g})."
	)

7. tools/prereg/aggregate_calibration.py (209 lines) and its 139 lines of tests appear nowhere in the PR description, are invoked by nothing, and the package docstring says the package "produces the two things" while shipping three. CLAUDE.md: "Record what you chose and why in the PR description" and "A new developer should understand the responsibilities from the package layout".

views-models/tools/prereg/__init__.py

Lines 1 to 31 in a77f1b0

	"""Evidence about a delivery that its structural checks cannot produce.
	A delivery can pass every check in the chain and still be worthless. On 2026-09-30 one did: the
	manifest was valid, coverage was exactly ``land_gaul``'s 64,742 cells, the findability guard
	resolved all 111 objects — and ``tower_point``, the estimator views-faoapi serves, returned zero
	for every one of 2,333,448 cells. The posterior was empty. Nothing in the chain said so, and it
	was found by hand at 4am by someone computing anchor values for an unrelated probe.
	This package produces the two things that would have said so, on every run:
	python -m tools.prereg.posterior_health <pooled_dir> --mode rehearsal|production
	python -m tools.prereg.capture_anchors <pooled_dir> --mode ... --out P8_ANCHORS.json
	``posterior_health`` answers *is there anything in it?* — the fraction of cells with no signal in
	any draw, the largest draw, and how many cells carry a non-zero served point estimate. The same
	numbers mean opposite things for a rehearsal and a production run, so it is told which it is; it
	cannot be inferred from the data, and that is the whole reason a human kept having to.
	``capture_anchors`` implements pre-registration v3's **P8** — *the values served ARE the values
	we produced*. It records both the point estimate and the raw draws, because the point estimate
	alone could not discriminate on the run that motivated this: comparing 0.0 to 0.0 would have
	passed against an unrelated empty dataset.
	House rules. Both read the pooled frame as the ensemble writes it — ``y_pred.npy`` plus
	``identifiers.npz``, not ``PredictionFrame.load``'s ``values.npy``. Anchor selection is seeded, so
	it is reproducible and provably not steered by what the API returned. Neither tool can PREVENT
	anything: pooling and publishing are a single invocation, so by the time a posterior can be
	inspected it is already on the shelf. They exist to make it impossible to miss, not impossible
	to happen — the refusal, if one is ever wanted, belongs where the publish decision is made
	(views-models#523).
	"""

Issues 1, 3, 4 and 5 were reproduced locally rather than inferred.

🤖 Generated with Claude Code

_{- If this code review was useful, please react with 👍. Otherwise, react with 👎.}