From 423c922002216e99b37ae547ee0311761a541234 Mon Sep 17 00:00:00 2001 From: Polichinl Date: Tue, 29 Sep 2026 23:14:43 +0200 Subject: [PATCH 01/11] =?UTF-8?q?fix(#516):=20pin=20xarray=20and=20pandas?= =?UTF-8?q?=20=E2=80=94=20the=20fresh=20env=20built=20successfully=20and?= =?UTF-8?q?=20WRONG?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit #516 said a fresh un_fao environment is unbuildable. Measured on 2026-09-29, it is not: it builds, on a different pandas MAJOR, in silence. declared: views-datafactory>=1.9.0,<2.0.0 + numpy>=1.26.4,<2.0.0 resolved: numpy 1.26.4, xarray 2025.12.0, pandas 3.0.6 the pod that produced the first FAO delivery: xarray 2024.3.0, pandas 1.5.3 Neither xarray nor pandas was pinned anywhere. The working versions existed only as hand-pins on a pod that has since been destroyed, so a fresh pod silently picks different ones. A build failure is loud; this is not, which makes it the more dangerous of the two shapes the issue conflated. views-datafactory requires xarray outright and pandas only in an optional extra that is not installed, so pandas arrives THROUGH xarray. xarray is the sole carrier — datafactory's own pyproject comment says so — and pinning the carrier fixes it. The bound is measured, not guessed. xarray 2024.3.0 is the LAST release that accepts pandas 1.x; 2024.5.0 moved to pandas>=2.0 and 2024.9.0 to pandas>=2.1. A tidy-looking `<2025` cap would therefore have been WRONG: the cliff is inside the 2024 line, not at the year boundary. Verified by resolving the file, not by reading version numbers. Declared identically in both postprocessors because they share one prefix (C-116); pinning only one lets whichever runs last decide pandas for both, which is the same trap the existing numpy pin already documents one line above. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01GG2cY2HaqmUMpwUyR5V82K --- postprocessors/un_crafd/requirements.txt | 8 ++++++++ postprocessors/un_fao/requirements.txt | 25 ++++++++++++++++++++++++ 2 files changed, 33 insertions(+) diff --git a/postprocessors/un_crafd/requirements.txt b/postprocessors/un_crafd/requirements.txt index 233a0c59..02119f18 100644 --- a/postprocessors/un_crafd/requirements.txt +++ b/postprocessors/un_crafd/requirements.txt @@ -3,3 +3,11 @@ views-datafactory>=1.9.0,<2.0.0 # install into the same envs/views-postprocessing prefix (C-116), so this pin has to be # declared identically in both or whichever runs last decides numpy for both. numpy>=1.26.4,<2.0.0 + +# xarray and pandas pinned — see the note in un_fao/requirements.txt for the measurement +# and for why the bound stops at xarray 2024.5.0. Declared identically here because both +# postprocessors install into the same envs/views-postprocessing prefix (C-116): pinning +# only one lets whichever runs last decide pandas for both, which is the same trap the +# numpy pin above already documents. +xarray>=2024.1,<2024.5 +pandas>=1.5.3,<2.0.0 diff --git a/postprocessors/un_fao/requirements.txt b/postprocessors/un_fao/requirements.txt index a18a600e..bf5d4244 100644 --- a/postprocessors/un_fao/requirements.txt +++ b/postprocessors/un_fao/requirements.txt @@ -8,3 +8,28 @@ views-datafactory>=1.9.0,<2.0.0 # Declared identically in un_crafd/requirements.txt — one shared prefix (C-116), so # pinning only one lets the other upgrade numpy out from under it on its next run. numpy>=1.26.4,<2.0.0 + +# xarray and pandas are pinned because xarray is the CARRIER, and it was the only thing +# in this file's dependency closure free to move. views-datafactory declares +# `xarray>=2024.1,<2026` as a required dependency and `pandas>=2.0` only in its optional +# `pandas` extra — which is not installed — so pandas arrives transitively THROUGH xarray +# and nothing here constrained either one. +# +# Measured 2026-09-29, resolving this file as it stood: +# declared: views-datafactory>=1.9.0,<2.0.0 + numpy>=1.26.4,<2.0.0 +# resolved: numpy 1.26.4, xarray 2025.12.0, pandas 3.0.6 +# the pod that produced the first FAO delivery ran: xarray 2024.3.0, pandas 1.5.3 +# +# So this file did not fail to build — it built, on a different pandas MAJOR, silently. +# views-models#516 originally called the environment "unbuildable"; that was wrong and the +# wrong version is the more dangerous one, because a build failure is loud. The working +# versions existed only as hand-pins on a pod that has since been destroyed. +# +# The bound is measured, not guessed. xarray 2024.3.0 is the LAST release that accepts +# pandas 1.x; 2024.5.0 moved to `pandas>=2.0` and 2024.9.0 to `pandas>=2.1`. A tidy-looking +# `<2025` cap would therefore have been wrong — the cliff is inside the 2024 line, not at +# the year boundary. Lifting these means moving the platform off pandas 1.x deliberately, +# which views-pipeline-core already permits (`pandas>=1.5.3,<3.0`) and this codebase does +# not yet assume. +xarray>=2024.1,<2024.5 +pandas>=1.5.3,<2.0.0 From 53bfba920c34a80e1a2f870649f6a2bfb938d60c Mon Sep 17 00:00:00 2001 From: Polichinl Date: Tue, 29 Sep 2026 23:14:43 +0200 Subject: [PATCH 02/11] fix(#517,#523): install the appwrite extra, prove it at preflight, add --rehearsal MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Three findings from one /falsify audit of "we are ready for a new 40-lesson run" (FALSIFIED). All three had one cause: every fix that made the 2026-09-29 run work was applied BY HAND to the pod, and the pod was destroyed. The pod was the artefact and nothing in git described it. #517 — the publish path was never installed. The string "appwrite" appeared nowhere in pod_run_model.sh, so `_build_datastore` raised at publish time, AFTER the full training run. That is exactly how the first FAO delivery attempt failed. The extra is now requested explicitly with no version, so views-hydranet's own range still decides which pipeline-core is installed, and the client is IMPORTED in preflight — the whole script is written to fail early, and the one dependency that failed late was the one not checked there. Verified by a real resolve: appwrite 13.6.1 alongside views-hydranet 0.1.2 and views-pipeline-core 3.3.4, with pandas 1.5.3 and numpy 1.26.4 intact. The same resolve confirms the C-151 toolz override is necessary rather than folklore — without it the resolver lands on toolz 0.11.2. #523 — a deliberate cheap run was reachable only by deleting the guard against an accidental one. The >=300 floor stays; `--rehearsal ` is the escape hatch. It takes the count on the command line, because main.py has no hyperparameter override and a flag without a count still forces an edit to a tracked config. It patches the POD's clone only, then re-imports to CONFIRM the patch took — a substitution that silently missed would give a 300-lesson run wearing a rehearsal label, or the reverse. The marking matters more than the permission. A rehearsal's parquets are structurally identical to a production run's and pass every check including the runner's own, so a REHEARSAL file is written beside them and MANIFEST gains `mode:` and `config: PATCHED after checkout` — the git sha alone no longer describes the run. Found while reviewing this change, not in the audit: section 2 does not re-clone when .git exists, so a production run on a pod that had rehearsed would read the LEFTOVER patch. The floor still refused it, but blamed the committed config and advised --rehearsal — a correct refusal for the wrong reason, sending the operator to edit the wrong file. It now detects the dirty config and names it. Deferred, and it is the weaker half: this runner MARKS a rehearsal but cannot REFUSE to publish one, because the publish step is not in this script. That guard belongs in views-pipeline-core. An xfail(strict) stub for it was written and DELETED — its regex spanned the whole file under re.S and XPASSed by accident, i.e. the exact class of guard-that-cannot-fire this audit exists to find. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01GG2cY2HaqmUMpwUyR5V82K --- docs/runpod_run_guide.md | 36 ++- ...t_falsification_40_lesson_run_readiness.py | 251 ++++++++++++++++++ tools/podrun/pod_run_model.sh | 191 ++++++++++++- 3 files changed, 467 insertions(+), 11 deletions(-) create mode 100644 tests/test_falsification_40_lesson_run_readiness.py diff --git a/docs/runpod_run_guide.md b/docs/runpod_run_guide.md index f5fba401..e37c73a3 100644 --- a/docs/runpod_run_guide.md +++ b/docs/runpod_run_guide.md @@ -269,7 +269,38 @@ nohup setsid bash tools/podrun/pod_run_model.sh \ `setsid` matters: the run must survive your SSH session closing and your laptop sleeping. It refuses before spending GPU time if `.netrc` is missing, if `total_lessons` is still a -throwaway value, if `REGION` is not `"land"`, or if the converter is absent. +throwaway value, if `REGION` is not `"land"`, if the Appwrite client is not importable, or +if the converter is absent. + +### Rehearsing the chain first + +After a run that failed late, you usually want the whole chain exercised cheaply before you +commit to a full one. That is `--rehearsal`, and it takes the lesson count: + +```bash +nohup setsid bash tools/podrun/pod_run_model.sh --rehearsal 40 \ + > /workspace/.nohup 2>&1 < /dev/null & +``` + +It patches **the pod's clone** of `config_hyperparameters.py` to 40 lessons, re-reads the +config to confirm the patch took, and marks the output. Nothing tracked in git is edited — +the committed configs stay at their production value, which is the point: a rehearsal +obtained by committing a low lesson count is the failure mode this flag removes. + +A rehearsal leaves an extra file, **`REHEARSAL`**, beside the parquets, and `MANIFEST` gains +`mode: REHEARSAL — NOT FIT TO DELIVER`. You need them, because the parquets themselves are +indistinguishable from a real run's: same columns, same row counts, all finite and +non-negative, every structural check green. Nothing in the data will tell you the model +behind it is undertrained. + +**What a rehearsal does NOT prove.** It exercises the chain, not the capacity. A 40-lesson +run has a different duration and memory profile from a 300-lesson one, and memory is where +this platform has failed before. A green rehearsal means the hops connect; it does not mean +the full run will fit. + +**It marks, it does not refuse.** This runner uploads nothing, so it cannot stop a rehearsal +being published downstream — the `REHEARSAL` file is a warning to a person, not a guard. Do +not publish a directory that contains one. Monitor without disturbing it: @@ -296,8 +327,9 @@ The runner already does both. Its output is in `/workspace/deliver//`: |---|---| | `parquet/` | 13 files, one per origin, 2,333,448 rows each | | `draws/` | the full posterior, zstd — ~236× smaller than raw | -| `MANIFEST` | sizes, timestamps, git sha, lesson count | +| `MANIFEST` | sizes, timestamps, git sha, lesson count, and `mode:` | | `STATUS` | `OK`, or `FAILED:` | +| `REHEARSAL` | present **only** after `--rehearsal` — this output is not fit to deliver | From your laptop: diff --git a/tests/test_falsification_40_lesson_run_readiness.py b/tests/test_falsification_40_lesson_run_readiness.py new file mode 100644 index 00000000..aecfa7b5 --- /dev/null +++ b/tests/test_falsification_40_lesson_run_readiness.py @@ -0,0 +1,251 @@ +""" +Falsification audit of the claim, 2026-09-29: +"We are ready to start a new full 40-lesson FAO delivery run — nothing remains but +creating a pod." + +Source: /falsify skill, claim mode. Verdict: FALSIFIED, three hard falsifications. + +All three had ONE cause, and it is the finding worth keeping: every fix that made the +2026-09-29 run work was applied BY HAND to the pod, and the pod was destroyed. The +repository never learned any of them. A fresh pod reproduced the original broken state. +The environment that produced the first FAO delivery existed only as a running machine — +the pod was the artefact, and nothing in git described it. + + H1 the run could not be launched at all: pod_run_model.sh refuses total_lessons < 300 + and every config declares 300, so "40 lessons" was reachable only by editing the + guard out — which produced output indistinguishable from a real run. + H2 a fresh env built SUCCESSFULLY and WRONG: the declared requirements resolved to + xarray 2025.12.0 / pandas 3.0.6 where the working pod had 2024.3.0 / 1.5.3, and + neither was pinned anywhere. views-models#516 called this "unbuildable"; it was not, + and the silent version is the more dangerous of the two. + H3 `appwrite` appeared nowhere in pod_run_model.sh, so `_build_datastore` failed at + publish — after the full training run. That is how 2026-09-29 failed (#517). + +These tests now GUARD those three fixes rather than assert the defects. They are kept as a +single file because they share one cause: if a fourth instance appears, it belongs here. + +Two assertions in this file were written as predicted falsifications and SURVIVED — the +toolz ordering and the guide's chmod warning. They are retained deliberately. A +falsification file containing only the author's successful predictions is a record of an +argument, not of an audit. + +ONE ITEM IS DEFERRED, not fixed: a rehearsal's output is MARKED unfit to deliver but +nothing REFUSES to publish it. That guard belongs in views-pipeline-core, where the publish +happens, so this file cannot test it. See TestARehearsalCannotBeMistakenForADelivery. +""" + +import re +from pathlib import Path + +import pytest + +REPO = Path(__file__).resolve().parents[1] +PODRUN = REPO / "tools" / "podrun" / "pod_run_model.sh" + +HYDRANETS = ( + "purple_alien", "pink_pirate", "blue_stranger", "bright_starship", + "heavy_freighter", "blazing_meteor", "bold_comet", "violet_visitor", +) + + +class TestACheapRunIsReachableWithoutEditingATrackedFile: + """H1. The floor was right; its absence of an escape hatch was not. + + The >=300 floor protects against spending a full budget by ACCIDENT — a config left at + a sweep value, a stale checkout. It was never meant to forbid a deliberate cheap + end-to-end test, which is routine and is exactly what you want after a run that failed + late. Before `--rehearsal` the only way to get one was to delete the guard, and the + result looked identical on disk to a real run: the cheap test and the accident were + indistinguishable. That, not the wasted GPU time, was the hazard. + """ + + def test_the_rehearsal_flag_exists_and_takes_a_lesson_count(self): + src = PODRUN.read_text() + assert "--rehearsal" in src, "no escape hatch: a cheap end-to-end test needs the guard deleted" + assert "REHEARSAL_LESSONS" in src, ( + "--rehearsal must carry the lesson count on the command line. main.py has no " + "hyperparameter override, so a flag without a count still forces an edit to a " + "tracked config — the hand-edit this audit exists to remove." + ) + + def test_the_production_floor_is_still_enforced(self): + src = PODRUN.read_text() + assert re.search(r"if not requested and lessons < 300:", src), ( + "the >=300 floor must still apply when --rehearsal is NOT given. Adding the " + "escape hatch must not widen the hole it was protecting." + ) + + @pytest.mark.parametrize("model", HYDRANETS) + def test_tracked_configs_still_declare_the_production_count(self, model): + """A rehearsal patches the pod's clone. The committed config stays production truth.""" + cfg = REPO / "models" / model / "configs" / "config_hyperparameters.py" + hit = re.search(r"'total_lessons':\s*(\d+)", cfg.read_text()) + assert hit is not None, f"{model}: no total_lessons in {cfg}" + assert int(hit.group(1)) >= 300, ( + f"{model} declares total_lessons={hit.group(1)}. A rehearsal must never be " + "obtained by committing a low count — that is the state this audit found and " + "the reason --rehearsal patches only the pod's ephemeral checkout." + ) + + def test_a_leftover_rehearsal_patch_is_named_not_blamed_on_the_config(self): + """Found reviewing this change, not in the original audit. + + Section 2 does not re-clone when `.git` already exists, so a production run started + on a pod that has already rehearsed reads the LEFTOVER patch. The floor still refuses + it — but it would blame the committed config and advise `--rehearsal`, which is the + exact opposite of what the operator wants. A correct refusal for the wrong reason + sends someone to edit the wrong file. + """ + src = PODRUN.read_text() + assert "is MODIFIED in this checkout" in src, ( + "a production run must detect a leftover rehearsal patch and name it, rather " + "than reporting the patched value as though it were the committed config" + ) + + def test_the_patch_is_verified_by_re_reading_the_config(self): + """A substitution that silently missed would produce a 300-lesson run wearing a + rehearsal label, or a rehearsal wearing none. Both are worse than a refusal.""" + src = PODRUN.read_text() + assert "importlib.invalidate_caches()" in src and "still reports total_lessons" in src, ( + "the config patch must be confirmed by re-importing the config, not trusted " + "from re.subn's return value alone" + ) + + +class TestFreshEnvironmentReproducesTheOneThatWorked: + """H2 (views-models#516, reframed). Measured, not reasoned about. + + declared: views-datafactory>=1.9.0,<2.0.0 + numpy>=1.26.4,<2.0.0 + resolved: numpy 1.26.4, xarray 2025.12.0, pandas 3.0.6 + the pod that delivered: xarray 2024.3.0, pandas 1.5.3 + + views-datafactory requires xarray outright and pandas only in an optional extra that is + not installed, so pandas arrives THROUGH xarray — xarray is the sole carrier, as + datafactory's own pyproject comment says. Pinning the carrier fixes it. + + The bound is measured: xarray 2024.3.0 is the LAST release accepting pandas 1.x; 2024.5.0 + moved to pandas>=2.0 and 2024.9.0 to pandas>=2.1. A tidy `<2025` cap would have been + WRONG — the cliff is inside the 2024 line, not at the year boundary. + """ + + @pytest.mark.parametrize("postprocessor", ["un_fao", "un_crafd"]) + def test_the_carrier_and_the_passenger_are_both_pinned(self, postprocessor): + req = REPO / "postprocessors" / postprocessor / "requirements.txt" + text = req.read_text() + missing = [p for p in ("xarray", "pandas") if not re.search(rf"^{p}[><=~]", text, re.M)] + assert not missing, ( + f"{req.relative_to(REPO)} does not pin {missing}. Unpinned, this file resolves " + "to pandas 3.0.6 — a different MAJOR from every run that has ever succeeded, " + "with no error. See views-models#516." + ) + + def test_both_postprocessors_declare_identical_pins(self): + """C-116: one shared prefix. Pinning one lets whichever runs last decide for both.""" + pins = {} + for name in ("un_fao", "un_crafd"): + text = (REPO / "postprocessors" / name / "requirements.txt").read_text() + pins[name] = sorted( + re.findall(r"^((?:numpy|pandas|xarray)[><=~][^\s#]*)", text, re.M) + ) + assert pins["un_fao"] == pins["un_crafd"], ( + f"un_fao pins {pins['un_fao']} but un_crafd pins {pins['un_crafd']}. Both " + "install into envs/views-postprocessing (C-116); divergent pins mean the last " + "postprocessor to run silently decides the versions for the other." + ) + + +class TestThePodCanActuallyPublish: + """H3 (views-models#517). The publish path has to be installed and PROVEN at preflight. + + Verified by a real resolve on 2026-09-29: requesting the extra yields appwrite 13.6.1 + alongside views-hydranet 0.1.2 and views-pipeline-core 3.3.4 — both of which carry the + fixes shipped that day — with pandas 1.5.3 and numpy 1.26.4 intact. + """ + + def test_the_appwrite_extra_is_installed(self): + src = PODRUN.read_text() + assert "views-pipeline-core[appwrite]" in src, ( + "the appwrite extra is never installed, so the run trains for hours and then " + "cannot publish. This is the failure of 2026-09-29. See views-models#517." + ) + + def test_the_publish_path_is_checked_before_the_money_is_spent(self): + """An import in preflight costs seconds. Discovering it at publish costs the run.""" + src = PODRUN.read_text() + verify = src.split("stage verify_env", 1)[-1].split("stage", 1)[0] + assert "appwrite" in verify, ( + "pod_run_model.sh verifies torch, tlz, hydranet and datafactory at preflight but " + "not the Appwrite client. The whole script is written to fail early; the one " + "dependency that failed late was the one not checked here." + ) + + def test_the_toolz_override_is_reasserted_after_the_last_install(self): + """GREEN, and it survived as a prediction. Confirmed necessary, not folklore: the + resolver really does land on toolz 0.11.2 for this dependency set.""" + src = PODRUN.read_text() + installs = [m.start() for m in re.finditer(r"pip install", src)] + overrides = [m.start() for m in re.finditer(r"toolz>=0\.12", src)] + assert overrides, "the C-151 toolz override is gone entirely" + assert max(overrides) > max(installs), ( + "the C-151 toolz override must be the LAST install: anything after it " + "re-resolves the prefix and can pull toolz back under 0.12, which happened by " + "hand on 2026-09-29 (1.1.0 -> 0.11.2) with no error." + ) + + +class TestARehearsalCannotBeMistakenForADelivery: + """The half that actually prevents harm. + + A rehearsal's parquets are structurally identical to a production run's — same columns, + row counts, names, all finite and non-negative. Every structural check passes. Nothing + in the DATA says the model behind it is undertrained. + """ + + def test_the_run_marks_its_output(self): + src = PODRUN.read_text() + assert '"$OUT/REHEARSAL"' in src and "NOT FIT TO DELIVER" in src, ( + "a rehearsal must leave an unmistakable marker beside its parquets" + ) + + def test_a_stale_marker_cannot_be_inherited(self): + """Worse than a stale marker on a real run: a rehearsal inheriting a production + run's ABSENCE of one.""" + src = PODRUN.read_text() + assert 'rm -f "$OUT/REHEARSAL"' in src, ( + "the marker must be cleared with the other stale per-run state, or a rehearsal " + "in a directory left by a production run carries no mark at all" + ) + + def test_the_manifest_admits_the_config_was_patched(self): + src = PODRUN.read_text() + assert "PATCHED after checkout" in src, ( + "on a rehearsal the MANIFEST's `git:` sha no longer describes the run — the " + "config was patched after checkout. A manifest that must be cross-read with a " + "command-line flag to be understood will be misread." + ) + + # NOT TESTED HERE, DELIBERATELY. pod_run_model.sh MARKS a rehearsal; it cannot REFUSE + # to publish one, because the publish step is not in this script — the header is accurate + # that this runner uploads nothing. The refusal belongs wherever a forecast is handed to + # a store, which is views-pipeline-core, so no assertion in THIS repo can observe it. + # + # An xfail(strict) stub was written here and deleted: its regex spanned the whole file + # under re.S and XPASSed by accident, i.e. it was the exact class of guard-that-cannot- + # fire this audit exists to find. A test that cannot observe its subject is worse than no + # test, because it reports coverage. Tracked as an issue instead; see the module + # docstring. Until that guard exists, $OUT/REHEARSAL is the only thing between an + # undertrained model and a partner, and that is weaker than it should be. + + +class TestPublishCredentialsHaveADurableHome: + """GREEN (survived), and the prediction was wrong in the useful direction: the guide + DOES record the /workspace chmod trap, so the operator-facing half of C-154 / #518 is + already durable. What remains is that placement is still a manual step with a correct + runbook — an operator task, not a defect, and not counted as a falsification.""" + + def test_the_guide_warns_that_workspace_ignores_chmod(self): + guide = REPO / "docs" / "runpod_run_guide.md" + assert re.search(r"/workspace.*chmod|chmod.*/workspace", guide.read_text(), re.I | re.S), ( + "/workspace is a network filesystem that reports chmod success and leaves the " + "file world-readable. C-154 / views-models#518." + ) diff --git a/tools/podrun/pod_run_model.sh b/tools/podrun/pod_run_model.sh index ccd63040..4727bd32 100755 --- a/tools/podrun/pod_run_model.sh +++ b/tools/podrun/pod_run_model.sh @@ -1,5 +1,5 @@ #!/usr/bin/env bash -# pod_run_model.sh +# pod_run_model.sh [--rehearsal] # # ┌──────────────────────────────────────────────────────────────────────────────────────┐ # │ VERSION 0.1.0 — PROVISIONAL. NOT part of the monthly production run. │ @@ -21,12 +21,51 @@ # /workspace/deliver//STATUS OK or FAILED: # /workspace/deliver//FAILURE the reason, on failure only (written directly) # /workspace/deliver//run.log the whole transcript +# /workspace/deliver//REHEARSAL present ONLY on a --rehearsal run +# +# TWO MODES. +# (default) production. total_lessons must be >= 300, and the run refuses otherwise. +# --rehearsal exercises the whole chain on a deliberately undertrained model at +# lessons. Patches the POD's clone of the config — never a tracked file +# — and marks the output as unfit to deliver. +# +# The lesson floor exists because a 300-lesson budget can be spent by ACCIDENT — a config +# left at a sweep value, a stale checkout. It was never meant to forbid a deliberate cheap +# end-to-end test, which is a routine and necessary thing to want, especially after a run +# that failed late. Before --rehearsal existed the only way to get one was to edit the guard +# out, which produced output indistinguishable from a real run: the cheap test and the +# accident looked the same on disk. That is the actual hazard, and it is what REHEARSAL and +# the MANIFEST `mode:` line address. Wasted GPU time is recoverable; an undertrained +# forecast reaching a partner as a real one is not. # # Progress is readable from outside at any time: cat /workspace/deliver//STAGE set -uo pipefail -MODEL="${1:?usage: pod_run_model.sh }" +# --rehearsal takes the lesson count rather than reading it from the config, so a rehearsal +# needs NO edit to a tracked file. main.py has no hyperparameter override, so the count has +# to reach the model through its config; this script patches the POD's ephemeral clone and +# verifies the patch (see the config check). The count is REQUIRED — there is exactly one way +# to ask for a rehearsal, and it states the number out loud in the command that starts it. +USAGE='usage: pod_run_model.sh [--rehearsal ] ' +REHEARSAL_LESSONS="" +while [ $# -gt 0 ]; do + case "$1" in + --rehearsal) + [ $# -ge 2 ] || { echo "!!! --rehearsal needs a lesson count, e.g. --rehearsal 40" >&2 + echo "!!! $USAGE" >&2; exit 1; } + REHEARSAL_LESSONS="$2" + case "$REHEARSAL_LESSONS" in + ''|*[!0-9]*) echo "!!! --rehearsal needs a positive integer, got: $REHEARSAL_LESSONS" >&2 + exit 1 ;; + esac + [ "$REHEARSAL_LESSONS" -ge 1 ] || { echo "!!! --rehearsal 0 has nothing to rehearse" >&2; exit 1; } + shift 2 ;; + --*) echo "!!! unknown option: $1" >&2; echo "!!! $USAGE" >&2; exit 1 ;; + *) break ;; + esac +done +MODEL="${1:?$USAGE}" ROOT=/workspace REPO=$ROOT/views-models VENV=$ROOT/venv @@ -38,6 +77,10 @@ mkdir -p "$OUT" # or `cat STATUS` reports that old failure for the whole of this run -- and the header # advertises STATUS/STAGE as the way to watch from outside, so a stale one actively misleads. rm -f "$OUT/STATUS" +# Same reasoning for the rehearsal marker: a production run in a directory left behind by an +# earlier rehearsal must not inherit its "undeliverable" mark, and — far worse — a rehearsal +# must not inherit a previous production run's ABSENCE of one. +rm -f "$OUT/REHEARSAL" "$OUT/.lessons" exec > >(tee -a "$LOG") 2>&1 stage() { echo "$1" > "$OUT/STAGE"; echo "=== [$(date +%H:%M:%S)] $1 ==="; } @@ -76,10 +119,22 @@ if [ ! -x "$VENV/bin/python" ]; then stage install_python uv venv --python 3.11 "$VENV" || die "venv creation failed" + # views-pipeline-core[appwrite] is requested EXPLICITLY, with no version, so + # views-hydranet's own range still decides which pipeline-core is installed. The extra is + # what carries the Appwrite SDK, and without it `_build_datastore` raises at publish time + # — which is AFTER the full training run. That is exactly how the first FAO delivery + # attempt failed on 2026-09-29 (views-models#517); it was fixed by hand on a pod that no + # longer exists, so the repository never learned it. uv pip install --python "$VENV/bin/python" \ - "views-hydranet~=0.1.1" "views-datafactory>=1.9.0,<2.0.0" || die "pip install failed" + "views-hydranet~=0.1.1" "views-datafactory>=1.9.0,<2.0.0" \ + "views-pipeline-core[appwrite]" || die "pip install failed" # register C-151: viewser pins toolz<0.12, which cannot import tlz submodules on # Python 3.11 and breaks EVERY datafactory fetch. Override after resolution. + # + # This MUST stay the last install in this block. Any pip install appended below it + # re-resolves this prefix and can silently pull toolz back under 0.12 — which happened on + # 2026-09-29, when installing the appwrite extra by hand reverted it 1.1.0 -> 0.11.2 with + # no error. Pinned by tests/test_falsification_40_lesson_run_readiness.py. uv pip install --python "$VENV/bin/python" "toolz>=0.12.1" || die "toolz override failed" else echo "venv already present — reusing" @@ -90,6 +145,18 @@ stage verify_env import torch, tlz.curried, views_hydranet, views_pipeline_core, datafactory_query assert torch.cuda.is_available(), "CUDA not available" print("torch", torch.__version__, "cap", torch.cuda.get_device_capability()) + +# The publish path is verified HERE, in preflight, not discovered at publish time. Without +# the appwrite extra this import is the only thing between a green-looking pod and a run +# that trains for hours and then cannot hand over its forecasts (#517). Importing the +# client is a weaker check than publishing, but it is the strongest one available before +# there is anything to publish — and it is the check whose absence cost the first delivery. +import appwrite # noqa: F401 — the SDK itself; views-pipeline-core[appwrite] provides it +from views_pipeline_core.modules.appwrite import file as _appwrite_file # noqa: F401 +print("appwrite client importable — the publish path exists") + +import pandas, numpy +print("pandas", pandas.__version__, "numpy", numpy.__version__) PY # ── 2. repo ─────────────────────────────────────────────────────────────────────── @@ -108,8 +175,12 @@ stage check_config # the same defect this repo has already shipped once (#501, "the guard that was not one"): a # substring assertion satisfied by a COMMENT recording the value's history, so the guard passed # on the wrong region. A comment cannot satisfy this one. +# REHEARSAL/OUT/MODEL reach the script through the environment: the heredoc is quoted, so +# the shell does not interpolate into it, and that is deliberate — the config values must +# come from the config file, not from string substitution. +REHEARSAL_LESSONS="$REHEARSAL_LESSONS" OUT="$OUT" MODEL="$MODEL" \ "$VENV/bin/python" - "$REPO/models/$MODEL" <<'CFGCHECK' || die "config check failed" -import importlib.util, sys +import importlib, importlib.util, os, re, subprocess, sys from pathlib import Path model = Path(sys.argv[1]) @@ -121,11 +192,73 @@ def load(name): spec.loader.exec_module(mod) return mod -hp = load("config_hyperparameters").get_hp_config() -lessons = hp["total_lessons"] +cfg_path = model / "configs" / "config_hyperparameters.py" +requested = os.environ.get("REHEARSAL_LESSONS") or "" + +lessons = load("config_hyperparameters").get_hp_config()["total_lessons"] print("total_lessons:", lessons, "(read from the config, not the file text)") -if lessons < 300: - sys.exit("total_lessons is %s, expected >= 300 - this pod would train a throwaway model" % lessons) + +if not requested: + # A rehearsal patches this pod's clone, and section 2 does NOT re-clone when .git already + # exists — so a production run started on a pod that has rehearsed reads the LEFTOVER + # patch. It would refuse (the floor holds), but it would blame the committed config and + # tell the operator to use --rehearsal, which is the opposite of what they want. Name the + # real cause instead. Checked before the floor so the accurate message wins. + dirty = subprocess.run( + ["git", "-C", str(model), "status", "--porcelain", "--", str(cfg_path)], + capture_output=True, text=True, + ) + if dirty.returncode == 0 and dirty.stdout.strip(): + sys.exit( + "%s is MODIFIED in this checkout, so total_lessons=%s is not what the committed\n" + " config says. This pod has almost certainly run --rehearsal already, and that\n" + " patch is still in place. A production run must start from the committed config:\n" + " git -C %s checkout -- %s\n" + " Refusing rather than training on a config neither of us chose." + % (cfg_path, lessons, model, cfg_path.relative_to(model)) + ) + +if requested: + # Patch the POD's clone. This is a throwaway checkout on rented hardware; the tracked + # config keeps saying 300, which is the production truth and must not be edited to get a + # cheap test. Substitution is anchored on the same literal the file is known to contain. + target = int(requested) + text = cfg_path.read_text() + patched, n = re.subn(r"('total_lessons'\s*:\s*)\d+", r"\g<1>%d" % target, text) + if n != 1: + sys.exit( + "--rehearsal %d: expected exactly one 'total_lessons': in %s, found %d.\n" + " Refusing rather than guessing which one to patch." + % (target, cfg_path, n) + ) + cfg_path.write_text(patched) + + # VERIFY by re-importing, not by trusting the substitution. A patch that silently failed + # would otherwise produce a 300-lesson run wearing a rehearsal label, or the reverse. + importlib.invalidate_caches() + lessons = load("config_hyperparameters").get_hp_config()["total_lessons"] + if lessons != target: + sys.exit( + "--rehearsal %d: patched %s but it still reports total_lessons=%s. Not proceeding." + % (target, cfg_path, lessons) + ) + print("REHEARSAL: patched the pod's config to %d lessons and re-read it to confirm." % lessons) + print(" Output will be marked NOT FIT TO DELIVER.") + +# Recorded for the MANIFEST, so the manifest reports the value the run was GATED on rather +# than re-deriving it by grepping the file text. Two readings of one number can disagree. +Path(os.environ["OUT"], ".lessons").write_text(str(lessons)) + +if not requested and lessons < 300: + sys.exit( + "total_lessons is %s, expected >= 300 - this pod would train a throwaway model.\n" + " If you MEANT a cheap end-to-end test, that is what --rehearsal is for:\n" + " pod_run_model.sh --rehearsal %s %s\n" + " It takes the lesson count on the command line, patches only the pod's clone, and\n" + " marks the output as unfit to deliver — so a deliberate cheap run cannot be\n" + " confused with an accidental one, and no tracked config has to be edited." + % (lessons, lessons, os.environ.get("MODEL", "")) + ) region = getattr(load("config_queryset"), "REGION", None) print("REGION:", region) @@ -210,10 +343,50 @@ stage manifest echo "compressed: $(du -sh "$OUT/draws" | cut -f1)" echo "parquets: $(du -sh "$OUT/parquet" | cut -f1)" echo "git: $(git -C "$REPO" rev-parse --short HEAD)" - echo "lessons: $(grep -oE "'total_lessons': [0-9]+" "$REPO/models/$MODEL/configs/config_hyperparameters.py" | head -1)" + # The value the run was GATED on, written by the config check — not a second, independent + # grep of the file text, which could disagree with it and be believed. + echo "lessons: $(cat "$OUT/.lessons" 2>/dev/null || echo unknown)" + if [ -n "$REHEARSAL_LESSONS" ]; then + echo "mode: REHEARSAL — NOT FIT TO DELIVER" + # Stated explicitly because the `git:` line above no longer fully describes the run: the + # pod's config_hyperparameters.py was patched after checkout, so that sha alone would + # imply 300 lessons. A manifest that has to be cross-read with a flag is a manifest that + # will be misread. + echo "config: PATCHED after checkout — total_lessons forced to $REHEARSAL_LESSONS" + else + echo "mode: production" + echo "config: as committed at the git sha above" + fi } > "$OUT/MANIFEST" cat "$OUT/MANIFEST" +# ── the mark that makes a rehearsal unmistakable downstream ─────────────────────── +# A rehearsal's parquets are structurally identical to a production run's: same columns, +# same row counts, same names, same finite non-negative values. Every check in section 4 +# passes. Nothing about the FILES says the model behind them is undertrained, which is why +# this has to be a separate artefact that travels with them. +# +# This MARKS; it cannot REFUSE. The publish step is not in this script (the header is +# accurate: this runner uploads nothing), so the refusal has to live wherever the forecast +# is handed to a store. Until it does, this file is the only thing standing between a +# rehearsal and a partner, and that is a weaker guarantee than it should be — tracked as +# the second half of the rehearsal work, not as done. +if [ -n "$REHEARSAL_LESSONS" ]; then + { + echo "THIS OUTPUT IS A REHEARSAL. DO NOT DELIVER IT." + echo + echo "model: $MODEL" + echo "lessons: $(cat "$OUT/.lessons" 2>/dev/null || echo unknown)" + echo "finished: $(date -u +%Y-%m-%dT%H:%M:%SZ)" + echo + echo "It was produced with --rehearsal to exercise the pipeline end to end. The model is" + echo "deliberately undertrained. The parquets are well-formed and will pass every" + echo "structural check, including this runner's own — that is precisely why this file" + echo "exists. Nothing in the data itself will tell you." + } > "$OUT/REHEARSAL" + echo "### WROTE $OUT/REHEARSAL — this output is NOT fit to deliver" +fi + echo OK > "$OUT/STATUS" stage done echo "### $MODEL — COMPLETE" From 1e081fb37fc2080f216394cd510590310608b309 Mon Sep 17 00:00:00 2001 From: Polichinl Date: Tue, 29 Sep 2026 23:28:35 +0200 Subject: [PATCH 03/11] =?UTF-8?q?fix(#518):=20the=20guards=20were=20decora?= =?UTF-8?q?tive=20=E2=80=94=20execute=20the=20script,=20and=20write=20the?= =?UTF-8?q?=20missing=20warning?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit An independent /falsify guard-mode audit of 53bfba92 reverted EVERY fix in that commit, kept the comments, and got 22/22 green. 20 of 24 mutations survived. 10 of 14 guards were DECORATIVE, 4 WEAK, none held. The mechanism is the lesson. Every assertion read pod_run_model.sh as TEXT, and that script is unusually well commented — each fix carries a paragraph naming the incident and quoting the exact strings. So THE BETTER THE COMMENT, THE WEAKER THE GUARD: deleting the code left the comment, and the comment satisfied the assertion. Deleting the appwrite extra from the install line was invisible because the comment above it names views-pipeline-core[appwrite]. Worst survivor: `os.environ.get("REHEARSAL_LESSONS") or ""` -> `or "40"`. One word. Every production run then patches itself to 40 lessons, the floor is dead, the leftover-patch detector is dead, and because the SHELL variable stays empty the MANIFEST says `mode: production` and no REHEARSAL marker is written. A 40-lesson model labelled a production delivery, suite fully green. AND ONE GUARD CERTIFIED A SAFETY PROPERTY THAT DID NOT EXIST. It claimed the RunPod guide documents the /workspace chmod trap. The guide did not: no "world-readable", no "network filesystem", no C-154, no #518. It passed because `/workspace` is on line 104 and `chmod` on line 180, joined by `.*` under re.S — the identical defect the previous commit message boasted of having found and deleted elsewhere. That is worse than no guard, because it stopped the next reader looking. The warning is now written (#518, C-154): /workspace is a network filesystem where chmod 600 returns success and does nothing, leaving a credential at mode 666 with no error to notice. What changed in the tests: - the config-check program is EXTRACTED FROM ITS HEREDOC AND EXECUTED against fixture models in real git repos. That block holds all the rehearsal/production logic and was wholly unguarded. Includes an adversarial fixture whose literal is patchable but whose get_hp_config() returns 300 regardless, so a "verification" that compared target against itself is caught. - remaining text assertions read COMMENT-STRIPPED source, so a comment can never stand in for code. - requirements guards assert SPECIFIER SEMANTICS via packaging.SpecifierSet — is 1.5.3 admitted, is 3.0.6 refused — instead of pin presence. `pandas>=3.0` passed the old one, and marker-gated pins (`; python_version < "3.10"`, inert on 3.11) defeated the identical-pins check while keeping the captured text identical. - the roster guard CALLS get_hp_config() instead of matching the literal, which is what pod_run_model.sh itself insists on, citing #501 "the guard that was not one". pod_run_model.sh warned against this exact failure in three places and the guards did it anyway. That is the argument for the independence rule: the author's own model named the trap three times in one commit and stepped into it. 35 tests, up from 22. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01GG2cY2HaqmUMpwUyR5V82K --- docs/runpod_run_guide.md | 16 + ...t_falsification_40_lesson_run_readiness.py | 587 ++++++++++++------ 2 files changed, 410 insertions(+), 193 deletions(-) diff --git a/docs/runpod_run_guide.md b/docs/runpod_run_guide.md index e37c73a3..4eacb405 100644 --- a/docs/runpod_run_guide.md +++ b/docs/runpod_run_guide.md @@ -180,6 +180,22 @@ grep -A2 '' ~/.netrc | ssh root@ -p -i ~/.ssh/id_ed25 'umask 077; cat > /root/.netrc; chmod 600 /root/.netrc' ``` +> **Put credentials on `/root`, never on `/workspace` — `/workspace` silently ignores +> `chmod`.** +> +> It is a network filesystem. `chmod 600` there **returns success and does nothing**: the +> file stays mode `666`, readable by every process on the machine, and there is no error to +> notice. `stat -c %a` is the only way to find out, and only if you think to look. +> +> Found on 2026-09-29 while placing the Appwrite publish credentials, which sat +> world-readable on rented hardware until they were moved to `/root/.secrets`. Registered as +> **views-models C-154**; see **#518**. +> +> This is why the command above writes to `/root/.netrc` and why the runner checks +> `stat -c %a /root/.netrc` rather than trusting its own `chmod`. Both are deliberate. The +> `umask 077` is what actually protects the file in transit — the `chmod` is a belt-and-braces +> check that happens to work *here* because `/root` is local disk. + Check it before trusting it — the pipe can silently deliver nothing: ```bash diff --git a/tests/test_falsification_40_lesson_run_readiness.py b/tests/test_falsification_40_lesson_run_readiness.py index aecfa7b5..b45abafe 100644 --- a/tests/test_falsification_40_lesson_run_readiness.py +++ b/tests/test_falsification_40_lesson_run_readiness.py @@ -5,39 +5,59 @@ Source: /falsify skill, claim mode. Verdict: FALSIFIED, three hard falsifications. -All three had ONE cause, and it is the finding worth keeping: every fix that made the -2026-09-29 run work was applied BY HAND to the pod, and the pod was destroyed. The -repository never learned any of them. A fresh pod reproduced the original broken state. -The environment that produced the first FAO delivery existed only as a running machine — -the pod was the artefact, and nothing in git described it. - - H1 the run could not be launched at all: pod_run_model.sh refuses total_lessons < 300 - and every config declares 300, so "40 lessons" was reachable only by editing the - guard out — which produced output indistinguishable from a real run. +All three had ONE cause: every fix that made the 2026-09-29 run work was applied BY HAND +to the pod, and the pod was destroyed. The repository never learned any of them. The pod +was the artefact and nothing in git described it. + + H1 the run could not be launched: pod_run_model.sh refuses total_lessons < 300 and every + config declares 300, so a cheap end-to-end test was reachable only by deleting the + guard — producing output indistinguishable from a real run. H2 a fresh env built SUCCESSFULLY and WRONG: the declared requirements resolved to - xarray 2025.12.0 / pandas 3.0.6 where the working pod had 2024.3.0 / 1.5.3, and - neither was pinned anywhere. views-models#516 called this "unbuildable"; it was not, - and the silent version is the more dangerous of the two. + xarray 2025.12.0 / pandas 3.0.6 where the working pod had 2024.3.0 / 1.5.3, neither + pinned. #516 called this "unbuildable"; it is not, and the silent version is worse. H3 `appwrite` appeared nowhere in pod_run_model.sh, so `_build_datastore` failed at publish — after the full training run. That is how 2026-09-29 failed (#517). -These tests now GUARD those three fixes rather than assert the defects. They are kept as a -single file because they share one cause: if a fourth instance appears, it belongs here. - -Two assertions in this file were written as predicted falsifications and SURVIVED — the -toolz ordering and the guide's chmod warning. They are retained deliberately. A -falsification file containing only the author's successful predictions is a record of an -argument, not of an audit. - -ONE ITEM IS DEFERRED, not fixed: a rehearsal's output is MARKED unfit to deliver but -nothing REFUSES to publish it. That guard belongs in views-pipeline-core, where the publish -happens, so this file cannot test it. See TestARehearsalCannotBeMistakenForADelivery. +──────────────────────────────────────────────────────────────────────────────────────── +SECOND AUDIT, same day: this file's FIRST version was 22 green assertions that protected +NOTHING. An independent /falsify guard-mode pass reverted every fix in the commit, kept the +comments, and got 22/22 green. 20 of 24 mutations survived; 10 guards were DECORATIVE. + +The mechanism, and it is the lesson: every assertion read the script as TEXT, and +pod_run_model.sh is unusually well commented — each fix carries a paragraph naming the +incident and quoting the exact strings. So THE BETTER THE COMMENT, THE WEAKER THE GUARD: +deleting the code left the comment, and the comment satisfied the assertion. + +Worst single case: `requested = os.environ.get("REHEARSAL_LESSONS") or ""` changed to +`or "40"`. One word. Every production run then patches itself to 40 lessons, the floor is +dead, and because the SHELL variable stays empty the MANIFEST says `mode: production` and +no REHEARSAL marker is written. A 40-lesson model, labelled a production delivery, 22/22 +green. Two of the five defects fused by a one-token diff. + +Worse still, one assertion certified a safety property that DOES NOT EXIST: it claimed the +RunPod guide documents the /workspace chmod trap. It does not. The regex matched because +`/workspace` appears on line 104 and `chmod` on line 180, joined by `.*` under re.S — the +identical defect the first commit message boasted of having found and deleted elsewhere. + +So this version: + * EXECUTES the config-check program against fixture models, rather than grepping it. That + block is where all the rehearsal/production logic lives and it was wholly unguarded. + * STRIPS COMMENTS before any remaining text assertion, so a comment can never stand in + for code. + * asserts version-specifier SEMANTICS (is 1.5.3 allowed? is 3.0.6 refused?) instead of + the mere presence of a pin — `pandas>=3.0` passed the old guard. + * CALLS get_hp_config() instead of pattern-matching the literal, which is what + pod_run_model.sh itself insists on doing, citing #501 "the guard that was not one". """ +import importlib.util import re +import subprocess +import sys from pathlib import Path import pytest +from packaging.specifiers import SpecifierSet REPO = Path(__file__).resolve().parents[1] PODRUN = REPO / "tools" / "podrun" / "pod_run_model.sh" @@ -48,204 +68,385 @@ ) -class TestACheapRunIsReachableWithoutEditingATrackedFile: - """H1. The floor was right; its absence of an escape hatch was not. +def _code_only(src: str) -> str: + """Shell source with comments removed. - The >=300 floor protects against spending a full budget by ACCIDENT — a config left at - a sweep value, a stale checkout. It was never meant to forbid a deliberate cheap - end-to-end test, which is routine and is exactly what you want after a run that failed - late. Before `--rehearsal` the only way to get one was to delete the guard, and the - result looked identical on disk to a real run: the cheap test and the accident were - indistinguishable. That, not the wasted GPU time, was the hazard. + Every decorative guard in the first version of this file was satisfied by a comment + after its code was deleted. Any assertion about what the script DOES must therefore + read only what the script RUNS. Full-line and trailing comments both go; this is + deliberately cruder than a shell parser, and cruder is the safe direction here — it + removes text an assertion might otherwise lean on. """ + out = [] + for line in src.splitlines(): + stripped = re.sub(r"(? str: + """The config-check program, lifted out of its heredoc so it can be RUN.""" + src = PODRUN.read_text() + m = re.search(r"<<'CFGCHECK'[^\n]*\n(.*?)\nCFGCHECK\s*?\n", src, re.S) + assert m, "the CFGCHECK heredoc is gone from pod_run_model.sh — re-read the script" + return m.group(1) + + +def _make_model(tmp_path: Path, lessons_body: str, region: str = '"land"') -> Path: + """A fixture model directory inside a real git repo. - def test_the_rehearsal_flag_exists_and_takes_a_lesson_count(self): - src = PODRUN.read_text() - assert "--rehearsal" in src, "no escape hatch: a cheap end-to-end test needs the guard deleted" - assert "REHEARSAL_LESSONS" in src, ( - "--rehearsal must carry the lesson count on the command line. main.py has no " - "hyperparameter override, so a flag without a count still forces an edit to a " - "tracked config — the hand-edit this audit exists to remove." + Real git, because the leftover-patch detector runs `git status --porcelain`; a fake + would let that guard pass without the mechanism it depends on existing. + """ + model = tmp_path / "model" + (model / "configs").mkdir(parents=True) + (model / "configs" / "config_hyperparameters.py").write_text(lessons_body) + (model / "configs" / "config_queryset.py").write_text(f"REGION = {region}\n") + for cmd in ( + ["git", "init", "-q", "."], + ["git", "config", "user.email", "t@t"], + ["git", "config", "user.name", "t"], + ["git", "add", "-A"], + ["git", "commit", "-qm", "fixture"], + ): + subprocess.run(cmd, cwd=model, check=True, capture_output=True) + return model + + +def _run_cfgcheck(model: Path, out: Path, rehearsal: str = "") -> subprocess.CompletedProcess: + out.mkdir(parents=True, exist_ok=True) + script = out / "_cfgcheck.py" + script.write_text(_extract_cfgcheck()) + return subprocess.run( + [sys.executable, str(script), str(model)], + capture_output=True, text=True, + env={ + "PATH": "/usr/bin:/bin:/usr/local/bin", + "REHEARSAL_LESSONS": rehearsal, + "OUT": str(out), + "MODEL": "fixture_model", + "HOME": str(out), + }, + ) + + +def _config_lessons(model: Path) -> int: + """What get_hp_config() RETURNS — not what the file text says.""" + path = model / "configs" / "config_hyperparameters.py" + spec = importlib.util.spec_from_file_location(f"_probe_{id(path)}", path) + mod = importlib.util.module_from_spec(spec) + spec.loader.exec_module(mod) + return mod.get_hp_config()["total_lessons"] + + +PRODUCTION_CONFIG = "def get_hp_config():\n return {'total_lessons': 300}\n" +CHEAP_CONFIG = "def get_hp_config():\n return {'total_lessons': 40}\n" + + +class TestTheProductionFloorActuallyRefuses: + """H1, executed. The floor is the only thing standing between an accidental config and + a full GPU budget, and the first version of this file asserted its SOURCE LINE while a + mutation that changed `sys.exit(` to `print(` sailed through: the run announced + "expected >= 300" and trained anyway.""" + + def test_a_cheap_config_is_refused_with_a_nonzero_exit(self, tmp_path): + model = _make_model(tmp_path, CHEAP_CONFIG) + r = _run_cfgcheck(model, tmp_path / "out") + assert r.returncode != 0, ( + "a 40-lesson config must REFUSE in production mode. Exit code 0 here means the " + f"pod trains a throwaway model on paid hardware.\nstdout: {r.stdout}" + ) + assert "--rehearsal" in (r.stdout + r.stderr), ( + "the refusal must name the supported way to get a cheap run, or the operator " + "edits the guard out — which is the state this audit found" ) - def test_the_production_floor_is_still_enforced(self): - src = PODRUN.read_text() - assert re.search(r"if not requested and lessons < 300:", src), ( - "the >=300 floor must still apply when --rehearsal is NOT given. Adding the " - "escape hatch must not widen the hole it was protecting." + def test_a_production_config_passes(self, tmp_path): + model = _make_model(tmp_path, PRODUCTION_CONFIG) + r = _run_cfgcheck(model, tmp_path / "out") + assert r.returncode == 0, f"a 300-lesson production run must proceed.\n{r.stdout}\n{r.stderr}" + + def test_production_mode_does_not_patch_the_config(self, tmp_path): + """The one-word mutation `or ""` -> `or "40"` made every production run patch itself + to 40 lessons while the MANIFEST still said `mode: production`. Nothing saw it.""" + model = _make_model(tmp_path, PRODUCTION_CONFIG) + _run_cfgcheck(model, tmp_path / "out") + assert _config_lessons(model) == 300, ( + "a production run must not modify the config it was given. If this fails, a run " + "labelled 'production' in the MANIFEST trained on a patched lesson count." ) + assert (tmp_path / "out" / ".lessons").read_text().strip() == "300" - @pytest.mark.parametrize("model", HYDRANETS) - def test_tracked_configs_still_declare_the_production_count(self, model): - """A rehearsal patches the pod's clone. The committed config stays production truth.""" - cfg = REPO / "models" / model / "configs" / "config_hyperparameters.py" - hit = re.search(r"'total_lessons':\s*(\d+)", cfg.read_text()) - assert hit is not None, f"{model}: no total_lessons in {cfg}" - assert int(hit.group(1)) >= 300, ( - f"{model} declares total_lessons={hit.group(1)}. A rehearsal must never be " - "obtained by committing a low count — that is the state this audit found and " - "the reason --rehearsal patches only the pod's ephemeral checkout." - ) - - def test_a_leftover_rehearsal_patch_is_named_not_blamed_on_the_config(self): - """Found reviewing this change, not in the original audit. - - Section 2 does not re-clone when `.git` already exists, so a production run started - on a pod that has already rehearsed reads the LEFTOVER patch. The floor still refuses - it — but it would blame the committed config and advise `--rehearsal`, which is the - exact opposite of what the operator wants. A correct refusal for the wrong reason - sends someone to edit the wrong file. - """ - src = PODRUN.read_text() - assert "is MODIFIED in this checkout" in src, ( - "a production run must detect a leftover rehearsal patch and name it, rather " - "than reporting the patched value as though it were the committed config" + +class TestRehearsalPatchesOnlyThePodAndProvesIt: + """H1 / #523, executed.""" + + def test_rehearsal_patches_the_config_and_records_the_real_value(self, tmp_path): + model = _make_model(tmp_path, PRODUCTION_CONFIG) + r = _run_cfgcheck(model, tmp_path / "out", rehearsal="40") + assert r.returncode == 0, f"--rehearsal 40 must proceed.\n{r.stdout}\n{r.stderr}" + assert _config_lessons(model) == 40, "the pod's config must actually be patched" + assert (tmp_path / "out" / ".lessons").read_text().strip() == "40", ( + "the MANIFEST reads .lessons; it must carry the value the run was gated on" ) - def test_the_patch_is_verified_by_re_reading_the_config(self): - """A substitution that silently missed would produce a 300-lesson run wearing a - rehearsal label, or a rehearsal wearing none. Both are worse than a refusal.""" - src = PODRUN.read_text() - assert "importlib.invalidate_caches()" in src and "still reports total_lessons" in src, ( - "the config patch must be confirmed by re-importing the config, not trusted " - "from re.subn's return value alone" + def test_a_patch_that_does_not_take_is_refused_not_believed(self, tmp_path): + """The adversarial case the 'verification' exists for. + + Here the literal is patchable but `get_hp_config()` returns 300 regardless — a + second assignment wins. A verification that trusted re.subn's return value, or that + compared `target` against itself, would report a successful 40-lesson rehearsal + while the model trains 300 lessons. The mutation that did exactly that (replacing + the re-import with `lessons = target`) survived the first version of this file. + """ + sneaky = ( + "def get_hp_config():\n" + " d = {'total_lessons': 300}\n" + " d['total_lessons'] = 300\n" + " return d\n" + ) + model = _make_model(tmp_path, sneaky) + r = _run_cfgcheck(model, tmp_path / "out", rehearsal="40") + assert r.returncode != 0, ( + "the patch did not take — get_hp_config() still returns 300 — and the run " + f"proceeded anyway. It must refuse.\nstdout: {r.stdout}" ) + assert "still reports total_lessons" in (r.stdout + r.stderr) + + def test_a_leftover_patch_stops_a_later_production_run(self, tmp_path): + """Section 2 does not re-clone when .git exists, so a production run on a pod that + has rehearsed reads the LEFTOVER patch. It must name that, not blame the committed + config and send the operator to edit the wrong file.""" + model = _make_model(tmp_path, PRODUCTION_CONFIG) + assert _run_cfgcheck(model, tmp_path / "o1", rehearsal="40").returncode == 0 + r = _run_cfgcheck(model, tmp_path / "o2") + assert r.returncode != 0, "a production run must refuse a dirty config" + assert "MODIFIED in this checkout" in (r.stdout + r.stderr), ( + "the refusal must name the leftover rehearsal patch. Blaming the committed " + "config is a correct refusal for the wrong reason." + ) + + def test_the_region_gate_still_fires(self, tmp_path): + """Not part of this audit's findings — pinned because the rehearsal work edited the + block this lives in, and it guards 'this would not be a global-land run'.""" + model = _make_model(tmp_path, PRODUCTION_CONFIG, region='"africa"') + r = _run_cfgcheck(model, tmp_path / "out") + assert r.returncode != 0 and "REGION" in (r.stdout + r.stderr) + +class TestTrackedConfigsDeclareTheProductionCount: + """A3. Calls get_hp_config() rather than matching the literal. + + pod_run_model.sh lines 173-176 refuse to pattern-match a config, citing #501 "the guard + that was not one — a substring assertion satisfied by a COMMENT recording the value's + history". The first version of this guard pattern-matched the config. A mutation that + appended `d['total_lessons'] = 40` before the return left the 300 literal untouched and + the guard green, while the pod would read 40. + """ + + @pytest.mark.parametrize("model", HYDRANETS) + def test_the_value_the_pod_will_read_is_a_production_count(self, model): + cfg_dir = REPO / "models" / model / "configs" + assert (cfg_dir / "config_hyperparameters.py").exists(), f"{model}: no config" + lessons = _config_lessons(cfg_dir.parent) + assert lessons >= 300, ( + f"{model}: get_hp_config() returns total_lessons={lessons}. A rehearsal must be " + "obtained with --rehearsal, which patches only the pod's clone — never by " + "committing a low count." + ) -class TestFreshEnvironmentReproducesTheOneThatWorked: - """H2 (views-models#516, reframed). Measured, not reasoned about. - declared: views-datafactory>=1.9.0,<2.0.0 + numpy>=1.26.4,<2.0.0 - resolved: numpy 1.26.4, xarray 2025.12.0, pandas 3.0.6 - the pod that delivered: xarray 2024.3.0, pandas 1.5.3 +class TestTheEnvironmentPinsAreCORRECTNotMerelyPRESENT: + """H2 (#516). Asserts the measurement, not a proxy for it. - views-datafactory requires xarray outright and pandas only in an optional extra that is - not installed, so pandas arrives THROUGH xarray — xarray is the sole carrier, as - datafactory's own pyproject comment says. Pinning the carrier fixes it. + The first version checked that a pin EXISTED. `xarray>=2025.12,<2026` and + `pandas>=3.0,<4.0` — the exact versions measured as the defect — passed it, and its own + failure message ("this file resolves to pandas 3.0.6") was unreachable in the state that + resolves pandas 3.0.6. - The bound is measured: xarray 2024.3.0 is the LAST release accepting pandas 1.x; 2024.5.0 - moved to pandas>=2.0 and 2024.9.0 to pandas>=2.1. A tidy `<2025` cap would have been - WRONG — the cliff is inside the 2024 line, not at the year boundary. + Measured 2026-09-29: xarray 2024.3.0 is the LAST release accepting pandas 1.x; 2024.5.0 + moved to pandas>=2.0 and 2024.9.0 to pandas>=2.1. So a `<2025` cap would look right and + be wrong — the cliff is inside the 2024 line. """ + # (package, must be allowed, must be refused, why the refused one matters) + CASES = ( + ("pandas", "1.5.3", "3.0.6", "the version a fresh resolve silently picked"), + ("pandas", "1.5.3", "2.1.0", "any pandas 2.x is a different major from the platform's"), + ("xarray", "2024.3.0", "2025.12.0", "the version a fresh resolve silently picked"), + ("xarray", "2024.3.0", "2024.11.0", "already requires pandas>=2.1 — inside the 2024 line"), + ("numpy", "1.26.4", "2.0.0", "the pandas wheel is built against the numpy 1.x C ABI"), + ) + + @staticmethod + def _spec(postprocessor: str, package: str) -> SpecifierSet: + text = (REPO / "postprocessors" / postprocessor / "requirements.txt").read_text() + found = [ + ln.strip() for ln in text.splitlines() + if re.match(rf"^\s*{package}\s*[><=!~]", ln) and ";" not in ln + ] + assert found, ( + f"{postprocessor}/requirements.txt declares no unconditional pin for {package}. " + "Unpinned, this file resolves pandas to 3.0.6 with no error. See #516. " + "(A marker-gated pin is excluded deliberately: `; python_version < \"3.10\"` is " + "inert on the 3.11 the runner builds, and looked identical to a real pin.)" + ) + assert len(found) == 1, f"{postprocessor}: {package} pinned {len(found)} times: {found}" + return SpecifierSet(found[0][len(package):].strip()) + @pytest.mark.parametrize("postprocessor", ["un_fao", "un_crafd"]) - def test_the_carrier_and_the_passenger_are_both_pinned(self, postprocessor): - req = REPO / "postprocessors" / postprocessor / "requirements.txt" - text = req.read_text() - missing = [p for p in ("xarray", "pandas") if not re.search(rf"^{p}[><=~]", text, re.M)] - assert not missing, ( - f"{req.relative_to(REPO)} does not pin {missing}. Unpinned, this file resolves " - "to pandas 3.0.6 — a different MAJOR from every run that has ever succeeded, " - "with no error. See views-models#516." - ) - - def test_both_postprocessors_declare_identical_pins(self): - """C-116: one shared prefix. Pinning one lets whichever runs last decide for both.""" - pins = {} - for name in ("un_fao", "un_crafd"): - text = (REPO / "postprocessors" / name / "requirements.txt").read_text() - pins[name] = sorted( - re.findall(r"^((?:numpy|pandas|xarray)[><=~][^\s#]*)", text, re.M) - ) - assert pins["un_fao"] == pins["un_crafd"], ( - f"un_fao pins {pins['un_fao']} but un_crafd pins {pins['un_crafd']}. Both " - "install into envs/views-postprocessing (C-116); divergent pins mean the last " - "postprocessor to run silently decides the versions for the other." - ) - - -class TestThePodCanActuallyPublish: - """H3 (views-models#517). The publish path has to be installed and PROVEN at preflight. - - Verified by a real resolve on 2026-09-29: requesting the extra yields appwrite 13.6.1 - alongside views-hydranet 0.1.2 and views-pipeline-core 3.3.4 — both of which carry the - fixes shipped that day — with pandas 1.5.3 and numpy 1.26.4 intact. + @pytest.mark.parametrize("package,allowed,refused,why", CASES) + def test_the_pin_admits_the_working_version_and_refuses_the_broken_one( + self, postprocessor, package, allowed, refused, why + ): + spec = self._spec(postprocessor, package) + assert spec.contains(allowed), ( + f"{postprocessor}: {package}{spec} excludes {allowed}, which is what every " + "successful run has used. This env would not build." + ) + assert not spec.contains(refused), ( + f"{postprocessor}: {package}{spec} ADMITS {refused} — {why}. The pin exists but " + "does not constrain what it was added to constrain." + ) + + @pytest.mark.parametrize("package", ["numpy", "pandas", "xarray"]) + def test_both_postprocessors_constrain_each_package_identically(self, package): + """C-116: one shared prefix, so the last postprocessor to run decides for both. + + Compares SPECIFIER SEMANTICS, not captured strings. The old string compare was + defeated by a marker suffix that made un_fao's pins inert while keeping the captured + text identical to un_crafd's real ones. + """ + fao, crafd = self._spec("un_fao", package), self._spec("un_crafd", package) + probes = ["1.5.3", "2.0.0", "2.1.0", "3.0.6", "1.26.4", "2024.3.0", "2024.11.0", "2025.12.0"] + differ = [v for v in probes if fao.contains(v) != crafd.contains(v)] + assert not differ, ( + f"un_fao pins {package}{fao} and un_crafd pins {package}{crafd}; they disagree " + f"on {differ}. Both install into envs/views-postprocessing (C-116), so whichever " + "runs last silently decides these versions for the other." + ) + + +class TestThePublishPathIsInstalledAndProven: + """H3 (#517). Asserted against comment-stripped source. + + Both of the first version's assertions here were satisfied by a five-line prose comment + that named `views-pipeline-core[appwrite]`: deleting the extra from the install line and + commenting out both imports left the suite green, with preflight printing "appwrite + client importable — the publish path exists" having imported nothing. + """ + + def test_the_appwrite_extra_is_in_an_install_command(self): + code = _code_only(PODRUN.read_text()) + install_lines = [ + ln for ln in code.splitlines() + if "pip install" in ln or (ln.strip().startswith('"') and "appwrite" in ln) + ] + assert any("views-pipeline-core[appwrite]" in ln for ln in install_lines), ( + "no install command requests the appwrite extra. Without it _build_datastore " + "raises at publish, AFTER the full training run — the 2026-09-29 failure (#517). " + f"install lines seen: {install_lines}" + ) + + def test_preflight_imports_the_client_not_just_mentions_it(self): + code = _code_only(PODRUN.read_text()) + verify = code.split("stage verify_env", 1)[-1].split("stage ", 1)[0] + assert re.search(r"^\s*import appwrite", verify, re.M) or re.search( + r"^\s*from views_pipeline_core\.modules\.appwrite import", verify, re.M + ), ( + "verify_env must IMPORT the Appwrite client, not merely mention it. The whole " + "script is written to fail early, and the one dependency that failed late was " + "the one not checked here." + ) + + def test_the_toolz_override_is_an_install_and_is_last(self): + code = _code_only(PODRUN.read_text()) + installs = [m.start() for m in re.finditer(r"pip install", code)] + overrides = [ + m.start() for m in re.finditer(r"pip install[^\n]*toolz>=0\.12", code) + ] + assert overrides, ( + "the C-151 toolz override is not an install command any more. A comment saying " + "the base image ships toolz>=0.12.1 satisfied the old guard while C-151 was " + "reintroduced; the resolver really does land on toolz 0.11.2 for this set." + ) + assert max(overrides) == max(installs), ( + "the toolz override must be the LAST pip install: anything after it re-resolves " + "the prefix and can pull toolz back under 0.12, which happened by hand on " + "2026-09-29 (1.1.0 -> 0.11.2) with no error." + ) + + +class TestARehearsalIsMarkedInTheOutput: + """The last line of defence, since nothing REFUSES to publish a rehearsal. + + Asserted against comment-stripped source. The old guards passed with the marker-writing + block deleted outright — `"$OUT/REHEARSAL"` was supplied by the `rm -f` on line 83 and + "NOT FIT TO DELIVER" by a MANIFEST line. """ - def test_the_appwrite_extra_is_installed(self): - src = PODRUN.read_text() - assert "views-pipeline-core[appwrite]" in src, ( - "the appwrite extra is never installed, so the run trains for hours and then " - "cannot publish. This is the failure of 2026-09-29. See views-models#517." + def test_the_marker_is_written_not_only_removed(self): + code = _code_only(PODRUN.read_text()) + writes = re.findall(r'>\s*"\$OUT/REHEARSAL"', code) + assert writes, ( + "nothing WRITES $OUT/REHEARSAL. A guard satisfied by the `rm -f` that clears it " + "is green in the state where no rehearsal is ever marked." ) - def test_the_publish_path_is_checked_before_the_money_is_spent(self): - """An import in preflight costs seconds. Discovering it at publish costs the run.""" - src = PODRUN.read_text() - verify = src.split("stage verify_env", 1)[-1].split("stage", 1)[0] - assert "appwrite" in verify, ( - "pod_run_model.sh verifies torch, tlz, hydranet and datafactory at preflight but " - "not the Appwrite client. The whole script is written to fail early; the one " - "dependency that failed late was the one not checked here." + def test_the_marker_is_cleared_before_the_run_not_after(self): + code = _code_only(PODRUN.read_text()) + clear = code.index('rm -f "$OUT/REHEARSAL"') + assert clear < code.index("stage preflight"), ( + "the marker must be cleared with the other stale per-run state, before the run. " + "Moved to the end, a successful rehearsal deletes its own marker on the way out." ) - def test_the_toolz_override_is_reasserted_after_the_last_install(self): - """GREEN, and it survived as a prediction. Confirmed necessary, not folklore: the - resolver really does land on toolz 0.11.2 for this dependency set.""" - src = PODRUN.read_text() - installs = [m.start() for m in re.finditer(r"pip install", src)] - overrides = [m.start() for m in re.finditer(r"toolz>=0\.12", src)] - assert overrides, "the C-151 toolz override is gone entirely" - assert max(overrides) > max(installs), ( - "the C-151 toolz override must be the LAST install: anything after it " - "re-resolves the prefix and can pull toolz back under 0.12, which happened by " - "hand on 2026-09-29 (1.1.0 -> 0.11.2) with no error." + def test_the_manifest_reports_the_mode_on_the_correct_branch(self): + """Swapping the two MANIFEST branches made production runs claim PATCHED and + rehearsals claim as-committed. The old guard checked only that the string existed.""" + code = _code_only(PODRUN.read_text()) + m = re.search( + r'if \[ -n "\$REHEARSAL_LESSONS" \]; then(.*?)else(.*?)fi', + code, re.S, ) + assert m, "the MANIFEST mode branch is not in the expected shape; re-read it" + rehearsal_branch, production_branch = m.group(1), m.group(2) + assert "NOT FIT TO DELIVER" in rehearsal_branch and "PATCHED after checkout" in rehearsal_branch + assert "NOT FIT TO DELIVER" not in production_branch, ( + "the production branch of the MANIFEST claims the output is unfit to deliver" + ) + assert "PATCHED" not in production_branch, ( + "the production branch claims the config was patched after checkout" + ) + +class TestTheGuideDocumentsTheChmodTrap: + """C-154 / #518. This property DID NOT EXIST when it was first asserted. -class TestARehearsalCannotBeMistakenForADelivery: - """The half that actually prevents harm. + The original assertion claimed the guide records the /workspace chmod trap and passed + because `/workspace` appears at line 104 and `chmod` at line 180, joined by `.*` under + re.S. An independent audit read the guide and found nothing: no "world-readable", no + "network filesystem", no C-154, no #518. The guard certified a safety property that was + absent, which is worse than having no guard, because it stopped the next reader looking. - A rehearsal's parquets are structurally identical to a production run's — same columns, - row counts, names, all finite and non-negative. Every structural check passes. Nothing - in the DATA says the model behind it is undertrained. + The warning has since been written. This asserts its substance on a bounded window, not + two words anywhere in a 400-line file. """ - def test_the_run_marks_its_output(self): - src = PODRUN.read_text() - assert '"$OUT/REHEARSAL"' in src and "NOT FIT TO DELIVER" in src, ( - "a rehearsal must leave an unmistakable marker beside its parquets" - ) - - def test_a_stale_marker_cannot_be_inherited(self): - """Worse than a stale marker on a real run: a rehearsal inheriting a production - run's ABSENCE of one.""" - src = PODRUN.read_text() - assert 'rm -f "$OUT/REHEARSAL"' in src, ( - "the marker must be cleared with the other stale per-run state, or a rehearsal " - "in a directory left by a production run carries no mark at all" - ) - - def test_the_manifest_admits_the_config_was_patched(self): - src = PODRUN.read_text() - assert "PATCHED after checkout" in src, ( - "on a rehearsal the MANIFEST's `git:` sha no longer describes the run — the " - "config was patched after checkout. A manifest that must be cross-read with a " - "command-line flag to be understood will be misread." - ) - - # NOT TESTED HERE, DELIBERATELY. pod_run_model.sh MARKS a rehearsal; it cannot REFUSE - # to publish one, because the publish step is not in this script — the header is accurate - # that this runner uploads nothing. The refusal belongs wherever a forecast is handed to - # a store, which is views-pipeline-core, so no assertion in THIS repo can observe it. - # - # An xfail(strict) stub was written here and deleted: its regex spanned the whole file - # under re.S and XPASSed by accident, i.e. it was the exact class of guard-that-cannot- - # fire this audit exists to find. A test that cannot observe its subject is worse than no - # test, because it reports coverage. Tracked as an issue instead; see the module - # docstring. Until that guard exists, $OUT/REHEARSAL is the only thing between an - # undertrained model and a partner, and that is weaker than it should be. - - -class TestPublishCredentialsHaveADurableHome: - """GREEN (survived), and the prediction was wrong in the useful direction: the guide - DOES record the /workspace chmod trap, so the operator-facing half of C-154 / #518 is - already durable. What remains is that placement is still a manual step with a correct - runbook — an operator task, not a defect, and not counted as a falsification.""" - - def test_the_guide_warns_that_workspace_ignores_chmod(self): - guide = REPO / "docs" / "runpod_run_guide.md" - assert re.search(r"/workspace.*chmod|chmod.*/workspace", guide.read_text(), re.I | re.S), ( - "/workspace is a network filesystem that reports chmod success and leaves the " - "file world-readable. C-154 / views-models#518." + def test_the_trap_is_documented_in_substance(self): + text = (REPO / "docs" / "runpod_run_guide.md").read_text() + assert re.search(r"C-154|#518", text), "the guide cites neither C-154 nor #518" + windows = [ + text[m.start(): m.start() + 700] + for m in re.finditer(r"chmod", text) + ] + assert any( + re.search(r"/workspace", w) + and re.search(r"ignore|silent|no effect|world-readable|666", w, re.I) + for w in windows + ), ( + "the guide must state, near a chmod instruction, that /workspace SILENTLY " + "ignores chmod and leaves the credential world-readable. Two words 76 lines " + "apart is what the deleted version of this guard accepted." ) From f2a657d00e52d027be8c102523bc7f837200d056 Mon Sep 17 00:00:00 2001 From: Polichinl Date: Tue, 29 Sep 2026 23:32:01 +0200 Subject: [PATCH 04/11] fix(#509): floor views-datafactory at 1.13.0, and guard the shell arg parsing MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two gaps found by replaying the guard-mode audit's mutations against the rewritten guards — the replay is the point, since the previous version passed 22/22 with every fix reverted. #509 — the pod installed views-datafactory>=1.9.0. The credential-handling fixes landed in 1.13.0: before it the client could carry a netrc credential across a redirect to another host and embed it in error messages. This script only ever runs on hardware we do not own, carrying exactly that credential, and the guide already prescribes >=1.13.0 explicitly while the script did not. A resolver picks 1.13.0 anyway today, but "the resolver will probably do the right thing" is precisely the reasoning that put xarray 2025.12.0 and pandas 3.0.6 into a fresh environment (#516). Verified: the floor still resolves to 1.13.0 with hydranet 0.1.2, pipeline-core 3.3.4, appwrite 13.6.1, pandas 1.5.3. The shell argument parser was unguarded. Executing the config-check program does not cover it — the flag never reaches Python if the shell refuses it first. Deleting the `--rehearsal)` case arm removes H1's escape hatch entirely and left every guard green, because the string survives in the header comment and in USAGE. Now executed against the real script: the flag and its count must be CONSUMED (distinguished from "unknown option" by which error appears), a missing count must be refused, a non-integer must be refused, and — the control, without which the first assertion proves nothing — a genuinely unknown option must still be rejected. Argument parsing precedes `mkdir -p "$OUT"`, so these cases exit before touching the filesystem. MUTATION REPLAY — every surviving mutation from the audit is now caught: X7 `or ""` -> `or "40"` (production runs patch themselves) 3 failed X8 --rehearsal becomes a bare flag 3 failed X22 the --rehearsal case arm deleted 3 failed X1 extra dropped from the install line, comment kept 1 failed X2 both appwrite imports commented out, prose comment kept 1 failed X5 2026-09-29 install order restored + reassuring comment 1 failed X6 the floor prints instead of exiting 1 failed X9 leftover-patch detector flipped to the wrong branch 1 failed X10 patch verification becomes `if target != target` 1 failed X11 the marker-writing block deleted 1 failed X12 stale-marker clear moved to the end of the run 1 failed X13 MANIFEST mode branches swapped 1 failed X14 both files pinned to xarray 2025.12 / pandas 3.0 8 failed X15 upper bounds dropped 8 failed X17 pins marker-gated inert on 3.11 6 failed X18 get_hp_config() returns 40, the 300 literal untouched 1 failed X20b toolz override replaced by a comment 1 failed control: the guide's chmod warning deleted 1 failed X16 (`xarray == 2024.3.0`, a stricter and CORRECT pin) previously failed with a message claiming no pin existed; it now passes, so the guard no longer raises a false alarm against a legal PEP 508 spelling. Two honest notes. X18 was recorded SURVIVED in the audit but had not applied — the config returns a dict literal, so the mutation's sed matched nothing. Re-applied correctly, it is caught. And X19 (replacing the credential chmod with an unrelated one) now passes, because the /workspace warning it was meant to defeat genuinely exists; the control above proves the guard fires when that warning is removed. 40 tests, up from 35. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01GG2cY2HaqmUMpwUyR5V82K --- ...t_falsification_40_lesson_run_readiness.py | 68 +++++++++++++++++++ 1 file changed, 68 insertions(+) diff --git a/tests/test_falsification_40_lesson_run_readiness.py b/tests/test_falsification_40_lesson_run_readiness.py index b45abafe..313f1b69 100644 --- a/tests/test_falsification_40_lesson_run_readiness.py +++ b/tests/test_falsification_40_lesson_run_readiness.py @@ -235,6 +235,58 @@ def test_the_region_gate_still_fires(self, tmp_path): assert r.returncode != 0 and "REGION" in (r.stdout + r.stderr) +class TestTheRehearsalFlagIsActuallyParsed: + """The shell argument parser, executed. + + An independent audit deleted the `--rehearsal)` case arm outright — so `--rehearsal` + fell through to `--*` and was rejected as an unknown option, removing H1's escape hatch + entirely — and every guard stayed green, because the string `--rehearsal` survived in the + header comment and in USAGE. Executing the config-check program does not cover this: the + flag never reaches Python if the shell refuses it first. + + These run the real script. Argument parsing precedes `mkdir -p "$OUT"`, so each case + below exits before the script touches the filesystem or spends anything. + """ + + @staticmethod + def _run(*args): + return subprocess.run( + ["bash", str(PODRUN), *args], + capture_output=True, text=True, cwd=str(REPO), timeout=60, + ) + + def test_the_flag_and_its_count_are_consumed_not_rejected(self): + """With the count consumed, the script must get as far as demanding a model name.""" + r = self._run("--rehearsal", "40") + combined = r.stdout + r.stderr + assert "unknown option" not in combined, ( + "--rehearsal was rejected as an unknown option, so the case arm parsing it is " + f"gone and a cheap run is unreachable again.\n{combined}" + ) + assert "usage" in combined.lower(), ( + f"expected the missing-model-name usage error once the flag was consumed.\n{combined}" + ) + + def test_a_count_is_required(self): + r = self._run("--rehearsal") + assert r.returncode != 0 + assert "lesson count" in (r.stdout + r.stderr), ( + "--rehearsal with no count must say so. Silently defaulting is how a rehearsal " + "becomes indistinguishable from a production run." + ) + + def test_a_non_integer_count_is_refused(self): + r = self._run("--rehearsal", "forty", "purple_alien") + assert r.returncode != 0 + assert "positive integer" in (r.stdout + r.stderr) + + def test_a_genuinely_unknown_option_is_still_refused(self): + """The control: the `--*` arm must keep working, or the test above proves nothing.""" + r = self._run("--bogus", "purple_alien") + assert r.returncode != 0 + assert "unknown option" in (r.stdout + r.stderr) + + class TestTrackedConfigsDeclareTheProductionCount: """A3. Calls get_hp_config() rather than matching the literal. @@ -349,6 +401,22 @@ def test_the_appwrite_extra_is_in_an_install_command(self): f"install lines seen: {install_lines}" ) + def test_datafactory_is_floored_where_the_credential_fixes_landed(self): + """#509. The runner only ever executes on hardware we do not own, carrying a netrc + credential. Before views-datafactory 1.13.0 the client could carry that credential + across a redirect to another host and embed it in error messages. The model + requirements still say >=1.9.0 and a resolver will usually pick the newest — but + "the resolver will probably do the right thing" is the reasoning that put pandas + 3.0.6 into a fresh environment (#516).""" + code = _code_only(PODRUN.read_text()) + floors = re.findall(r"views-datafactory>=(\d+)\.(\d+)", code) + assert floors, "the pod install does not request views-datafactory at all" + for major, minor in floors: + assert (int(major), int(minor)) >= (1, 13), ( + f"pod_run_model.sh installs views-datafactory>={major}.{minor}; the " + "credential-handling fixes landed in 1.13.0 (#509)." + ) + def test_preflight_imports_the_client_not_just_mentions_it(self): code = _code_only(PODRUN.read_text()) verify = code.split("stage verify_env", 1)[-1].split("stage ", 1)[0] From 4c8112390c1812ab683a002505374a709e28d605 Mon Sep 17 00:00:00 2001 From: Polichinl Date: Tue, 29 Sep 2026 23:40:42 +0200 Subject: [PATCH 05/11] fix(#509): actually apply the 1.13.0 floor the previous commit only tested for MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The previous commit shipped the guard and not the fix. The sequence: edit the install line, write the test, verify the test fires by reverting the line with sed, then `git checkout -- tools/podrun/pod_run_model.sh` to undo the sed — which reverted to the last COMMIT, discarding the uncommitted 1.13.0 edit along with the sed. The commit then captured the test plus a 1.9.0 install line. Caught by that test on the next full-suite run, which is the whole argument for writing guards that execute: a text-matching guard for this would have been satisfied by the comment above the line, which names 1.13.0 and explains why. Also worth recording, because it made the failure hard to see for two runs: this suite runs pytest-randomly, so ordering varies between runs. Two earlier runs reported 8058 passed/209 skipped and 7758 passed/470 skipped/3 failed with an unchanged tree, and the three failures were ordering interference in pre-existing tests, not in anything here. With `-p no:randomly` the counts are stable at 8076 passed/209 skipped/5 xfailed and the only failure was this one. Verify against a fixed order before concluding a change is responsible for a red suite. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01GG2cY2HaqmUMpwUyR5V82K --- tools/podrun/pod_run_model.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tools/podrun/pod_run_model.sh b/tools/podrun/pod_run_model.sh index 4727bd32..27d4c73b 100755 --- a/tools/podrun/pod_run_model.sh +++ b/tools/podrun/pod_run_model.sh @@ -126,7 +126,7 @@ if [ ! -x "$VENV/bin/python" ]; then # attempt failed on 2026-09-29 (views-models#517); it was fixed by hand on a pod that no # longer exists, so the repository never learned it. uv pip install --python "$VENV/bin/python" \ - "views-hydranet~=0.1.1" "views-datafactory>=1.9.0,<2.0.0" \ + "views-hydranet~=0.1.1" "views-datafactory>=1.13.0,<2.0.0" \ "views-pipeline-core[appwrite]" || die "pip install failed" # register C-151: viewser pins toolz<0.12, which cannot import tlz submodules on # Python 3.11 and breaks EVERY datafactory fetch. Override after resolution. From 9ec04af03608832d6cf3143f8afceae495e26507 Mon Sep 17 00:00:00 2001 From: Polichinl Date: Tue, 29 Sep 2026 23:55:45 +0200 Subject: [PATCH 06/11] =?UTF-8?q?feat(#499):=20script=20the=20FAO=20delive?= =?UTF-8?q?ry=20chain=20=E2=80=94=20it=20existed=20only=20as=20commands=20?= =?UTF-8?q?typed=20on=20a=20pod?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The fourth instance of the pattern the /falsify audit found three of, and the one that mattered most. On 2026-09-29 the whole FAO delivery — eight forecasting runs, the rusty_bucket pool, the publish, the un_fao postprocessor — was typed by hand on a rented pod. It worked, the pod was destroyed, and nothing recorded what had been done. The audit missed it because it audited the script that exists (pod_run_model.sh) instead of asking what the delivery needs. pod_run_model.sh runs `-r calibration -t -e` and its header is accurate that it uploads nothing: it is Track A, the research frames. Nothing in tools/ drove Track B at all — `grep -rln "prediction_store\|r forecasting" tools/` returns one unrelated file. This is not a stopgap. fimbulthul, which #499 names for every Track B step, has been unreachable since 2026-09-25, so a pod is the only hardware there is. pod_run_model.sh gains `--forecast`: `-r forecasting -t -f`, and it SKIPS sections 4 and 5. Those are calibration deliverables — a forecast has one origin, so the 13-parquet count check would refuse it, and what the FAO chain consumes is the pooled ensemble output. Kept in this script rather than a copy because everything before section 3 is identical and already exercised; a second copy would be a second place for C-151 to rot. pod_run_fao_delivery.sh drives the chain. Two details that are not obvious and that cost real time to establish: - `-sa/--saved` on the ensemble is REQUIRED. Without it rusty_bucket refetches instead of pooling the member forecasts just written, and the eight runs are wasted. - the two legs need DIFFERENT INTERPRETERS. This pod builds a uv venv; tools/launcher/postprocessor.sh uses `conda shell.bash hook` / `conda create --prefix`. A pod can satisfy one and not the other, and step 4 is last — so a missing conda kills the delivery after every GPU hour is spent. Preflight now checks for it. `--preflight` is the mode that protects the money. At 300 lessons step 1 alone is ~16 GPU hours, and everything it checks is knowable in seconds: checkout, venv, netrc, the three publish secrets, conda, that un_fao's REGION resolves to land_gaul rather than a disarmed value (C-110), GPU, disk. It accumulates and reports every problem rather than dying on the first, because a round trip per problem is billed by the second. The publish secrets are checked HERE and not at first publish because views-pipeline-core's PredictionStoreConfig claims to read them "once at startup and fail loud … preventing silent failures after hours of training" and does not — it is called from _build_datastore, after training (views-pipeline-core#557). Until that moves, this preflight is the only check that happens before the money is spent. A rehearsal is marked at every level, and the marker says the uncomfortable part out loud: nothing downstream refuses a marked rehearsal (#523), so its undertrained forecasts DO reach the FAO shelf and ARE servable. That is deliberate — it is the only way to test the chain — and it is why the final stage reads back what landed with `tools.liveness`, by name, rather than inferring success from an exit code. On 2026-09-29 the publish reported success having written nothing (C-155). 19 guards, written after the lesson that 10 of 14 of the previous set were decorative. The parser and `--preflight` are EXECUTED; the remaining text assertions read comment-stripped source and cover only ordering facts that cannot be executed without a GPU, a store and a partner bucket. PODRUN_ROOT exists so preflight is exercisable off a pod; it relocates the workspace and relaxes no check. Every preflight check was proven able to fire, including by shadowing nvidia-smi with a failing stub and stripping conda from PATH — a check that cannot fail is the defect this repository keeps finding. Two defects in this script were found by its own tests: an unchecked `mkdir -p "$OUT"` that let it run on into a broken tee off a pod, and a lock message that reported "another delivery is in progress" for a directory that simply was not writable. What these guards do NOT cover, stated rather than implied: nothing here runs a forecast, publishes, or reaches the FAO. The first real exercise is a rehearsal on a pod. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01GG2cY2HaqmUMpwUyR5V82K --- tests/test_fao_delivery_runner.py | 226 ++++++++++++++++++++++++ tools/podrun/pod_run_fao_delivery.sh | 254 +++++++++++++++++++++++++++ tools/podrun/pod_run_model.sh | 75 +++++++- 3 files changed, 550 insertions(+), 5 deletions(-) create mode 100644 tests/test_fao_delivery_runner.py create mode 100755 tools/podrun/pod_run_fao_delivery.sh diff --git a/tests/test_fao_delivery_runner.py b/tests/test_fao_delivery_runner.py new file mode 100644 index 00000000..389da0ad --- /dev/null +++ b/tests/test_fao_delivery_runner.py @@ -0,0 +1,226 @@ +""" +Guards for `tools/podrun/pod_run_fao_delivery.sh` — the FAO delivery chain (views-models#499 +Track B), which until 2026-09-29 existed only as commands typed by hand on a rented pod. + +These are written after a /falsify guard-mode audit found that 10 of 14 guards for the +sibling script were DECORATIVE: they read the script as text, and the script's own comments +contained the strings they asserted, so deleting the code left the guard green. So here: + + * the argument parser is EXECUTED, not matched; + * `--preflight` is EXECUTED, which is the whole mode's value — it must refuse on a machine + that cannot deliver, and it must refuse for the right reason; + * text assertions read COMMENT-STRIPPED source, and are limited to ordering facts that + cannot be executed here (the steps themselves need a GPU, a store and a partner bucket). + +What these guards cannot cover, stated rather than implied: no test here runs a forecast, +publishes, or reaches the FAO. The first real exercise of this script is a rehearsal on a +pod, and that is what `--rehearsal` and `--preflight` exist to make cheap. +""" + +import os +import re +import subprocess +from pathlib import Path + +import pytest + +REPO = Path(__file__).resolve().parents[1] +FAO = REPO / "tools" / "podrun" / "pod_run_fao_delivery.sh" + +HYDRANETS = { + "purple_alien", "pink_pirate", "blue_stranger", "bright_starship", + "heavy_freighter", "blazing_meteor", "bold_comet", "violet_visitor", +} + + +def _code_only(src: str) -> str: + out = [] + for line in src.splitlines(): + stripped = re.sub(r"(?= 2, ( + f"preflight reported {out.count('MISSING:')} problems and stopped. It should " + "accumulate and report them together.\n" + out + ) + + def test_preflight_only_never_reaches_a_run_stage(self, result): + out = result.stdout + result.stderr + for forbidden in ("pool_and_publish", "un_fao_postprocessor", "forecast:"): + assert forbidden not in out, f"--preflight reached stage {forbidden}" + + +class TestTheChainIsInTheRightOrderWithTheRightFlags: + """Ordering facts, asserted on comment-stripped source because executing them needs a + GPU, a prediction store and a partner bucket.""" + + @pytest.fixture(scope="class") + def code(self): + return _code_only(FAO.read_text()) + + def test_the_ensemble_pools_from_saved_member_forecasts(self, code): + """Without -sa/--saved the ensemble refetches instead of pooling what the eight runs + just produced, and those GPU hours are wasted.""" + m = re.search(r"main\.py -r forecasting[^\n|]*", code) + assert m, "the ensemble invocation is not in the expected shape" + assert " -sa" in m.group(0), f"--saved is missing from: {m.group(0)}" + assert " -f" in m.group(0), f"--forecast is missing from: {m.group(0)}" + assert " -p" in m.group(0), f"--prediction_store is missing from: {m.group(0)}" + + def test_the_models_run_before_the_pool_which_runs_before_the_postprocessor(self, code): + forecast = code.index("--forecast") + pool = code.index("main.py -r forecasting") + postproc = code.index("postprocessors/un_fao/run.sh") + assert forecast < pool < postproc, ( + "the chain is out of order: the eight forecasts must precede the pool, and the " + "pool must precede the postprocessor that curates land -> land_gaul" + ) + + def test_the_whole_roster_is_named(self, code): + declared = set(re.search(r'MODELS="([^"]+)"', code).group(1).split()) + assert declared == HYDRANETS, ( + f"the roster here is {sorted(declared)}; rusty_bucket's members are " + f"{sorted(HYDRANETS)}. Pooling a partial roster changes the forecast silently." + ) + + def test_a_model_that_did_not_report_OK_stops_the_delivery(self, code): + assert 'STATUS" 2>/dev/null)" = "OK"' in code, ( + "the orchestrator must check each model's STATUS before pooling. An exit code " + "alone does not distinguish 'trained' from 'wrote a partial forecast'." + ) + + def test_wandb_is_forced_offline_on_the_ensemble_leg(self, code): + """Without this, main.py calls wandb.login() and blocks on a prompt no one is + watching. The first forecasting run on a pod died exactly there.""" + tail = code[code.index("pool_and_publish"):] + assert "WANDB_MODE=offline" in tail + + +class TestARehearsalIsMarkedEverywhereItCanBe: + """A rehearsal reaches the FAO shelf, because nothing downstream refuses one (#523). So + the marking is the only protection, and it has to be loud.""" + + @pytest.fixture(scope="class") + def code(self): + return _code_only(FAO.read_text()) + + def test_the_rehearsal_flag_reaches_the_per_model_runs(self, code): + assert "REH_ARGS" in code and "--rehearsal $REHEARSAL_LESSONS" in code, ( + "the lesson count must be forwarded to each model run, or the models train in " + "full while the delivery is labelled a rehearsal" + ) + + def test_the_marker_admits_the_forecasts_reached_the_shelf(self, code): + """The dangerous half. A reader who sees 'REHEARSAL' may assume it was contained; + it was not, and the file has to say so.""" + assert "ON THE FAO SHELF" in code, ( + "the rehearsal marker must state that the undertrained forecasts ARE published " + "and servable, not merely that the run was a rehearsal" + ) + + def test_the_marker_is_cleared_at_the_start_of_every_run(self, code): + clear = code.index('rm -f "$OUT/STATUS"') + assert "REHEARSAL" in code[clear:clear + 120], ( + "a production delivery must not inherit a previous rehearsal's marker, and far " + "worse, a rehearsal must not inherit a production run's absence of one" + ) + + def test_what_landed_is_read_back_by_name(self, code): + """`tools.liveness` answers 'is it there?' rather than 'did we send it?'. On + 2026-09-29 the publish reported success having written nothing (C-155).""" + assert "tools.liveness" in code diff --git a/tools/podrun/pod_run_fao_delivery.sh b/tools/podrun/pod_run_fao_delivery.sh new file mode 100755 index 00000000..54467cb0 --- /dev/null +++ b/tools/podrun/pod_run_fao_delivery.sh @@ -0,0 +1,254 @@ +#!/usr/bin/env bash +# pod_run_fao_delivery.sh [--rehearsal ] [--preflight] +# +# ┌──────────────────────────────────────────────────────────────────────────────────────┐ +# │ VERSION 0.1.0 — PROVISIONAL. First script for the FAO leg; no CI runs it. │ +# │ It PUBLISHES to a store a partner consumes. Read --preflight output before trusting │ +# │ it with GPU hours, and read the REHEARSAL note below before trusting it with FAO. │ +# └──────────────────────────────────────────────────────────────────────────────────────┘ +# +# Track B of views-models#499: the eight HydraNets forecast on global land, `rusty_bucket` +# pools them, the pooled forecast is published to the shelf, and the `un_fao` postprocessor +# curates `land` -> `land_gaul` and hands over to `unfao_bucket`. +# +# WHY THIS EXISTS. On 2026-09-29 every one of these steps was typed by hand on a rented pod. +# It worked, the pod was destroyed, and nothing recorded what had been done — so the next +# delivery started from the original broken state. That is the same defect a /falsify audit +# found three instances of in `pod_run_model.sh` (#516, #517, #523), and this is the fourth: +# the procedure existed only as commands in a terminal. `fimbulthul` has been unreachable +# since 2026-09-25, so a pod is not a stopgap here — it is the only hardware. +# +# THE ORDER MATTERS AND IS NOT OBVIOUS: +# 1. the 8 models -r forecasting -t -f (each ~2h at 300 lessons) +# 2. rusty_bucket -r forecasting -f -sa pooled from the SAVED member forecasts +# 3. publish -p wire shards to production_forecasts +# 4. un_fao postprocessors/un_fao/run.sh +# `-sa/--saved` at step 2 is required: without it the ensemble refetches instead of pooling +# what step 1 just produced. +# +# REHEARSAL. `--rehearsal ` patches only the pod's clone (never a tracked config), +# and every artefact it produces is MARKED unfit to deliver. But marking is all this script +# can do: nothing downstream REFUSES a marked rehearsal, because the refusal belongs in +# views-pipeline-core and is an ADR-013 contract change (views-models#523). A rehearsal +# therefore reaches the FAO shelf exactly like a real run. That is deliberate — it is the +# only way to test the chain — and it is why step 4 prints what it published by name. +# +# Progress: cat /workspace/deliver/_fao/STAGE · tail -f /workspace/deliver/_fao/run.log + +set -uo pipefail + +USAGE='usage: pod_run_fao_delivery.sh [--rehearsal ] [--preflight]' +REHEARSAL_LESSONS="" +PREFLIGHT_ONLY=0 +while [ $# -gt 0 ]; do + case "$1" in + --rehearsal) + [ $# -ge 2 ] || { echo "!!! --rehearsal needs a lesson count, e.g. --rehearsal 40" >&2 + echo "!!! $USAGE" >&2; exit 1; } + REHEARSAL_LESSONS="$2" + case "$REHEARSAL_LESSONS" in + ''|*[!0-9]*) echo "!!! --rehearsal needs a positive integer, got: $REHEARSAL_LESSONS" >&2 + exit 1 ;; + esac + [ "$REHEARSAL_LESSONS" -ge 1 ] || { echo "!!! --rehearsal 0 has nothing to rehearse" >&2; exit 1; } + shift 2 ;; + --preflight) PREFLIGHT_ONLY=1; shift ;; + --*) echo "!!! unknown option: $1" >&2; echo "!!! $USAGE" >&2; exit 1 ;; + *) echo "!!! unexpected argument: $1" >&2; echo "!!! $USAGE" >&2; exit 1 ;; + esac +done + +# PODRUN_ROOT exists so this script can be exercised off a pod — the tests run its parser and +# its preflight, and a hardcoded /workspace makes both untestable. It relocates the workspace +# and nothing else: every substantive check below (credentials, conda, region, GPU, disk) is +# unaffected by it, so an override cannot turn a refusal into a pass. +ROOT="${PODRUN_ROOT:-/workspace}" +REPO=$ROOT/views-models +VENV=$ROOT/venv +OUT=$ROOT/deliver/_fao +LOG=$OUT/run.log +ENSEMBLE=rusty_bucket +MODELS="violet_visitor bold_comet blazing_meteor blue_stranger heavy_freighter pink_pirate bright_starship purple_alien" + +# Checked, because an unchecked mkdir here fails off a pod and the script then runs on into +# a broken tee and a lock check that reports "another delivery is in progress" — a misleading +# message for a machine that simply has no /workspace. Found by the tests for this file. +mkdir -p "$OUT" 2>/dev/null || { + echo "!!! cannot create $OUT" >&2 + echo "!!! This script runs ON A POD, where $ROOT exists. To exercise it elsewhere, set" >&2 + echo "!!! PODRUN_ROOT=/some/writable/dir" >&2 + exit 1 +} +rm -f "$OUT/STATUS" "$OUT/REHEARSAL" "$OUT/PUBLISHED" +exec > >(tee -a "$LOG") 2>&1 + +stage() { echo "$1" > "$OUT/STAGE"; echo "=== [$(date +%H:%M:%S)] $1 ==="; } +die() { + echo "FAILED:$(cat "$OUT/STAGE" 2>/dev/null)" > "$OUT/STATUS" + printf '%s\n' "$1" > "$OUT/FAILURE" + echo "!!! $1" + exit 1 +} + +if ! mkdir "$OUT/.lock" 2>/dev/null; then + # Distinguished deliberately: "exists" and "could not be created" need different actions, + # and reporting the first for the second sends the operator looking for a run that never + # started. + if [ -d "$OUT/.lock" ]; then + echo "!!! $OUT/.lock exists — another FAO delivery is in progress on this pod." + echo "!!! If you are certain it is dead: rmdir $OUT/.lock" + else + echo "!!! could not create $OUT/.lock — $OUT is not writable." + fi + exit 1 +fi +trap 'rmdir "$OUT/.lock" 2>/dev/null' EXIT + +echo "### FAO delivery — started $(date -u +%Y-%m-%dT%H:%M:%SZ)" +[ -n "$REHEARSAL_LESSONS" ] && echo "### REHEARSAL at $REHEARSAL_LESSONS lessons — output will be marked unfit to deliver" + +# ── 0. preflight ────────────────────────────────────────────────────────────────────── +# Everything knowable before spending money, and the whole reason --preflight exists as a +# mode you can run on a fresh pod for seconds. At 300 lessons step 1 alone is ~16 GPU hours; +# discovering a missing credential or a missing conda after that is the failure this block is +# written to make impossible. Each check names what to do, not just what is wrong. +stage preflight +FAIL=0 +note() { echo " MISSING: $*"; FAIL=1; } + +[ -d "$REPO/.git" ] || note "$REPO is not a checkout — clone views-models first (see docs/runpod_run_guide.md Phase 2)" +[ -x "$VENV/bin/python" ] || note "$VENV does not exist — run pod_run_model.sh once to build it, or follow Phase 2" + +# The datafactory credential, for the fetch. +[ -s /root/.netrc ] || note "/root/.netrc — the datafactory fetch needs it (guide Step 2.3)" +if [ -s /root/.netrc ] && [ "$(stat -c %a /root/.netrc 2>/dev/null)" != "600" ]; then + chmod 600 /root/.netrc + echo " fixed: /root/.netrc was not 600" +fi + +# The publish credentials. Three of the nine are real secrets; the rest are identifiers. +# Checked HERE and not at first publish, because views-pipeline-core's own +# PredictionStoreConfig claims to read these "once at startup and fail loud" and does not — +# it is called from _build_datastore, i.e. after training (views-pipeline-core#557). Until +# that moves, this is the only check that happens before the money is spent. +for v in APPWRITE_ENDPOINT APPWRITE_DATASTORE_PROJECT_ID APPWRITE_DATASTORE_API_KEY; do + eval "val=\${$v:-}" + [ -n "$val" ] || note "\$$v — a publish secret; see reports/fao_delivery_runbook.md" +done +case "${APPWRITE_DATASTORE_API_KEY:-}" in + *[![:print:]]*) note "\$APPWRITE_DATASTORE_API_KEY contains a non-printable character — re-paste it" ;; +esac + +# conda, for the postprocessor leg ONLY. tools/launcher/postprocessor.sh uses +# `conda shell.bash hook` / `conda create --prefix` / `conda activate`, while this pod builds +# a uv venv — so the two legs need different interpreters and a pod can satisfy one and not +# the other. Without conda the delivery dies at step 4, after every GPU hour is spent. +command -v conda >/dev/null 2>&1 || note "conda — the un_fao postprocessor launcher requires it (tools/launcher/postprocessor.sh:72). A uv venv is not enough." + +# The coordinate registry, which the FAO queryset derives its region from. +if [ -x "$VENV/bin/python" ] && [ -d "$REPO" ]; then + REGION=$(cd "$REPO" && "$VENV/bin/python" -c \ + 'import sys; sys.path.insert(0,"."); from postprocessors.un_fao.configs.config_queryset import REGION; print(REGION)' 2>/dev/null) + if [ "$REGION" = "land_gaul" ]; then + echo " un_fao REGION resolves to land_gaul" + else + note "un_fao REGION resolved to '${REGION:-}', expected land_gaul — the FAO wire is disarmed (C-110)" + fi +fi + +nvidia-smi -L >/dev/null 2>&1 || note "no GPU visible" +AVAIL_GB=$(df -BG --output=avail "$ROOT" 2>/dev/null | tail -1 | tr -dc '0-9') +# Eight forecasting runs plus the pooled ensemble, on one disk. +[ "${AVAIL_GB:-0}" -ge 60 ] || note "only ${AVAIL_GB:-0}GB free on $ROOT; eight forecasts plus the pool need ~60GB" + +if [ "$FAIL" = "1" ]; then + echo + die "preflight failed — nothing above costs GPU time to fix. Fix them and re-run --preflight." +fi +echo "preflight OK — GPU visible, ${AVAIL_GB}GB free, credentials present, conda present, wire armed" + +if [ "$PREFLIGHT_ONLY" = "1" ]; then + echo OK > "$OUT/STATUS" + stage preflight_only_done + echo "### --preflight only: nothing was run, nothing was published." + echo "### Re-run without --preflight to start the delivery." + exit 0 +fi + +# ── 1. the eight, forecasting ───────────────────────────────────────────────────────── +REH_ARGS="" +[ -n "$REHEARSAL_LESSONS" ] && REH_ARGS="--rehearsal $REHEARSAL_LESSONS" +for M in $MODELS; do + stage "forecast:$M" + # Delegated rather than reimplemented: pod_run_model.sh already carries the config + # patch-and-verify, the appwrite check and the C-151 override, and a second copy of + # those is a second place for them to rot. + bash "$REPO/tools/podrun/pod_run_model.sh" $REH_ARGS --forecast "$M" \ + || die "forecast failed for $M — see /workspace/deliver/$M/FAILURE" + [ "$(cat "$ROOT/deliver/$M/STATUS" 2>/dev/null)" = "OK" ] \ + || die "$M did not report OK — refusing to pool a partial roster" +done + +# ── 2-3. pool and publish ───────────────────────────────────────────────────────────── +stage pool_and_publish +cd "$REPO/ensembles/$ENSEMBLE" || die "cannot enter $ENSEMBLE" +export WANDB_MODE=offline WANDB_SILENT=true +# -sa/--saved pools the member forecasts just written. Without it the ensemble refetches and +# the eight runs above are wasted. -p publishes the wire shards (ADR-013). +"$VENV/bin/python" main.py -r forecasting -f -sa -p || die "$ENSEMBLE pool-and-publish exited non-zero" + +# ── 4. the FAO postprocessor ────────────────────────────────────────────────────────── +stage un_fao_postprocessor +cd "$REPO" || die "cannot enter repo" +# A refusal naming DeliveryNotFindableError is views-postprocessing 1.4.0 WORKING: that build +# verifies a delivery by what it refuses, and the message says which case it hit. Do not read +# it as this script failing. +bash postprocessors/un_fao/run.sh || die "the un_fao postprocessor exited non-zero — read the message before assuming the worst; DeliveryNotFindableError means 1.4.0's findability guard fired and it names every object it checked" + +# ── 5. what actually landed ─────────────────────────────────────────────────────────── +stage report +# By name, from the live surface, not inferred from an exit code. `python -m tools.liveness` +# is the six-surface dashboard; it answers "is it there?" rather than "did we send it?". +"$VENV/bin/python" -m tools.liveness > "$OUT/PUBLISHED" 2>&1 \ + || echo "(liveness reported non-zero — read $OUT/PUBLISHED)" >> "$OUT/PUBLISHED" +tail -40 "$OUT/PUBLISHED" + +stage manifest +{ + echo "delivery: un_fao via $ENSEMBLE" + echo "finished: $(date -u +%Y-%m-%dT%H:%M:%SZ)" + echo "models: $MODELS" + echo "git: $(git -C "$REPO" rev-parse --short HEAD)" + if [ -n "$REHEARSAL_LESSONS" ]; then + echo "mode: REHEARSAL — NOT FIT TO DELIVER" + echo "lessons: $REHEARSAL_LESSONS (pod clones patched; tracked configs untouched)" + else + echo "mode: production" + echo "lessons: as committed at the git sha above" + fi +} > "$OUT/MANIFEST" +cat "$OUT/MANIFEST" + +if [ -n "$REHEARSAL_LESSONS" ]; then + { + echo "THIS DELIVERY IS A REHEARSAL. THE FORECASTS ON THE FAO SHELF ARE UNDERTRAINED." + echo + echo "lessons: $REHEARSAL_LESSONS" + echo "finished: $(date -u +%Y-%m-%dT%H:%M:%SZ)" + echo + echo "It ran the whole chain to prove the hops connect. Nothing downstream refuses a" + echo "marked rehearsal (views-models#523), so these forecasts ARE on the shelf and the" + echo "FAO API can serve them. They are structurally indistinguishable from real ones." + echo "Supersede or remove them before anyone reads them as a forecast." + } > "$OUT/REHEARSAL" + echo + echo "############################################################################" + echo "### REHEARSAL COMPLETE — undertrained forecasts are NOW ON THE FAO SHELF." + echo "### They are indistinguishable from real ones. Supersede them." + echo "### $OUT/REHEARSAL" + echo "############################################################################" +fi + +echo OK > "$OUT/STATUS" +stage done +echo "### FAO delivery — COMPLETE" diff --git a/tools/podrun/pod_run_model.sh b/tools/podrun/pod_run_model.sh index 27d4c73b..a4c10d8f 100755 --- a/tools/podrun/pod_run_model.sh +++ b/tools/podrun/pod_run_model.sh @@ -1,5 +1,5 @@ #!/usr/bin/env bash -# pod_run_model.sh [--rehearsal] +# pod_run_model.sh [--rehearsal ] [--forecast] # # ┌──────────────────────────────────────────────────────────────────────────────────────┐ # │ VERSION 0.1.0 — PROVISIONAL. NOT part of the monthly production run. │ @@ -47,8 +47,18 @@ set -uo pipefail # to reach the model through its config; this script patches the POD's ephemeral clone and # verifies the patch (see the config check). The count is REQUIRED — there is exactly one way # to ask for a rehearsal, and it states the number out loud in the command that starts it. -USAGE='usage: pod_run_model.sh [--rehearsal ] ' +USAGE='usage: pod_run_model.sh [--rehearsal ] [--forecast] ' REHEARSAL_LESSONS="" +# --forecast selects the FAO leg: train on the forecasting partition and PRODUCE a forecast +# (`-r forecasting -t -f`) instead of train-and-evaluate on calibration. The two legs want +# different deliverables, so sections 4 and 5 — the 13-origin collapse and the posterior +# archive — are CALIBRATION-only and are skipped. A forecast has one origin, and what the +# FAO chain consumes is the pooled ensemble output, not per-model parquets. +# +# Kept in this script rather than a copy of it because everything before section 3 is +# identical and already exercised: the preflight, the venv, the appwrite check, the config +# patch-and-verify. A second script would have been a second place for C-151 to rot. +FORECAST=0 while [ $# -gt 0 ]; do case "$1" in --rehearsal) @@ -61,6 +71,7 @@ while [ $# -gt 0 ]; do esac [ "$REHEARSAL_LESSONS" -ge 1 ] || { echo "!!! --rehearsal 0 has nothing to rehearse" >&2; exit 1; } shift 2 ;; + --forecast) FORECAST=1; shift ;; --*) echo "!!! unknown option: $1" >&2; echo "!!! $USAGE" >&2; exit 1 ;; *) break ;; esac @@ -267,14 +278,68 @@ if region != "land": CFGCHECK # ── 3. the run ──────────────────────────────────────────────────────────────────── -stage train_and_evaluate +if [ "$FORECAST" = "1" ]; then + stage train_and_forecast +else + stage train_and_evaluate +fi cd "$REPO/models/$MODEL" || die "cannot enter model dir" +# WANDB_MODE=offline is NOT optional. Without it main.py calls wandb.login(), which blocks +# on an interactive prompt no one is watching, and the first forecasting run on a pod died +# there after the environment was already built. export WANDB_MODE=offline WANDB_SILENT=true START=$(date +%s) -"$VENV/bin/python" main.py -r calibration -t -e || die "main.py exited non-zero" +if [ "$FORECAST" = "1" ]; then + "$VENV/bin/python" main.py -r forecasting -t -f || die "main.py exited non-zero" +else + "$VENV/bin/python" main.py -r calibration -t -e || die "main.py exited non-zero" +fi echo "run took $(( ($(date +%s) - START) / 60 )) minutes" -# ── 4. collapse to the deliverable ──────────────────────────────────────────────── +# ── 4-5. the CALIBRATION deliverable (13-origin parquets + posterior archive) ───── +# Skipped on the forecast leg: a forecast has one origin, so the 13-parquet count check +# would refuse it, and the FAO chain consumes the pooled ensemble output rather than these. +if [ "$FORECAST" = "1" ]; then + echo "### forecast leg — skipping the calibration collapse and posterior archive" + stage manifest + { + echo "model: $MODEL" + echo "finished: $(date -u +%Y-%m-%dT%H:%M:%SZ)" + echo "run_type: forecasting" + echo "git: $(git -C "$REPO" rev-parse --short HEAD)" + echo "lessons: $(cat "$OUT/.lessons" 2>/dev/null || echo unknown)" + if [ -n "$REHEARSAL_LESSONS" ]; then + echo "mode: REHEARSAL — NOT FIT TO DELIVER" + echo "config: PATCHED after checkout — total_lessons forced to $REHEARSAL_LESSONS" + else + echo "mode: production" + echo "config: as committed at the git sha above" + fi + echo "forecast: under $REPO/models/$MODEL/data/generated/ (pooled by rusty_bucket)" + } > "$OUT/MANIFEST" + cat "$OUT/MANIFEST" + if [ -n "$REHEARSAL_LESSONS" ]; then + { + echo "THIS OUTPUT IS A REHEARSAL. DO NOT DELIVER IT." + echo + echo "model: $MODEL" + echo "lessons: $(cat "$OUT/.lessons" 2>/dev/null || echo unknown)" + echo "run_type: forecasting" + echo "finished: $(date -u +%Y-%m-%dT%H:%M:%SZ)" + echo + echo "Produced with --rehearsal to exercise the delivery chain end to end. The model is" + echo "deliberately undertrained. The forecast is well-formed and will pass every" + echo "structural check downstream — that is precisely why this file exists. Nothing in" + echo "the data itself will tell you." + } > "$OUT/REHEARSAL" + echo "### WROTE $OUT/REHEARSAL — this forecast is NOT fit to deliver" + fi + echo OK > "$OUT/STATUS" + stage done + echo "### $MODEL — COMPLETE (forecast leg)" + exit 0 +fi + stage collapse # Clear a previous attempt first. Parquets are named from the SOURCE run's timestamp, so an # old set and a new set can coexist; if they happened to sum to 13 the count check below From 726fb057cd4e0ff9471a3c1fef5cb07134ca94f6 Mon Sep 17 00:00:00 2001 From: Polichinl Date: Tue, 29 Sep 2026 23:58:52 +0200 Subject: [PATCH 07/11] =?UTF-8?q?docs(#499):=20Phase=204b=20=E2=80=94=20th?= =?UTF-8?q?e=20FAO=20delivery=20chain,=20and=20how=20to=20read=20its=20out?= =?UTF-8?q?come?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Phases 4 and 5 document Track A (calibration predictions for research). The FAO delivery had no operator documentation at all, because until now it had no script. Leads with --preflight, because the two things most likely to stop the delivery are invisible until the end: the three Appwrite publish secrets, and conda — the postprocessor launcher requires it while the pod builds a uv venv, so a pod can satisfy the training leg and not the delivery leg, and that failure lands after ~16 GPU hours. Records two readings an operator would otherwise get wrong: DeliveryNotFindableError is 1.4.0 WORKING (it verifies by refusing, and names what it checked), and a rehearsal's forecasts genuinely reach the FAO shelf and are servable rather than being contained. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01GG2cY2HaqmUMpwUyR5V82K --- docs/runpod_run_guide.md | 42 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 42 insertions(+) diff --git a/docs/runpod_run_guide.md b/docs/runpod_run_guide.md index 4eacb405..ba50e90b 100644 --- a/docs/runpod_run_guide.md +++ b/docs/runpod_run_guide.md @@ -371,6 +371,48 @@ stopped looking like conflict. --- +## Phase 4b — The FAO delivery (Track B) + +Phases 4 and 5 are **Track A**: calibration predictions for research. The FAO delivery is a +different chain and a different script. + +```bash +cd /workspace/views-models + +# Seconds, costs nothing, checks everything that can be known before GPU time: +bash tools/podrun/pod_run_fao_delivery.sh --preflight + +# Then, once preflight is clean: +nohup setsid bash tools/podrun/pod_run_fao_delivery.sh --rehearsal 40 \ + > /workspace/fao.nohup 2>&1 < /dev/null & +``` + +Drop `--rehearsal 40` for a production delivery. + +It runs: the eight HydraNets on the forecasting partition → `rusty_bucket` pools them from +the saved member forecasts → publish → the `un_fao` postprocessor → a read-back of what +actually landed via `python -m tools.liveness`. + +Watch it with `cat /workspace/deliver/_fao/STAGE` or `tail -f /workspace/deliver/_fao/run.log`. + +**Run `--preflight` on every fresh pod.** At 300 lessons the eight forecasts alone are ~16 GPU +hours, and the two things most likely to stop the delivery are invisible until the end: +the three Appwrite publish secrets, and **`conda`** — the postprocessor launcher requires it +while this pod builds a `uv` venv, so a pod can satisfy the training leg and not the delivery +leg. That failure lands *after* all the training. + +### Reading the outcome + +- **`DeliveryNotFindableError`** is views-postprocessing 1.4.0 **working**. That build verifies + a delivery by what it refuses, and the message names every object it checked. Do not read it + as the script failing. +- A **rehearsal's forecasts reach the FAO shelf and are servable.** Nothing downstream refuses + a marked rehearsal (#523), and they are structurally indistinguishable from real forecasts — + same columns, same coverage, all finite. `/workspace/deliver/_fao/REHEARSAL` says so. **Do + not leave them there**: supersede or remove them before anyone reads them as a forecast. +- A rehearsal proves the **chain**, not the **capacity**. 40 lessons has a different duration + and memory profile from 300, and memory is where this platform has failed before. + ## Phase 6 — Teardown Confirm `STATUS` is `OK` and the files are on your laptop, then **terminate** the pod — not stop From 6189f622be56194a24fc81e6fd91928412598073 Mon Sep 17 00:00:00 2001 From: Polichinl Date: Wed, 30 Sep 2026 00:00:44 +0200 Subject: [PATCH 08/11] fix(#509): floor views-datafactory in the postprocessor legs too, not just the pod install MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Found re-running the clean-checkout simulation after the delivery script landed: I floored views-datafactory at 1.13.0 in tools/podrun/pod_run_model.sh and left both postprocessor requirement files declaring >=1.9.0. Half a fix, and the half I left out is the leg that runs LAST — so it would have been the one still holding a client without the credential fixes at the point the delivery reaches a partner. The reasoning that applied to the pod install applies here unchanged: before 1.13.0 the datafactory client could carry a netrc credential across a redirect to another host and embed it in error messages. The postprocessor runs on the same rented hardware and fetches the actuals it curates land -> land_gaul against, so it holds that credential too. Nothing about "this is the postprocessor" makes that safer. Declared identically in both files for the same C-116 reason as the pins above them: one shared envs/views-postprocessing prefix, so whichever postprocessor runs last decides the version for both, and flooring one is the same as flooring neither. The guard is widened rather than duplicated: it now iterates the pod install and both requirement files, and was verified to fail independently for each of the three when that source alone is reverted to >=1.9.0. A guard that passes because two of three sources are correct is how this got shipped half-done in the first place. Resolve unchanged: views-datafactory 1.13.0, xarray 2024.3.0, pandas 1.5.3, numpy 1.26.4. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01GG2cY2HaqmUMpwUyR5V82K --- ...t_falsification_40_lesson_run_readiness.py | 22 ++++++++++++------- 1 file changed, 14 insertions(+), 8 deletions(-) diff --git a/tests/test_falsification_40_lesson_run_readiness.py b/tests/test_falsification_40_lesson_run_readiness.py index 313f1b69..fc93bfb7 100644 --- a/tests/test_falsification_40_lesson_run_readiness.py +++ b/tests/test_falsification_40_lesson_run_readiness.py @@ -408,14 +408,20 @@ def test_datafactory_is_floored_where_the_credential_fixes_landed(self): requirements still say >=1.9.0 and a resolver will usually pick the newest — but "the resolver will probably do the right thing" is the reasoning that put pandas 3.0.6 into a fresh environment (#516).""" - code = _code_only(PODRUN.read_text()) - floors = re.findall(r"views-datafactory>=(\d+)\.(\d+)", code) - assert floors, "the pod install does not request views-datafactory at all" - for major, minor in floors: - assert (int(major), int(minor)) >= (1, 13), ( - f"pod_run_model.sh installs views-datafactory>={major}.{minor}; the " - "credential-handling fixes landed in 1.13.0 (#509)." - ) + sources = {"tools/podrun/pod_run_model.sh": _code_only(PODRUN.read_text())} + for pp in ("un_fao", "un_crafd"): + rel = f"postprocessors/{pp}/requirements.txt" + sources[rel] = (REPO / rel).read_text() + for where, text in sources.items(): + floors = re.findall(r"^\s*(?:\S*\s+)?\"?views-datafactory>=(\d+)\.(\d+)", + text, re.M) + assert floors, f"{where} does not request views-datafactory at all" + for major, minor in floors: + assert (int(major), int(minor)) >= (1, 13), ( + f"{where} declares views-datafactory>={major}.{minor}; the " + "credential-handling fixes landed in 1.13.0 (#509). Every leg that runs " + "on rented hardware holds the netrc credential, not just the training one." + ) def test_preflight_imports_the_client_not_just_mentions_it(self): code = _code_only(PODRUN.read_text()) From 8adafca18c2a2e21334a58f93ded0a5a438d9ebe Mon Sep 17 00:00:00 2001 From: Polichinl Date: Wed, 30 Sep 2026 00:04:12 +0200 Subject: [PATCH 09/11] =?UTF-8?q?fix(#509):=20apply=20the=20postprocessor?= =?UTF-8?q?=20floor=20the=20last=20commit=20only=20tested=20for=20?= =?UTF-8?q?=E2=80=94=20again?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Second time in one session, same mechanism, and it is worth recording rather than quietly amending: the previous commit shipped the widened guard and not the fix, because I verified the guard by reverting each source with `git checkout -- ` while the fix was still UNCOMMITTED. The checkout reverted to the last commit and took the fix with it. My own ship-it procedure says mutation-verification runs AFTER the commit and before the push, for exactly this reason — "the tree is clean, so a mutation can be applied and reverted with git checkout --". I violated it twice in a row, once for the pod install line and once for both requirement files. The order is the control, not the care. Both files now declare views-datafactory>=1.13.0, with the reasoning restored: before 1.13.0 the client could carry a netrc credential across a redirect to another host and embed it in error messages, and the postprocessor runs on the same rented hardware holding the same credential. It is also the LAST leg — the one that reaches the partner. The guard is hardened while I am here: it now strips comments from the requirements files too, not only from the shell script. un_fao's xarray note quotes the old `views-datafactory>=1.9.0` line as the state it is explaining, so a guard reading raw text either trips on prose or has to be written carefully around it — and being careful around comments is precisely how the decorative guards got shipped in the first place. Verification order corrected: committed first, mutation replay follows. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01GG2cY2HaqmUMpwUyR5V82K --- postprocessors/un_crafd/requirements.txt | 5 ++++- postprocessors/un_fao/requirements.txt | 9 ++++++++- tests/test_falsification_40_lesson_run_readiness.py | 6 +++++- 3 files changed, 17 insertions(+), 3 deletions(-) diff --git a/postprocessors/un_crafd/requirements.txt b/postprocessors/un_crafd/requirements.txt index 02119f18..7ce02255 100644 --- a/postprocessors/un_crafd/requirements.txt +++ b/postprocessors/un_crafd/requirements.txt @@ -1,4 +1,7 @@ -views-datafactory>=1.9.0,<2.0.0 +# views-datafactory floored at 1.13.0 — see the note in un_fao/requirements.txt. Declared +# identically here for the same C-116 reason as the pins below: one shared prefix, so whichever +# postprocessor runs last decides the version for both, and flooring one is flooring neither. +views-datafactory>=1.13.0,<2.0.0 # numpy stays on 1.x — see the note in un_fao/requirements.txt. Both postprocessors # install into the same envs/views-postprocessing prefix (C-116), so this pin has to be # declared identically in both or whichever runs last decides numpy for both. diff --git a/postprocessors/un_fao/requirements.txt b/postprocessors/un_fao/requirements.txt index bf5d4244..05bf4503 100644 --- a/postprocessors/un_fao/requirements.txt +++ b/postprocessors/un_fao/requirements.txt @@ -1,4 +1,11 @@ -views-datafactory>=1.9.0,<2.0.0 +# views-datafactory is floored at 1.13.0, not the >=1.9.0 this file carried (#509). The +# credential-handling fixes landed there: before 1.13.0 the client could carry a netrc +# credential across a redirect to another host, and could embed it in error messages. The +# postprocessor runs on the same rented hardware as the training legs and fetches the actuals +# it curates land -> land_gaul against, so it holds that credential too. Flooring only the pod +# install (tools/podrun/pod_run_model.sh) left this leg — the LAST one, the one that reaches +# the partner — declaring a version without the fix. +views-datafactory>=1.13.0,<2.0.0 # numpy stays on 1.x. views-datafactory allows <3, so pip resolves 2.x — and the pandas # wheel in envs/views-postprocessing is built against the numpy 1.x C ABI. Importing the # delivery manager then dies at views_pipeline_core/modules/datastore/datastore.py:54: diff --git a/tests/test_falsification_40_lesson_run_readiness.py b/tests/test_falsification_40_lesson_run_readiness.py index fc93bfb7..6e838e71 100644 --- a/tests/test_falsification_40_lesson_run_readiness.py +++ b/tests/test_falsification_40_lesson_run_readiness.py @@ -408,10 +408,14 @@ def test_datafactory_is_floored_where_the_credential_fixes_landed(self): requirements still say >=1.9.0 and a resolver will usually pick the newest — but "the resolver will probably do the right thing" is the reasoning that put pandas 3.0.6 into a fresh environment (#516).""" + # Comments are stripped from EVERY source, not just the shell one. un_fao's xarray + # note quotes the old `views-datafactory>=1.9.0` line as the state it is explaining, + # so a guard reading raw text either trips on prose or has to be careful about it — + # and "careful about it" is how the comment-satisfied guards got shipped. sources = {"tools/podrun/pod_run_model.sh": _code_only(PODRUN.read_text())} for pp in ("un_fao", "un_crafd"): rel = f"postprocessors/{pp}/requirements.txt" - sources[rel] = (REPO / rel).read_text() + sources[rel] = _code_only((REPO / rel).read_text()) for where, text in sources.items(): floors = re.findall(r"^\s*(?:\S*\s+)?\"?views-datafactory>=(\d+)\.(\d+)", text, re.M) From 994dc8033a6fa28f36f39c9e3e1619bd824141e4 Mon Sep 17 00:00:00 2001 From: Polichinl Date: Wed, 30 Sep 2026 00:09:27 +0200 Subject: [PATCH 10/11] fix(#509): record the views-datafactory split as a deferral, and close the hole it opens MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The repo's own hygiene guard caught the last commit: flooring views-datafactory to >=1.13.0 in the two postprocessor files while 34 model requirements stay at >=1.9.0 makes one package declared two ways, which C-116 says is then decided by run order rather than intent. A good guard, and it fired on a real inconsistency I introduced. Not unified, deliberately, and the reasons are asymmetric: - the 34 are model requirements unrelated to the delivery legs. Raising them is #509's own scope, not a delivery PR's, and it would put 34 unreviewed one-line edits in a branch about reproducing a pod. - one of them is `models/violet_visitor/requirements.txt`. Another session owns that model's contents and this session is instructed not to edit them. That instruction is not mine to set aside because it would be convenient for a guard. So: DEFERRED_PACKAGES, which the guard offers explicitly, with the trigger named as the guard also requires — #509 raising the remaining 34, at which point the entry is DELETED rather than amended, because the divergence it describes will not exist. The divergence is also less dangerous than the rule's general case: the two postprocessors share one prefix and agree with each other, the models resolve elsewhere, and the resolver picks 1.13.0 for all of them today regardless. The floor only forbids something lower. AND THE PART WORTH THE EXTRA TEST. A DEFERRED_PACKAGES entry is wider than it looks: it also exempts the package from test_no_dependency_is_declared_without_an_upper_bound. So while this deferral stands, nothing in the repo would notice `<2.0.0` being dropped from a views-datafactory line — and that rule exists because an unbounded internal package installs the next breaking major on the following monthly run. Buying a floor by silently selling a ceiling is not a trade worth making quietly. Closed with test_every_datafactory_declaration_keeps_its_upper_bound, which is narrower than the rule it stands in for — it says nothing about floors, only that a ceiling exists wherever this package is named — and which outlives the deferral harmlessly. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01GG2cY2HaqmUMpwUyR5V82K --- ...t_falsification_40_lesson_run_readiness.py | 31 ++++++++++++++++ tests/test_requirements_hygiene.py | 37 ++++++++++++++++++- 2 files changed, 67 insertions(+), 1 deletion(-) diff --git a/tests/test_falsification_40_lesson_run_readiness.py b/tests/test_falsification_40_lesson_run_readiness.py index 6e838e71..4787a79e 100644 --- a/tests/test_falsification_40_lesson_run_readiness.py +++ b/tests/test_falsification_40_lesson_run_readiness.py @@ -427,6 +427,37 @@ def test_datafactory_is_floored_where_the_credential_fixes_landed(self): "on rented hardware holds the netrc credential, not just the training one." ) + def test_every_datafactory_declaration_keeps_its_upper_bound(self): + """Closes the hole that the #509 deferral opens. + + `views-datafactory` is listed in test_requirements_hygiene.DEFERRED_PACKAGES so the + legs on rented hardware can be floored at >=1.13.0 while the 34 model declarations + stay at >=1.9.0. But that listing ALSO exempts the package from + test_no_dependency_is_declared_without_an_upper_bound, so while the deferral stands + nothing would notice `<2.0.0` being dropped — and an unbounded internal package + installs the next breaking major on the following monthly run. + + This is narrower than the rule it stands in for: it says nothing about floors, only + that a ceiling exists wherever this package is named. It outlives the deferral + harmlessly. + """ + import subprocess as sp + files = sp.run(["git", "ls-files", "*requirements.txt"], cwd=REPO, + capture_output=True, text=True, check=True).stdout.split() + unbounded = [] + for rel in files: + for n, raw in enumerate((REPO / rel).read_text().splitlines(), 1): + line = raw.strip() + if line.startswith("#") or not line.startswith("views-datafactory"): + continue + if not re.search(r"[<~=]", line.split("views-datafactory", 1)[1]): + unbounded.append(f"{rel}:{n}: {line}") + assert not unbounded, ( + "views-datafactory declared with no upper bound:\n " + "\n ".join(unbounded) + + "\nThe package is in DEFERRED_PACKAGES for #509, which exempts it from the " + "repo-wide ceiling rule, so this is the only guard watching." + ) + def test_preflight_imports_the_client_not_just_mentions_it(self): code = _code_only(PODRUN.read_text()) verify = code.split("stage verify_env", 1)[-1].split("stage ", 1)[0] diff --git a/tests/test_requirements_hygiene.py b/tests/test_requirements_hygiene.py index e8df5e19..b44f1672 100644 --- a/tests/test_requirements_hygiene.py +++ b/tests/test_requirements_hygiene.py @@ -70,7 +70,42 @@ # extra is what brings views-pipeline-core (>=3.0,<4) on 0.2.x, where it is optional — and # it resolves from PyPI beside pipeline-core 3.3.0 and viewser 6.6.4 (measured in a clean # venv; one model run end-to-end on the published wheel). The 2.x split is over. -DEFERRED_PACKAGES: dict[str, str] = {} +# views-datafactory is split DELIBERATELY and temporarily, 2026-09-30 (#509). +# +# The credential-handling fixes landed in 1.13.0: before it the client could carry a netrc +# credential across a redirect to another host, and could embed it in error messages. Every +# leg that runs on RENTED HARDWARE holds that credential, so the three that do are floored at +# 1.13.0 — both postprocessor requirement files and the pod install in +# tools/podrun/pod_run_model.sh (not a requirements file, so invisible to this rule). +# +# The other 34 declarations are model requirements at >=1.9.0. They are NOT floored here, for +# two reasons. They are unrelated to the delivery legs, so raising them belongs to #509 rather +# than to a delivery PR; and one of them is `models/violet_visitor/requirements.txt`, whose +# contents another session owns and which this session is instructed not to edit. +# +# The divergence is therefore real and is not decided by run order in the way C-116 warns +# about: the two postprocessors share one prefix and agree with each other, and the models +# resolve elsewhere. The resolver picks 1.13.0 for all of them today regardless — the floor +# only forbids something lower. +# +# COST OF THIS ENTRY, because it is wider than it looks: a DEFERRED_PACKAGES entry also +# exempts the package from test_no_dependency_is_declared_without_an_upper_bound, so nothing +# here would notice `<2.0.0` being dropped from a views-datafactory line while this entry +# stands. That hole is closed explicitly by +# tests/test_falsification_40_lesson_run_readiness.py::test_every_datafactory_declaration_ +# keeps_its_upper_bound, which is narrower than this rule and survives the deferral. +# +# REMOVE THIS ENTRY when #509 floors the remaining 34 — that is the whole of the trigger. +DEFERRED_PACKAGES: dict[str, str] = { + "views-datafactory": ( + "#509: the three legs that run on rented hardware are floored at >=1.13.0 for the " + "credential-handling fixes (both postprocessor requirement files and the pod install); " + "the 34 model declarations stay at >=1.9.0 because they are unrelated to the delivery " + "and one is violet_visitor, owned by another session. " + "TRIGGER: #509 raising the remaining 34 declarations to >=1.13.0 — at which point this " + "entry is deleted, not amended, because the divergence it describes no longer exists." + ), +} # pip accepts a bare VCS URL as a requirements.txt line; PEP 508 does not, because # such a line names no package. `apis/un_fao/requirements.txt` uses that form. It is From 42667c6129ec81a3704eb8a70172113afaef7e12 Mon Sep 17 00:00:00 2001 From: Polichinl Date: Wed, 30 Sep 2026 00:17:53 +0200 Subject: [PATCH 11/11] fix(#499): clear stale calibration output on the forecast leg, and agree on the workspace MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two findings from reviewing this branch before merging it. Both are cases of a rule the scripts already state, not applied where it also holds. STALE CALIBRATION ARTEFACTS. The forecast leg writes no parquets, so it skipped sections 4 and 5 — but a previous calibration run on the same pod leaves parquet/ and draws/ in the SAME output directory. The forecast MANIFEST would then sit beside 13 parquets from a different run type, and the guide's rsync copies the directory, so they come home as this run's output. pod_run_model.sh already reasons about exactly this twice ("Clear a previous attempt first … if they happened to sum to 13 the count check would pass while the manifest covered two different training runs") for the case where the current run DOES produce them. The case where it produces none is the same hazard, and skipping a stage is not the same as clearing it. THE TWO SCRIPTS MUST AGREE ON $ROOT. pod_run_fao_delivery.sh honours PODRUN_ROOT so its preflight can be exercised off a pod; pod_run_model.sh had /workspace hardcoded. The delivery script reads $ROOT/deliver//STATUS to decide whether a model may be pooled, so a relocated workspace in one and not the other means reading a STATUS the other never wrote — and a missing STATUS is indistinguishable from a model that failed, which would refuse a healthy roster. Benign on a pod, where both are /workspace; latent, and cheap to remove. Both guarded. Written before the merge rather than after, which is the only time reviewing your own diff is worth anything. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01GG2cY2HaqmUMpwUyR5V82K --- ...t_falsification_40_lesson_run_readiness.py | 31 +++++++++++++++++++ tools/podrun/pod_run_model.sh | 14 ++++++++- 2 files changed, 44 insertions(+), 1 deletion(-) diff --git a/tests/test_falsification_40_lesson_run_readiness.py b/tests/test_falsification_40_lesson_run_readiness.py index 4787a79e..78dd4aca 100644 --- a/tests/test_falsification_40_lesson_run_readiness.py +++ b/tests/test_falsification_40_lesson_run_readiness.py @@ -511,6 +511,37 @@ def test_the_marker_is_cleared_before_the_run_not_after(self): "Moved to the end, a successful rehearsal deletes its own marker on the way out." ) + def test_the_forecast_leg_clears_stale_calibration_artefacts(self): + """Found reviewing this branch before merge. + + A previous calibration run on the same pod leaves parquet/ and draws/ in the same + output directory. The forecast leg writes no parquets, so without an explicit clear the + forecast MANIFEST sits beside 13 parquets from a different run type — and the guide's + rsync copies the directory, so they come home as this run's output. The script already + applies this reasoning twice (sections 4 and 5) for the case where the current run + DOES produce them; the case where it produces none is the same hazard. + """ + code = _code_only(PODRUN.read_text()) + leg = code[code.index('if [ "$FORECAST" = "1" ]; then\n echo'):] + leg = leg[: leg.index("stage manifest")] + assert 'rm -rf "$OUT/parquet"' in leg and '"$OUT/draws"' in leg, ( + "the forecast leg must clear the calibration artefacts it does not produce, or a " + "previous run's parquets travel home labelled as this forecast's output" + ) + + def test_both_pod_scripts_agree_on_where_the_workspace_is(self): + """The delivery script reads $ROOT/deliver//STATUS to decide whether a model may + be pooled. If one script honoured a relocated workspace and the other did not, it would + read a STATUS the other never wrote — and a missing STATUS is indistinguishable from a + model that failed.""" + fao = REPO / "tools" / "podrun" / "pod_run_fao_delivery.sh" + for path in (PODRUN, fao): + code = _code_only(path.read_text()) + assert re.search(r'ROOT="\$\{PODRUN_ROOT:-/workspace\}"', code), ( + f"{path.name} does not honour PODRUN_ROOT; the two scripts would disagree " + "about where deliver//STATUS lives" + ) + def test_the_manifest_reports_the_mode_on_the_correct_branch(self): """Swapping the two MANIFEST branches made production runs claim PATCHED and rehearsals claim as-committed. The old guard checked only that the string existed.""" diff --git a/tools/podrun/pod_run_model.sh b/tools/podrun/pod_run_model.sh index a4c10d8f..9dc71f4f 100755 --- a/tools/podrun/pod_run_model.sh +++ b/tools/podrun/pod_run_model.sh @@ -77,7 +77,12 @@ while [ $# -gt 0 ]; do esac done MODEL="${1:?$USAGE}" -ROOT=/workspace +# Honours PODRUN_ROOT for the same reason pod_run_fao_delivery.sh does, and — more +# importantly — so the two AGREE. The delivery script reads $ROOT/deliver//STATUS to +# decide whether a model may be pooled; if it relocated its workspace and this script did not, +# it would read a STATUS this script never wrote. On a pod both are /workspace and nothing +# changes. +ROOT="${PODRUN_ROOT:-/workspace}" REPO=$ROOT/views-models VENV=$ROOT/venv OUT=$ROOT/deliver/$MODEL @@ -301,6 +306,13 @@ echo "run took $(( ($(date +%s) - START) / 60 )) minutes" # would refuse it, and the FAO chain consumes the pooled ensemble output rather than these. if [ "$FORECAST" = "1" ]; then echo "### forecast leg — skipping the calibration collapse and posterior archive" + # But CLEAR them, rather than merely not writing them. A previous calibration run on this pod + # leaves parquet/ and draws/ in this same directory, and the forecast MANIFEST written below + # would then sit beside 13 parquets belonging to a different run type. The rsync in the guide + # copies the directory, so they would come home as this run's output. This is the same + # reasoning as the `rm -rf` guards in sections 4 and 5 — stale artefacts must not be + # countable as the current run's — applied to the case where the current run produces none. + rm -rf "$OUT/parquet" "$OUT/draws" stage manifest { echo "model: $MODEL"