diff --git a/docs/runpod_run_guide.md b/docs/runpod_run_guide.md index f5fba401..ba50e90b 100644 --- a/docs/runpod_run_guide.md +++ b/docs/runpod_run_guide.md @@ -180,6 +180,22 @@ grep -A2 '' ~/.netrc | ssh root@ -p -i ~/.ssh/id_ed25 'umask 077; cat > /root/.netrc; chmod 600 /root/.netrc' ``` +> **Put credentials on `/root`, never on `/workspace` — `/workspace` silently ignores +> `chmod`.** +> +> It is a network filesystem. `chmod 600` there **returns success and does nothing**: the +> file stays mode `666`, readable by every process on the machine, and there is no error to +> notice. `stat -c %a` is the only way to find out, and only if you think to look. +> +> Found on 2026-09-29 while placing the Appwrite publish credentials, which sat +> world-readable on rented hardware until they were moved to `/root/.secrets`. Registered as +> **views-models C-154**; see **#518**. +> +> This is why the command above writes to `/root/.netrc` and why the runner checks +> `stat -c %a /root/.netrc` rather than trusting its own `chmod`. Both are deliberate. The +> `umask 077` is what actually protects the file in transit — the `chmod` is a belt-and-braces +> check that happens to work *here* because `/root` is local disk. + Check it before trusting it — the pipe can silently deliver nothing: ```bash @@ -269,7 +285,38 @@ nohup setsid bash tools/podrun/pod_run_model.sh \ `setsid` matters: the run must survive your SSH session closing and your laptop sleeping. It refuses before spending GPU time if `.netrc` is missing, if `total_lessons` is still a -throwaway value, if `REGION` is not `"land"`, or if the converter is absent. +throwaway value, if `REGION` is not `"land"`, if the Appwrite client is not importable, or +if the converter is absent. + +### Rehearsing the chain first + +After a run that failed late, you usually want the whole chain exercised cheaply before you +commit to a full one. That is `--rehearsal`, and it takes the lesson count: + +```bash +nohup setsid bash tools/podrun/pod_run_model.sh --rehearsal 40 \ + > /workspace/.nohup 2>&1 < /dev/null & +``` + +It patches **the pod's clone** of `config_hyperparameters.py` to 40 lessons, re-reads the +config to confirm the patch took, and marks the output. Nothing tracked in git is edited — +the committed configs stay at their production value, which is the point: a rehearsal +obtained by committing a low lesson count is the failure mode this flag removes. + +A rehearsal leaves an extra file, **`REHEARSAL`**, beside the parquets, and `MANIFEST` gains +`mode: REHEARSAL — NOT FIT TO DELIVER`. You need them, because the parquets themselves are +indistinguishable from a real run's: same columns, same row counts, all finite and +non-negative, every structural check green. Nothing in the data will tell you the model +behind it is undertrained. + +**What a rehearsal does NOT prove.** It exercises the chain, not the capacity. A 40-lesson +run has a different duration and memory profile from a 300-lesson one, and memory is where +this platform has failed before. A green rehearsal means the hops connect; it does not mean +the full run will fit. + +**It marks, it does not refuse.** This runner uploads nothing, so it cannot stop a rehearsal +being published downstream — the `REHEARSAL` file is a warning to a person, not a guard. Do +not publish a directory that contains one. Monitor without disturbing it: @@ -296,8 +343,9 @@ The runner already does both. Its output is in `/workspace/deliver//`: |---|---| | `parquet/` | 13 files, one per origin, 2,333,448 rows each | | `draws/` | the full posterior, zstd — ~236× smaller than raw | -| `MANIFEST` | sizes, timestamps, git sha, lesson count | +| `MANIFEST` | sizes, timestamps, git sha, lesson count, and `mode:` | | `STATUS` | `OK`, or `FAILED:` | +| `REHEARSAL` | present **only** after `--rehearsal` — this output is not fit to deliver | From your laptop: @@ -323,6 +371,48 @@ stopped looking like conflict. --- +## Phase 4b — The FAO delivery (Track B) + +Phases 4 and 5 are **Track A**: calibration predictions for research. The FAO delivery is a +different chain and a different script. + +```bash +cd /workspace/views-models + +# Seconds, costs nothing, checks everything that can be known before GPU time: +bash tools/podrun/pod_run_fao_delivery.sh --preflight + +# Then, once preflight is clean: +nohup setsid bash tools/podrun/pod_run_fao_delivery.sh --rehearsal 40 \ + > /workspace/fao.nohup 2>&1 < /dev/null & +``` + +Drop `--rehearsal 40` for a production delivery. + +It runs: the eight HydraNets on the forecasting partition → `rusty_bucket` pools them from +the saved member forecasts → publish → the `un_fao` postprocessor → a read-back of what +actually landed via `python -m tools.liveness`. + +Watch it with `cat /workspace/deliver/_fao/STAGE` or `tail -f /workspace/deliver/_fao/run.log`. + +**Run `--preflight` on every fresh pod.** At 300 lessons the eight forecasts alone are ~16 GPU +hours, and the two things most likely to stop the delivery are invisible until the end: +the three Appwrite publish secrets, and **`conda`** — the postprocessor launcher requires it +while this pod builds a `uv` venv, so a pod can satisfy the training leg and not the delivery +leg. That failure lands *after* all the training. + +### Reading the outcome + +- **`DeliveryNotFindableError`** is views-postprocessing 1.4.0 **working**. That build verifies + a delivery by what it refuses, and the message names every object it checked. Do not read it + as the script failing. +- A **rehearsal's forecasts reach the FAO shelf and are servable.** Nothing downstream refuses + a marked rehearsal (#523), and they are structurally indistinguishable from real forecasts — + same columns, same coverage, all finite. `/workspace/deliver/_fao/REHEARSAL` says so. **Do + not leave them there**: supersede or remove them before anyone reads them as a forecast. +- A rehearsal proves the **chain**, not the **capacity**. 40 lessons has a different duration + and memory profile from 300, and memory is where this platform has failed before. + ## Phase 6 — Teardown Confirm `STATUS` is `OK` and the files are on your laptop, then **terminate** the pod — not stop diff --git a/postprocessors/un_crafd/requirements.txt b/postprocessors/un_crafd/requirements.txt index 233a0c59..7ce02255 100644 --- a/postprocessors/un_crafd/requirements.txt +++ b/postprocessors/un_crafd/requirements.txt @@ -1,5 +1,16 @@ -views-datafactory>=1.9.0,<2.0.0 +# views-datafactory floored at 1.13.0 — see the note in un_fao/requirements.txt. Declared +# identically here for the same C-116 reason as the pins below: one shared prefix, so whichever +# postprocessor runs last decides the version for both, and flooring one is flooring neither. +views-datafactory>=1.13.0,<2.0.0 # numpy stays on 1.x — see the note in un_fao/requirements.txt. Both postprocessors # install into the same envs/views-postprocessing prefix (C-116), so this pin has to be # declared identically in both or whichever runs last decides numpy for both. numpy>=1.26.4,<2.0.0 + +# xarray and pandas pinned — see the note in un_fao/requirements.txt for the measurement +# and for why the bound stops at xarray 2024.5.0. Declared identically here because both +# postprocessors install into the same envs/views-postprocessing prefix (C-116): pinning +# only one lets whichever runs last decide pandas for both, which is the same trap the +# numpy pin above already documents. +xarray>=2024.1,<2024.5 +pandas>=1.5.3,<2.0.0 diff --git a/postprocessors/un_fao/requirements.txt b/postprocessors/un_fao/requirements.txt index a18a600e..05bf4503 100644 --- a/postprocessors/un_fao/requirements.txt +++ b/postprocessors/un_fao/requirements.txt @@ -1,4 +1,11 @@ -views-datafactory>=1.9.0,<2.0.0 +# views-datafactory is floored at 1.13.0, not the >=1.9.0 this file carried (#509). The +# credential-handling fixes landed there: before 1.13.0 the client could carry a netrc +# credential across a redirect to another host, and could embed it in error messages. The +# postprocessor runs on the same rented hardware as the training legs and fetches the actuals +# it curates land -> land_gaul against, so it holds that credential too. Flooring only the pod +# install (tools/podrun/pod_run_model.sh) left this leg — the LAST one, the one that reaches +# the partner — declaring a version without the fix. +views-datafactory>=1.13.0,<2.0.0 # numpy stays on 1.x. views-datafactory allows <3, so pip resolves 2.x — and the pandas # wheel in envs/views-postprocessing is built against the numpy 1.x C ABI. Importing the # delivery manager then dies at views_pipeline_core/modules/datastore/datastore.py:54: @@ -8,3 +15,28 @@ views-datafactory>=1.9.0,<2.0.0 # Declared identically in un_crafd/requirements.txt — one shared prefix (C-116), so # pinning only one lets the other upgrade numpy out from under it on its next run. numpy>=1.26.4,<2.0.0 + +# xarray and pandas are pinned because xarray is the CARRIER, and it was the only thing +# in this file's dependency closure free to move. views-datafactory declares +# `xarray>=2024.1,<2026` as a required dependency and `pandas>=2.0` only in its optional +# `pandas` extra — which is not installed — so pandas arrives transitively THROUGH xarray +# and nothing here constrained either one. +# +# Measured 2026-09-29, resolving this file as it stood: +# declared: views-datafactory>=1.9.0,<2.0.0 + numpy>=1.26.4,<2.0.0 +# resolved: numpy 1.26.4, xarray 2025.12.0, pandas 3.0.6 +# the pod that produced the first FAO delivery ran: xarray 2024.3.0, pandas 1.5.3 +# +# So this file did not fail to build — it built, on a different pandas MAJOR, silently. +# views-models#516 originally called the environment "unbuildable"; that was wrong and the +# wrong version is the more dangerous one, because a build failure is loud. The working +# versions existed only as hand-pins on a pod that has since been destroyed. +# +# The bound is measured, not guessed. xarray 2024.3.0 is the LAST release that accepts +# pandas 1.x; 2024.5.0 moved to `pandas>=2.0` and 2024.9.0 to `pandas>=2.1`. A tidy-looking +# `<2025` cap would therefore have been wrong — the cliff is inside the 2024 line, not at +# the year boundary. Lifting these means moving the platform off pandas 1.x deliberately, +# which views-pipeline-core already permits (`pandas>=1.5.3,<3.0`) and this codebase does +# not yet assume. +xarray>=2024.1,<2024.5 +pandas>=1.5.3,<2.0.0 diff --git a/tests/test_falsification_40_lesson_run_readiness.py b/tests/test_falsification_40_lesson_run_readiness.py new file mode 100644 index 00000000..78dd4aca --- /dev/null +++ b/tests/test_falsification_40_lesson_run_readiness.py @@ -0,0 +1,592 @@ +""" +Falsification audit of the claim, 2026-09-29: +"We are ready to start a new full 40-lesson FAO delivery run — nothing remains but +creating a pod." + +Source: /falsify skill, claim mode. Verdict: FALSIFIED, three hard falsifications. + +All three had ONE cause: every fix that made the 2026-09-29 run work was applied BY HAND +to the pod, and the pod was destroyed. The repository never learned any of them. The pod +was the artefact and nothing in git described it. + + H1 the run could not be launched: pod_run_model.sh refuses total_lessons < 300 and every + config declares 300, so a cheap end-to-end test was reachable only by deleting the + guard — producing output indistinguishable from a real run. + H2 a fresh env built SUCCESSFULLY and WRONG: the declared requirements resolved to + xarray 2025.12.0 / pandas 3.0.6 where the working pod had 2024.3.0 / 1.5.3, neither + pinned. #516 called this "unbuildable"; it is not, and the silent version is worse. + H3 `appwrite` appeared nowhere in pod_run_model.sh, so `_build_datastore` failed at + publish — after the full training run. That is how 2026-09-29 failed (#517). + +──────────────────────────────────────────────────────────────────────────────────────── +SECOND AUDIT, same day: this file's FIRST version was 22 green assertions that protected +NOTHING. An independent /falsify guard-mode pass reverted every fix in the commit, kept the +comments, and got 22/22 green. 20 of 24 mutations survived; 10 guards were DECORATIVE. + +The mechanism, and it is the lesson: every assertion read the script as TEXT, and +pod_run_model.sh is unusually well commented — each fix carries a paragraph naming the +incident and quoting the exact strings. So THE BETTER THE COMMENT, THE WEAKER THE GUARD: +deleting the code left the comment, and the comment satisfied the assertion. + +Worst single case: `requested = os.environ.get("REHEARSAL_LESSONS") or ""` changed to +`or "40"`. One word. Every production run then patches itself to 40 lessons, the floor is +dead, and because the SHELL variable stays empty the MANIFEST says `mode: production` and +no REHEARSAL marker is written. A 40-lesson model, labelled a production delivery, 22/22 +green. Two of the five defects fused by a one-token diff. + +Worse still, one assertion certified a safety property that DOES NOT EXIST: it claimed the +RunPod guide documents the /workspace chmod trap. It does not. The regex matched because +`/workspace` appears on line 104 and `chmod` on line 180, joined by `.*` under re.S — the +identical defect the first commit message boasted of having found and deleted elsewhere. + +So this version: + * EXECUTES the config-check program against fixture models, rather than grepping it. That + block is where all the rehearsal/production logic lives and it was wholly unguarded. + * STRIPS COMMENTS before any remaining text assertion, so a comment can never stand in + for code. + * asserts version-specifier SEMANTICS (is 1.5.3 allowed? is 3.0.6 refused?) instead of + the mere presence of a pin — `pandas>=3.0` passed the old guard. + * CALLS get_hp_config() instead of pattern-matching the literal, which is what + pod_run_model.sh itself insists on doing, citing #501 "the guard that was not one". +""" + +import importlib.util +import re +import subprocess +import sys +from pathlib import Path + +import pytest +from packaging.specifiers import SpecifierSet + +REPO = Path(__file__).resolve().parents[1] +PODRUN = REPO / "tools" / "podrun" / "pod_run_model.sh" + +HYDRANETS = ( + "purple_alien", "pink_pirate", "blue_stranger", "bright_starship", + "heavy_freighter", "blazing_meteor", "bold_comet", "violet_visitor", +) + + +def _code_only(src: str) -> str: + """Shell source with comments removed. + + Every decorative guard in the first version of this file was satisfied by a comment + after its code was deleted. Any assertion about what the script DOES must therefore + read only what the script RUNS. Full-line and trailing comments both go; this is + deliberately cruder than a shell parser, and cruder is the safe direction here — it + removes text an assertion might otherwise lean on. + """ + out = [] + for line in src.splitlines(): + stripped = re.sub(r"(? str: + """The config-check program, lifted out of its heredoc so it can be RUN.""" + src = PODRUN.read_text() + m = re.search(r"<<'CFGCHECK'[^\n]*\n(.*?)\nCFGCHECK\s*?\n", src, re.S) + assert m, "the CFGCHECK heredoc is gone from pod_run_model.sh — re-read the script" + return m.group(1) + + +def _make_model(tmp_path: Path, lessons_body: str, region: str = '"land"') -> Path: + """A fixture model directory inside a real git repo. + + Real git, because the leftover-patch detector runs `git status --porcelain`; a fake + would let that guard pass without the mechanism it depends on existing. + """ + model = tmp_path / "model" + (model / "configs").mkdir(parents=True) + (model / "configs" / "config_hyperparameters.py").write_text(lessons_body) + (model / "configs" / "config_queryset.py").write_text(f"REGION = {region}\n") + for cmd in ( + ["git", "init", "-q", "."], + ["git", "config", "user.email", "t@t"], + ["git", "config", "user.name", "t"], + ["git", "add", "-A"], + ["git", "commit", "-qm", "fixture"], + ): + subprocess.run(cmd, cwd=model, check=True, capture_output=True) + return model + + +def _run_cfgcheck(model: Path, out: Path, rehearsal: str = "") -> subprocess.CompletedProcess: + out.mkdir(parents=True, exist_ok=True) + script = out / "_cfgcheck.py" + script.write_text(_extract_cfgcheck()) + return subprocess.run( + [sys.executable, str(script), str(model)], + capture_output=True, text=True, + env={ + "PATH": "/usr/bin:/bin:/usr/local/bin", + "REHEARSAL_LESSONS": rehearsal, + "OUT": str(out), + "MODEL": "fixture_model", + "HOME": str(out), + }, + ) + + +def _config_lessons(model: Path) -> int: + """What get_hp_config() RETURNS — not what the file text says.""" + path = model / "configs" / "config_hyperparameters.py" + spec = importlib.util.spec_from_file_location(f"_probe_{id(path)}", path) + mod = importlib.util.module_from_spec(spec) + spec.loader.exec_module(mod) + return mod.get_hp_config()["total_lessons"] + + +PRODUCTION_CONFIG = "def get_hp_config():\n return {'total_lessons': 300}\n" +CHEAP_CONFIG = "def get_hp_config():\n return {'total_lessons': 40}\n" + + +class TestTheProductionFloorActuallyRefuses: + """H1, executed. The floor is the only thing standing between an accidental config and + a full GPU budget, and the first version of this file asserted its SOURCE LINE while a + mutation that changed `sys.exit(` to `print(` sailed through: the run announced + "expected >= 300" and trained anyway.""" + + def test_a_cheap_config_is_refused_with_a_nonzero_exit(self, tmp_path): + model = _make_model(tmp_path, CHEAP_CONFIG) + r = _run_cfgcheck(model, tmp_path / "out") + assert r.returncode != 0, ( + "a 40-lesson config must REFUSE in production mode. Exit code 0 here means the " + f"pod trains a throwaway model on paid hardware.\nstdout: {r.stdout}" + ) + assert "--rehearsal" in (r.stdout + r.stderr), ( + "the refusal must name the supported way to get a cheap run, or the operator " + "edits the guard out — which is the state this audit found" + ) + + def test_a_production_config_passes(self, tmp_path): + model = _make_model(tmp_path, PRODUCTION_CONFIG) + r = _run_cfgcheck(model, tmp_path / "out") + assert r.returncode == 0, f"a 300-lesson production run must proceed.\n{r.stdout}\n{r.stderr}" + + def test_production_mode_does_not_patch_the_config(self, tmp_path): + """The one-word mutation `or ""` -> `or "40"` made every production run patch itself + to 40 lessons while the MANIFEST still said `mode: production`. Nothing saw it.""" + model = _make_model(tmp_path, PRODUCTION_CONFIG) + _run_cfgcheck(model, tmp_path / "out") + assert _config_lessons(model) == 300, ( + "a production run must not modify the config it was given. If this fails, a run " + "labelled 'production' in the MANIFEST trained on a patched lesson count." + ) + assert (tmp_path / "out" / ".lessons").read_text().strip() == "300" + + +class TestRehearsalPatchesOnlyThePodAndProvesIt: + """H1 / #523, executed.""" + + def test_rehearsal_patches_the_config_and_records_the_real_value(self, tmp_path): + model = _make_model(tmp_path, PRODUCTION_CONFIG) + r = _run_cfgcheck(model, tmp_path / "out", rehearsal="40") + assert r.returncode == 0, f"--rehearsal 40 must proceed.\n{r.stdout}\n{r.stderr}" + assert _config_lessons(model) == 40, "the pod's config must actually be patched" + assert (tmp_path / "out" / ".lessons").read_text().strip() == "40", ( + "the MANIFEST reads .lessons; it must carry the value the run was gated on" + ) + + def test_a_patch_that_does_not_take_is_refused_not_believed(self, tmp_path): + """The adversarial case the 'verification' exists for. + + Here the literal is patchable but `get_hp_config()` returns 300 regardless — a + second assignment wins. A verification that trusted re.subn's return value, or that + compared `target` against itself, would report a successful 40-lesson rehearsal + while the model trains 300 lessons. The mutation that did exactly that (replacing + the re-import with `lessons = target`) survived the first version of this file. + """ + sneaky = ( + "def get_hp_config():\n" + " d = {'total_lessons': 300}\n" + " d['total_lessons'] = 300\n" + " return d\n" + ) + model = _make_model(tmp_path, sneaky) + r = _run_cfgcheck(model, tmp_path / "out", rehearsal="40") + assert r.returncode != 0, ( + "the patch did not take — get_hp_config() still returns 300 — and the run " + f"proceeded anyway. It must refuse.\nstdout: {r.stdout}" + ) + assert "still reports total_lessons" in (r.stdout + r.stderr) + + def test_a_leftover_patch_stops_a_later_production_run(self, tmp_path): + """Section 2 does not re-clone when .git exists, so a production run on a pod that + has rehearsed reads the LEFTOVER patch. It must name that, not blame the committed + config and send the operator to edit the wrong file.""" + model = _make_model(tmp_path, PRODUCTION_CONFIG) + assert _run_cfgcheck(model, tmp_path / "o1", rehearsal="40").returncode == 0 + r = _run_cfgcheck(model, tmp_path / "o2") + assert r.returncode != 0, "a production run must refuse a dirty config" + assert "MODIFIED in this checkout" in (r.stdout + r.stderr), ( + "the refusal must name the leftover rehearsal patch. Blaming the committed " + "config is a correct refusal for the wrong reason." + ) + + def test_the_region_gate_still_fires(self, tmp_path): + """Not part of this audit's findings — pinned because the rehearsal work edited the + block this lives in, and it guards 'this would not be a global-land run'.""" + model = _make_model(tmp_path, PRODUCTION_CONFIG, region='"africa"') + r = _run_cfgcheck(model, tmp_path / "out") + assert r.returncode != 0 and "REGION" in (r.stdout + r.stderr) + + +class TestTheRehearsalFlagIsActuallyParsed: + """The shell argument parser, executed. + + An independent audit deleted the `--rehearsal)` case arm outright — so `--rehearsal` + fell through to `--*` and was rejected as an unknown option, removing H1's escape hatch + entirely — and every guard stayed green, because the string `--rehearsal` survived in the + header comment and in USAGE. Executing the config-check program does not cover this: the + flag never reaches Python if the shell refuses it first. + + These run the real script. Argument parsing precedes `mkdir -p "$OUT"`, so each case + below exits before the script touches the filesystem or spends anything. + """ + + @staticmethod + def _run(*args): + return subprocess.run( + ["bash", str(PODRUN), *args], + capture_output=True, text=True, cwd=str(REPO), timeout=60, + ) + + def test_the_flag_and_its_count_are_consumed_not_rejected(self): + """With the count consumed, the script must get as far as demanding a model name.""" + r = self._run("--rehearsal", "40") + combined = r.stdout + r.stderr + assert "unknown option" not in combined, ( + "--rehearsal was rejected as an unknown option, so the case arm parsing it is " + f"gone and a cheap run is unreachable again.\n{combined}" + ) + assert "usage" in combined.lower(), ( + f"expected the missing-model-name usage error once the flag was consumed.\n{combined}" + ) + + def test_a_count_is_required(self): + r = self._run("--rehearsal") + assert r.returncode != 0 + assert "lesson count" in (r.stdout + r.stderr), ( + "--rehearsal with no count must say so. Silently defaulting is how a rehearsal " + "becomes indistinguishable from a production run." + ) + + def test_a_non_integer_count_is_refused(self): + r = self._run("--rehearsal", "forty", "purple_alien") + assert r.returncode != 0 + assert "positive integer" in (r.stdout + r.stderr) + + def test_a_genuinely_unknown_option_is_still_refused(self): + """The control: the `--*` arm must keep working, or the test above proves nothing.""" + r = self._run("--bogus", "purple_alien") + assert r.returncode != 0 + assert "unknown option" in (r.stdout + r.stderr) + + +class TestTrackedConfigsDeclareTheProductionCount: + """A3. Calls get_hp_config() rather than matching the literal. + + pod_run_model.sh lines 173-176 refuse to pattern-match a config, citing #501 "the guard + that was not one — a substring assertion satisfied by a COMMENT recording the value's + history". The first version of this guard pattern-matched the config. A mutation that + appended `d['total_lessons'] = 40` before the return left the 300 literal untouched and + the guard green, while the pod would read 40. + """ + + @pytest.mark.parametrize("model", HYDRANETS) + def test_the_value_the_pod_will_read_is_a_production_count(self, model): + cfg_dir = REPO / "models" / model / "configs" + assert (cfg_dir / "config_hyperparameters.py").exists(), f"{model}: no config" + lessons = _config_lessons(cfg_dir.parent) + assert lessons >= 300, ( + f"{model}: get_hp_config() returns total_lessons={lessons}. A rehearsal must be " + "obtained with --rehearsal, which patches only the pod's clone — never by " + "committing a low count." + ) + + +class TestTheEnvironmentPinsAreCORRECTNotMerelyPRESENT: + """H2 (#516). Asserts the measurement, not a proxy for it. + + The first version checked that a pin EXISTED. `xarray>=2025.12,<2026` and + `pandas>=3.0,<4.0` — the exact versions measured as the defect — passed it, and its own + failure message ("this file resolves to pandas 3.0.6") was unreachable in the state that + resolves pandas 3.0.6. + + Measured 2026-09-29: xarray 2024.3.0 is the LAST release accepting pandas 1.x; 2024.5.0 + moved to pandas>=2.0 and 2024.9.0 to pandas>=2.1. So a `<2025` cap would look right and + be wrong — the cliff is inside the 2024 line. + """ + + # (package, must be allowed, must be refused, why the refused one matters) + CASES = ( + ("pandas", "1.5.3", "3.0.6", "the version a fresh resolve silently picked"), + ("pandas", "1.5.3", "2.1.0", "any pandas 2.x is a different major from the platform's"), + ("xarray", "2024.3.0", "2025.12.0", "the version a fresh resolve silently picked"), + ("xarray", "2024.3.0", "2024.11.0", "already requires pandas>=2.1 — inside the 2024 line"), + ("numpy", "1.26.4", "2.0.0", "the pandas wheel is built against the numpy 1.x C ABI"), + ) + + @staticmethod + def _spec(postprocessor: str, package: str) -> SpecifierSet: + text = (REPO / "postprocessors" / postprocessor / "requirements.txt").read_text() + found = [ + ln.strip() for ln in text.splitlines() + if re.match(rf"^\s*{package}\s*[><=!~]", ln) and ";" not in ln + ] + assert found, ( + f"{postprocessor}/requirements.txt declares no unconditional pin for {package}. " + "Unpinned, this file resolves pandas to 3.0.6 with no error. See #516. " + "(A marker-gated pin is excluded deliberately: `; python_version < \"3.10\"` is " + "inert on the 3.11 the runner builds, and looked identical to a real pin.)" + ) + assert len(found) == 1, f"{postprocessor}: {package} pinned {len(found)} times: {found}" + return SpecifierSet(found[0][len(package):].strip()) + + @pytest.mark.parametrize("postprocessor", ["un_fao", "un_crafd"]) + @pytest.mark.parametrize("package,allowed,refused,why", CASES) + def test_the_pin_admits_the_working_version_and_refuses_the_broken_one( + self, postprocessor, package, allowed, refused, why + ): + spec = self._spec(postprocessor, package) + assert spec.contains(allowed), ( + f"{postprocessor}: {package}{spec} excludes {allowed}, which is what every " + "successful run has used. This env would not build." + ) + assert not spec.contains(refused), ( + f"{postprocessor}: {package}{spec} ADMITS {refused} — {why}. The pin exists but " + "does not constrain what it was added to constrain." + ) + + @pytest.mark.parametrize("package", ["numpy", "pandas", "xarray"]) + def test_both_postprocessors_constrain_each_package_identically(self, package): + """C-116: one shared prefix, so the last postprocessor to run decides for both. + + Compares SPECIFIER SEMANTICS, not captured strings. The old string compare was + defeated by a marker suffix that made un_fao's pins inert while keeping the captured + text identical to un_crafd's real ones. + """ + fao, crafd = self._spec("un_fao", package), self._spec("un_crafd", package) + probes = ["1.5.3", "2.0.0", "2.1.0", "3.0.6", "1.26.4", "2024.3.0", "2024.11.0", "2025.12.0"] + differ = [v for v in probes if fao.contains(v) != crafd.contains(v)] + assert not differ, ( + f"un_fao pins {package}{fao} and un_crafd pins {package}{crafd}; they disagree " + f"on {differ}. Both install into envs/views-postprocessing (C-116), so whichever " + "runs last silently decides these versions for the other." + ) + + +class TestThePublishPathIsInstalledAndProven: + """H3 (#517). Asserted against comment-stripped source. + + Both of the first version's assertions here were satisfied by a five-line prose comment + that named `views-pipeline-core[appwrite]`: deleting the extra from the install line and + commenting out both imports left the suite green, with preflight printing "appwrite + client importable — the publish path exists" having imported nothing. + """ + + def test_the_appwrite_extra_is_in_an_install_command(self): + code = _code_only(PODRUN.read_text()) + install_lines = [ + ln for ln in code.splitlines() + if "pip install" in ln or (ln.strip().startswith('"') and "appwrite" in ln) + ] + assert any("views-pipeline-core[appwrite]" in ln for ln in install_lines), ( + "no install command requests the appwrite extra. Without it _build_datastore " + "raises at publish, AFTER the full training run — the 2026-09-29 failure (#517). " + f"install lines seen: {install_lines}" + ) + + def test_datafactory_is_floored_where_the_credential_fixes_landed(self): + """#509. The runner only ever executes on hardware we do not own, carrying a netrc + credential. Before views-datafactory 1.13.0 the client could carry that credential + across a redirect to another host and embed it in error messages. The model + requirements still say >=1.9.0 and a resolver will usually pick the newest — but + "the resolver will probably do the right thing" is the reasoning that put pandas + 3.0.6 into a fresh environment (#516).""" + # Comments are stripped from EVERY source, not just the shell one. un_fao's xarray + # note quotes the old `views-datafactory>=1.9.0` line as the state it is explaining, + # so a guard reading raw text either trips on prose or has to be careful about it — + # and "careful about it" is how the comment-satisfied guards got shipped. + sources = {"tools/podrun/pod_run_model.sh": _code_only(PODRUN.read_text())} + for pp in ("un_fao", "un_crafd"): + rel = f"postprocessors/{pp}/requirements.txt" + sources[rel] = _code_only((REPO / rel).read_text()) + for where, text in sources.items(): + floors = re.findall(r"^\s*(?:\S*\s+)?\"?views-datafactory>=(\d+)\.(\d+)", + text, re.M) + assert floors, f"{where} does not request views-datafactory at all" + for major, minor in floors: + assert (int(major), int(minor)) >= (1, 13), ( + f"{where} declares views-datafactory>={major}.{minor}; the " + "credential-handling fixes landed in 1.13.0 (#509). Every leg that runs " + "on rented hardware holds the netrc credential, not just the training one." + ) + + def test_every_datafactory_declaration_keeps_its_upper_bound(self): + """Closes the hole that the #509 deferral opens. + + `views-datafactory` is listed in test_requirements_hygiene.DEFERRED_PACKAGES so the + legs on rented hardware can be floored at >=1.13.0 while the 34 model declarations + stay at >=1.9.0. But that listing ALSO exempts the package from + test_no_dependency_is_declared_without_an_upper_bound, so while the deferral stands + nothing would notice `<2.0.0` being dropped — and an unbounded internal package + installs the next breaking major on the following monthly run. + + This is narrower than the rule it stands in for: it says nothing about floors, only + that a ceiling exists wherever this package is named. It outlives the deferral + harmlessly. + """ + import subprocess as sp + files = sp.run(["git", "ls-files", "*requirements.txt"], cwd=REPO, + capture_output=True, text=True, check=True).stdout.split() + unbounded = [] + for rel in files: + for n, raw in enumerate((REPO / rel).read_text().splitlines(), 1): + line = raw.strip() + if line.startswith("#") or not line.startswith("views-datafactory"): + continue + if not re.search(r"[<~=]", line.split("views-datafactory", 1)[1]): + unbounded.append(f"{rel}:{n}: {line}") + assert not unbounded, ( + "views-datafactory declared with no upper bound:\n " + "\n ".join(unbounded) + + "\nThe package is in DEFERRED_PACKAGES for #509, which exempts it from the " + "repo-wide ceiling rule, so this is the only guard watching." + ) + + def test_preflight_imports_the_client_not_just_mentions_it(self): + code = _code_only(PODRUN.read_text()) + verify = code.split("stage verify_env", 1)[-1].split("stage ", 1)[0] + assert re.search(r"^\s*import appwrite", verify, re.M) or re.search( + r"^\s*from views_pipeline_core\.modules\.appwrite import", verify, re.M + ), ( + "verify_env must IMPORT the Appwrite client, not merely mention it. The whole " + "script is written to fail early, and the one dependency that failed late was " + "the one not checked here." + ) + + def test_the_toolz_override_is_an_install_and_is_last(self): + code = _code_only(PODRUN.read_text()) + installs = [m.start() for m in re.finditer(r"pip install", code)] + overrides = [ + m.start() for m in re.finditer(r"pip install[^\n]*toolz>=0\.12", code) + ] + assert overrides, ( + "the C-151 toolz override is not an install command any more. A comment saying " + "the base image ships toolz>=0.12.1 satisfied the old guard while C-151 was " + "reintroduced; the resolver really does land on toolz 0.11.2 for this set." + ) + assert max(overrides) == max(installs), ( + "the toolz override must be the LAST pip install: anything after it re-resolves " + "the prefix and can pull toolz back under 0.12, which happened by hand on " + "2026-09-29 (1.1.0 -> 0.11.2) with no error." + ) + + +class TestARehearsalIsMarkedInTheOutput: + """The last line of defence, since nothing REFUSES to publish a rehearsal. + + Asserted against comment-stripped source. The old guards passed with the marker-writing + block deleted outright — `"$OUT/REHEARSAL"` was supplied by the `rm -f` on line 83 and + "NOT FIT TO DELIVER" by a MANIFEST line. + """ + + def test_the_marker_is_written_not_only_removed(self): + code = _code_only(PODRUN.read_text()) + writes = re.findall(r'>\s*"\$OUT/REHEARSAL"', code) + assert writes, ( + "nothing WRITES $OUT/REHEARSAL. A guard satisfied by the `rm -f` that clears it " + "is green in the state where no rehearsal is ever marked." + ) + + def test_the_marker_is_cleared_before_the_run_not_after(self): + code = _code_only(PODRUN.read_text()) + clear = code.index('rm -f "$OUT/REHEARSAL"') + assert clear < code.index("stage preflight"), ( + "the marker must be cleared with the other stale per-run state, before the run. " + "Moved to the end, a successful rehearsal deletes its own marker on the way out." + ) + + def test_the_forecast_leg_clears_stale_calibration_artefacts(self): + """Found reviewing this branch before merge. + + A previous calibration run on the same pod leaves parquet/ and draws/ in the same + output directory. The forecast leg writes no parquets, so without an explicit clear the + forecast MANIFEST sits beside 13 parquets from a different run type — and the guide's + rsync copies the directory, so they come home as this run's output. The script already + applies this reasoning twice (sections 4 and 5) for the case where the current run + DOES produce them; the case where it produces none is the same hazard. + """ + code = _code_only(PODRUN.read_text()) + leg = code[code.index('if [ "$FORECAST" = "1" ]; then\n echo'):] + leg = leg[: leg.index("stage manifest")] + assert 'rm -rf "$OUT/parquet"' in leg and '"$OUT/draws"' in leg, ( + "the forecast leg must clear the calibration artefacts it does not produce, or a " + "previous run's parquets travel home labelled as this forecast's output" + ) + + def test_both_pod_scripts_agree_on_where_the_workspace_is(self): + """The delivery script reads $ROOT/deliver//STATUS to decide whether a model may + be pooled. If one script honoured a relocated workspace and the other did not, it would + read a STATUS the other never wrote — and a missing STATUS is indistinguishable from a + model that failed.""" + fao = REPO / "tools" / "podrun" / "pod_run_fao_delivery.sh" + for path in (PODRUN, fao): + code = _code_only(path.read_text()) + assert re.search(r'ROOT="\$\{PODRUN_ROOT:-/workspace\}"', code), ( + f"{path.name} does not honour PODRUN_ROOT; the two scripts would disagree " + "about where deliver//STATUS lives" + ) + + def test_the_manifest_reports_the_mode_on_the_correct_branch(self): + """Swapping the two MANIFEST branches made production runs claim PATCHED and + rehearsals claim as-committed. The old guard checked only that the string existed.""" + code = _code_only(PODRUN.read_text()) + m = re.search( + r'if \[ -n "\$REHEARSAL_LESSONS" \]; then(.*?)else(.*?)fi', + code, re.S, + ) + assert m, "the MANIFEST mode branch is not in the expected shape; re-read it" + rehearsal_branch, production_branch = m.group(1), m.group(2) + assert "NOT FIT TO DELIVER" in rehearsal_branch and "PATCHED after checkout" in rehearsal_branch + assert "NOT FIT TO DELIVER" not in production_branch, ( + "the production branch of the MANIFEST claims the output is unfit to deliver" + ) + assert "PATCHED" not in production_branch, ( + "the production branch claims the config was patched after checkout" + ) + + +class TestTheGuideDocumentsTheChmodTrap: + """C-154 / #518. This property DID NOT EXIST when it was first asserted. + + The original assertion claimed the guide records the /workspace chmod trap and passed + because `/workspace` appears at line 104 and `chmod` at line 180, joined by `.*` under + re.S. An independent audit read the guide and found nothing: no "world-readable", no + "network filesystem", no C-154, no #518. The guard certified a safety property that was + absent, which is worse than having no guard, because it stopped the next reader looking. + + The warning has since been written. This asserts its substance on a bounded window, not + two words anywhere in a 400-line file. + """ + + def test_the_trap_is_documented_in_substance(self): + text = (REPO / "docs" / "runpod_run_guide.md").read_text() + assert re.search(r"C-154|#518", text), "the guide cites neither C-154 nor #518" + windows = [ + text[m.start(): m.start() + 700] + for m in re.finditer(r"chmod", text) + ] + assert any( + re.search(r"/workspace", w) + and re.search(r"ignore|silent|no effect|world-readable|666", w, re.I) + for w in windows + ), ( + "the guide must state, near a chmod instruction, that /workspace SILENTLY " + "ignores chmod and leaves the credential world-readable. Two words 76 lines " + "apart is what the deleted version of this guard accepted." + ) diff --git a/tests/test_fao_delivery_runner.py b/tests/test_fao_delivery_runner.py new file mode 100644 index 00000000..389da0ad --- /dev/null +++ b/tests/test_fao_delivery_runner.py @@ -0,0 +1,226 @@ +""" +Guards for `tools/podrun/pod_run_fao_delivery.sh` — the FAO delivery chain (views-models#499 +Track B), which until 2026-09-29 existed only as commands typed by hand on a rented pod. + +These are written after a /falsify guard-mode audit found that 10 of 14 guards for the +sibling script were DECORATIVE: they read the script as text, and the script's own comments +contained the strings they asserted, so deleting the code left the guard green. So here: + + * the argument parser is EXECUTED, not matched; + * `--preflight` is EXECUTED, which is the whole mode's value — it must refuse on a machine + that cannot deliver, and it must refuse for the right reason; + * text assertions read COMMENT-STRIPPED source, and are limited to ordering facts that + cannot be executed here (the steps themselves need a GPU, a store and a partner bucket). + +What these guards cannot cover, stated rather than implied: no test here runs a forecast, +publishes, or reaches the FAO. The first real exercise of this script is a rehearsal on a +pod, and that is what `--rehearsal` and `--preflight` exist to make cheap. +""" + +import os +import re +import subprocess +from pathlib import Path + +import pytest + +REPO = Path(__file__).resolve().parents[1] +FAO = REPO / "tools" / "podrun" / "pod_run_fao_delivery.sh" + +HYDRANETS = { + "purple_alien", "pink_pirate", "blue_stranger", "bright_starship", + "heavy_freighter", "blazing_meteor", "bold_comet", "violet_visitor", +} + + +def _code_only(src: str) -> str: + out = [] + for line in src.splitlines(): + stripped = re.sub(r"(?= 2, ( + f"preflight reported {out.count('MISSING:')} problems and stopped. It should " + "accumulate and report them together.\n" + out + ) + + def test_preflight_only_never_reaches_a_run_stage(self, result): + out = result.stdout + result.stderr + for forbidden in ("pool_and_publish", "un_fao_postprocessor", "forecast:"): + assert forbidden not in out, f"--preflight reached stage {forbidden}" + + +class TestTheChainIsInTheRightOrderWithTheRightFlags: + """Ordering facts, asserted on comment-stripped source because executing them needs a + GPU, a prediction store and a partner bucket.""" + + @pytest.fixture(scope="class") + def code(self): + return _code_only(FAO.read_text()) + + def test_the_ensemble_pools_from_saved_member_forecasts(self, code): + """Without -sa/--saved the ensemble refetches instead of pooling what the eight runs + just produced, and those GPU hours are wasted.""" + m = re.search(r"main\.py -r forecasting[^\n|]*", code) + assert m, "the ensemble invocation is not in the expected shape" + assert " -sa" in m.group(0), f"--saved is missing from: {m.group(0)}" + assert " -f" in m.group(0), f"--forecast is missing from: {m.group(0)}" + assert " -p" in m.group(0), f"--prediction_store is missing from: {m.group(0)}" + + def test_the_models_run_before_the_pool_which_runs_before_the_postprocessor(self, code): + forecast = code.index("--forecast") + pool = code.index("main.py -r forecasting") + postproc = code.index("postprocessors/un_fao/run.sh") + assert forecast < pool < postproc, ( + "the chain is out of order: the eight forecasts must precede the pool, and the " + "pool must precede the postprocessor that curates land -> land_gaul" + ) + + def test_the_whole_roster_is_named(self, code): + declared = set(re.search(r'MODELS="([^"]+)"', code).group(1).split()) + assert declared == HYDRANETS, ( + f"the roster here is {sorted(declared)}; rusty_bucket's members are " + f"{sorted(HYDRANETS)}. Pooling a partial roster changes the forecast silently." + ) + + def test_a_model_that_did_not_report_OK_stops_the_delivery(self, code): + assert 'STATUS" 2>/dev/null)" = "OK"' in code, ( + "the orchestrator must check each model's STATUS before pooling. An exit code " + "alone does not distinguish 'trained' from 'wrote a partial forecast'." + ) + + def test_wandb_is_forced_offline_on_the_ensemble_leg(self, code): + """Without this, main.py calls wandb.login() and blocks on a prompt no one is + watching. The first forecasting run on a pod died exactly there.""" + tail = code[code.index("pool_and_publish"):] + assert "WANDB_MODE=offline" in tail + + +class TestARehearsalIsMarkedEverywhereItCanBe: + """A rehearsal reaches the FAO shelf, because nothing downstream refuses one (#523). So + the marking is the only protection, and it has to be loud.""" + + @pytest.fixture(scope="class") + def code(self): + return _code_only(FAO.read_text()) + + def test_the_rehearsal_flag_reaches_the_per_model_runs(self, code): + assert "REH_ARGS" in code and "--rehearsal $REHEARSAL_LESSONS" in code, ( + "the lesson count must be forwarded to each model run, or the models train in " + "full while the delivery is labelled a rehearsal" + ) + + def test_the_marker_admits_the_forecasts_reached_the_shelf(self, code): + """The dangerous half. A reader who sees 'REHEARSAL' may assume it was contained; + it was not, and the file has to say so.""" + assert "ON THE FAO SHELF" in code, ( + "the rehearsal marker must state that the undertrained forecasts ARE published " + "and servable, not merely that the run was a rehearsal" + ) + + def test_the_marker_is_cleared_at_the_start_of_every_run(self, code): + clear = code.index('rm -f "$OUT/STATUS"') + assert "REHEARSAL" in code[clear:clear + 120], ( + "a production delivery must not inherit a previous rehearsal's marker, and far " + "worse, a rehearsal must not inherit a production run's absence of one" + ) + + def test_what_landed_is_read_back_by_name(self, code): + """`tools.liveness` answers 'is it there?' rather than 'did we send it?'. On + 2026-09-29 the publish reported success having written nothing (C-155).""" + assert "tools.liveness" in code diff --git a/tests/test_requirements_hygiene.py b/tests/test_requirements_hygiene.py index e8df5e19..b44f1672 100644 --- a/tests/test_requirements_hygiene.py +++ b/tests/test_requirements_hygiene.py @@ -70,7 +70,42 @@ # extra is what brings views-pipeline-core (>=3.0,<4) on 0.2.x, where it is optional — and # it resolves from PyPI beside pipeline-core 3.3.0 and viewser 6.6.4 (measured in a clean # venv; one model run end-to-end on the published wheel). The 2.x split is over. -DEFERRED_PACKAGES: dict[str, str] = {} +# views-datafactory is split DELIBERATELY and temporarily, 2026-09-30 (#509). +# +# The credential-handling fixes landed in 1.13.0: before it the client could carry a netrc +# credential across a redirect to another host, and could embed it in error messages. Every +# leg that runs on RENTED HARDWARE holds that credential, so the three that do are floored at +# 1.13.0 — both postprocessor requirement files and the pod install in +# tools/podrun/pod_run_model.sh (not a requirements file, so invisible to this rule). +# +# The other 34 declarations are model requirements at >=1.9.0. They are NOT floored here, for +# two reasons. They are unrelated to the delivery legs, so raising them belongs to #509 rather +# than to a delivery PR; and one of them is `models/violet_visitor/requirements.txt`, whose +# contents another session owns and which this session is instructed not to edit. +# +# The divergence is therefore real and is not decided by run order in the way C-116 warns +# about: the two postprocessors share one prefix and agree with each other, and the models +# resolve elsewhere. The resolver picks 1.13.0 for all of them today regardless — the floor +# only forbids something lower. +# +# COST OF THIS ENTRY, because it is wider than it looks: a DEFERRED_PACKAGES entry also +# exempts the package from test_no_dependency_is_declared_without_an_upper_bound, so nothing +# here would notice `<2.0.0` being dropped from a views-datafactory line while this entry +# stands. That hole is closed explicitly by +# tests/test_falsification_40_lesson_run_readiness.py::test_every_datafactory_declaration_ +# keeps_its_upper_bound, which is narrower than this rule and survives the deferral. +# +# REMOVE THIS ENTRY when #509 floors the remaining 34 — that is the whole of the trigger. +DEFERRED_PACKAGES: dict[str, str] = { + "views-datafactory": ( + "#509: the three legs that run on rented hardware are floored at >=1.13.0 for the " + "credential-handling fixes (both postprocessor requirement files and the pod install); " + "the 34 model declarations stay at >=1.9.0 because they are unrelated to the delivery " + "and one is violet_visitor, owned by another session. " + "TRIGGER: #509 raising the remaining 34 declarations to >=1.13.0 — at which point this " + "entry is deleted, not amended, because the divergence it describes no longer exists." + ), +} # pip accepts a bare VCS URL as a requirements.txt line; PEP 508 does not, because # such a line names no package. `apis/un_fao/requirements.txt` uses that form. It is diff --git a/tools/podrun/pod_run_fao_delivery.sh b/tools/podrun/pod_run_fao_delivery.sh new file mode 100755 index 00000000..54467cb0 --- /dev/null +++ b/tools/podrun/pod_run_fao_delivery.sh @@ -0,0 +1,254 @@ +#!/usr/bin/env bash +# pod_run_fao_delivery.sh [--rehearsal ] [--preflight] +# +# ┌──────────────────────────────────────────────────────────────────────────────────────┐ +# │ VERSION 0.1.0 — PROVISIONAL. First script for the FAO leg; no CI runs it. │ +# │ It PUBLISHES to a store a partner consumes. Read --preflight output before trusting │ +# │ it with GPU hours, and read the REHEARSAL note below before trusting it with FAO. │ +# └──────────────────────────────────────────────────────────────────────────────────────┘ +# +# Track B of views-models#499: the eight HydraNets forecast on global land, `rusty_bucket` +# pools them, the pooled forecast is published to the shelf, and the `un_fao` postprocessor +# curates `land` -> `land_gaul` and hands over to `unfao_bucket`. +# +# WHY THIS EXISTS. On 2026-09-29 every one of these steps was typed by hand on a rented pod. +# It worked, the pod was destroyed, and nothing recorded what had been done — so the next +# delivery started from the original broken state. That is the same defect a /falsify audit +# found three instances of in `pod_run_model.sh` (#516, #517, #523), and this is the fourth: +# the procedure existed only as commands in a terminal. `fimbulthul` has been unreachable +# since 2026-09-25, so a pod is not a stopgap here — it is the only hardware. +# +# THE ORDER MATTERS AND IS NOT OBVIOUS: +# 1. the 8 models -r forecasting -t -f (each ~2h at 300 lessons) +# 2. rusty_bucket -r forecasting -f -sa pooled from the SAVED member forecasts +# 3. publish -p wire shards to production_forecasts +# 4. un_fao postprocessors/un_fao/run.sh +# `-sa/--saved` at step 2 is required: without it the ensemble refetches instead of pooling +# what step 1 just produced. +# +# REHEARSAL. `--rehearsal ` patches only the pod's clone (never a tracked config), +# and every artefact it produces is MARKED unfit to deliver. But marking is all this script +# can do: nothing downstream REFUSES a marked rehearsal, because the refusal belongs in +# views-pipeline-core and is an ADR-013 contract change (views-models#523). A rehearsal +# therefore reaches the FAO shelf exactly like a real run. That is deliberate — it is the +# only way to test the chain — and it is why step 4 prints what it published by name. +# +# Progress: cat /workspace/deliver/_fao/STAGE · tail -f /workspace/deliver/_fao/run.log + +set -uo pipefail + +USAGE='usage: pod_run_fao_delivery.sh [--rehearsal ] [--preflight]' +REHEARSAL_LESSONS="" +PREFLIGHT_ONLY=0 +while [ $# -gt 0 ]; do + case "$1" in + --rehearsal) + [ $# -ge 2 ] || { echo "!!! --rehearsal needs a lesson count, e.g. --rehearsal 40" >&2 + echo "!!! $USAGE" >&2; exit 1; } + REHEARSAL_LESSONS="$2" + case "$REHEARSAL_LESSONS" in + ''|*[!0-9]*) echo "!!! --rehearsal needs a positive integer, got: $REHEARSAL_LESSONS" >&2 + exit 1 ;; + esac + [ "$REHEARSAL_LESSONS" -ge 1 ] || { echo "!!! --rehearsal 0 has nothing to rehearse" >&2; exit 1; } + shift 2 ;; + --preflight) PREFLIGHT_ONLY=1; shift ;; + --*) echo "!!! unknown option: $1" >&2; echo "!!! $USAGE" >&2; exit 1 ;; + *) echo "!!! unexpected argument: $1" >&2; echo "!!! $USAGE" >&2; exit 1 ;; + esac +done + +# PODRUN_ROOT exists so this script can be exercised off a pod — the tests run its parser and +# its preflight, and a hardcoded /workspace makes both untestable. It relocates the workspace +# and nothing else: every substantive check below (credentials, conda, region, GPU, disk) is +# unaffected by it, so an override cannot turn a refusal into a pass. +ROOT="${PODRUN_ROOT:-/workspace}" +REPO=$ROOT/views-models +VENV=$ROOT/venv +OUT=$ROOT/deliver/_fao +LOG=$OUT/run.log +ENSEMBLE=rusty_bucket +MODELS="violet_visitor bold_comet blazing_meteor blue_stranger heavy_freighter pink_pirate bright_starship purple_alien" + +# Checked, because an unchecked mkdir here fails off a pod and the script then runs on into +# a broken tee and a lock check that reports "another delivery is in progress" — a misleading +# message for a machine that simply has no /workspace. Found by the tests for this file. +mkdir -p "$OUT" 2>/dev/null || { + echo "!!! cannot create $OUT" >&2 + echo "!!! This script runs ON A POD, where $ROOT exists. To exercise it elsewhere, set" >&2 + echo "!!! PODRUN_ROOT=/some/writable/dir" >&2 + exit 1 +} +rm -f "$OUT/STATUS" "$OUT/REHEARSAL" "$OUT/PUBLISHED" +exec > >(tee -a "$LOG") 2>&1 + +stage() { echo "$1" > "$OUT/STAGE"; echo "=== [$(date +%H:%M:%S)] $1 ==="; } +die() { + echo "FAILED:$(cat "$OUT/STAGE" 2>/dev/null)" > "$OUT/STATUS" + printf '%s\n' "$1" > "$OUT/FAILURE" + echo "!!! $1" + exit 1 +} + +if ! mkdir "$OUT/.lock" 2>/dev/null; then + # Distinguished deliberately: "exists" and "could not be created" need different actions, + # and reporting the first for the second sends the operator looking for a run that never + # started. + if [ -d "$OUT/.lock" ]; then + echo "!!! $OUT/.lock exists — another FAO delivery is in progress on this pod." + echo "!!! If you are certain it is dead: rmdir $OUT/.lock" + else + echo "!!! could not create $OUT/.lock — $OUT is not writable." + fi + exit 1 +fi +trap 'rmdir "$OUT/.lock" 2>/dev/null' EXIT + +echo "### FAO delivery — started $(date -u +%Y-%m-%dT%H:%M:%SZ)" +[ -n "$REHEARSAL_LESSONS" ] && echo "### REHEARSAL at $REHEARSAL_LESSONS lessons — output will be marked unfit to deliver" + +# ── 0. preflight ────────────────────────────────────────────────────────────────────── +# Everything knowable before spending money, and the whole reason --preflight exists as a +# mode you can run on a fresh pod for seconds. At 300 lessons step 1 alone is ~16 GPU hours; +# discovering a missing credential or a missing conda after that is the failure this block is +# written to make impossible. Each check names what to do, not just what is wrong. +stage preflight +FAIL=0 +note() { echo " MISSING: $*"; FAIL=1; } + +[ -d "$REPO/.git" ] || note "$REPO is not a checkout — clone views-models first (see docs/runpod_run_guide.md Phase 2)" +[ -x "$VENV/bin/python" ] || note "$VENV does not exist — run pod_run_model.sh once to build it, or follow Phase 2" + +# The datafactory credential, for the fetch. +[ -s /root/.netrc ] || note "/root/.netrc — the datafactory fetch needs it (guide Step 2.3)" +if [ -s /root/.netrc ] && [ "$(stat -c %a /root/.netrc 2>/dev/null)" != "600" ]; then + chmod 600 /root/.netrc + echo " fixed: /root/.netrc was not 600" +fi + +# The publish credentials. Three of the nine are real secrets; the rest are identifiers. +# Checked HERE and not at first publish, because views-pipeline-core's own +# PredictionStoreConfig claims to read these "once at startup and fail loud" and does not — +# it is called from _build_datastore, i.e. after training (views-pipeline-core#557). Until +# that moves, this is the only check that happens before the money is spent. +for v in APPWRITE_ENDPOINT APPWRITE_DATASTORE_PROJECT_ID APPWRITE_DATASTORE_API_KEY; do + eval "val=\${$v:-}" + [ -n "$val" ] || note "\$$v — a publish secret; see reports/fao_delivery_runbook.md" +done +case "${APPWRITE_DATASTORE_API_KEY:-}" in + *[![:print:]]*) note "\$APPWRITE_DATASTORE_API_KEY contains a non-printable character — re-paste it" ;; +esac + +# conda, for the postprocessor leg ONLY. tools/launcher/postprocessor.sh uses +# `conda shell.bash hook` / `conda create --prefix` / `conda activate`, while this pod builds +# a uv venv — so the two legs need different interpreters and a pod can satisfy one and not +# the other. Without conda the delivery dies at step 4, after every GPU hour is spent. +command -v conda >/dev/null 2>&1 || note "conda — the un_fao postprocessor launcher requires it (tools/launcher/postprocessor.sh:72). A uv venv is not enough." + +# The coordinate registry, which the FAO queryset derives its region from. +if [ -x "$VENV/bin/python" ] && [ -d "$REPO" ]; then + REGION=$(cd "$REPO" && "$VENV/bin/python" -c \ + 'import sys; sys.path.insert(0,"."); from postprocessors.un_fao.configs.config_queryset import REGION; print(REGION)' 2>/dev/null) + if [ "$REGION" = "land_gaul" ]; then + echo " un_fao REGION resolves to land_gaul" + else + note "un_fao REGION resolved to '${REGION:-}', expected land_gaul — the FAO wire is disarmed (C-110)" + fi +fi + +nvidia-smi -L >/dev/null 2>&1 || note "no GPU visible" +AVAIL_GB=$(df -BG --output=avail "$ROOT" 2>/dev/null | tail -1 | tr -dc '0-9') +# Eight forecasting runs plus the pooled ensemble, on one disk. +[ "${AVAIL_GB:-0}" -ge 60 ] || note "only ${AVAIL_GB:-0}GB free on $ROOT; eight forecasts plus the pool need ~60GB" + +if [ "$FAIL" = "1" ]; then + echo + die "preflight failed — nothing above costs GPU time to fix. Fix them and re-run --preflight." +fi +echo "preflight OK — GPU visible, ${AVAIL_GB}GB free, credentials present, conda present, wire armed" + +if [ "$PREFLIGHT_ONLY" = "1" ]; then + echo OK > "$OUT/STATUS" + stage preflight_only_done + echo "### --preflight only: nothing was run, nothing was published." + echo "### Re-run without --preflight to start the delivery." + exit 0 +fi + +# ── 1. the eight, forecasting ───────────────────────────────────────────────────────── +REH_ARGS="" +[ -n "$REHEARSAL_LESSONS" ] && REH_ARGS="--rehearsal $REHEARSAL_LESSONS" +for M in $MODELS; do + stage "forecast:$M" + # Delegated rather than reimplemented: pod_run_model.sh already carries the config + # patch-and-verify, the appwrite check and the C-151 override, and a second copy of + # those is a second place for them to rot. + bash "$REPO/tools/podrun/pod_run_model.sh" $REH_ARGS --forecast "$M" \ + || die "forecast failed for $M — see /workspace/deliver/$M/FAILURE" + [ "$(cat "$ROOT/deliver/$M/STATUS" 2>/dev/null)" = "OK" ] \ + || die "$M did not report OK — refusing to pool a partial roster" +done + +# ── 2-3. pool and publish ───────────────────────────────────────────────────────────── +stage pool_and_publish +cd "$REPO/ensembles/$ENSEMBLE" || die "cannot enter $ENSEMBLE" +export WANDB_MODE=offline WANDB_SILENT=true +# -sa/--saved pools the member forecasts just written. Without it the ensemble refetches and +# the eight runs above are wasted. -p publishes the wire shards (ADR-013). +"$VENV/bin/python" main.py -r forecasting -f -sa -p || die "$ENSEMBLE pool-and-publish exited non-zero" + +# ── 4. the FAO postprocessor ────────────────────────────────────────────────────────── +stage un_fao_postprocessor +cd "$REPO" || die "cannot enter repo" +# A refusal naming DeliveryNotFindableError is views-postprocessing 1.4.0 WORKING: that build +# verifies a delivery by what it refuses, and the message says which case it hit. Do not read +# it as this script failing. +bash postprocessors/un_fao/run.sh || die "the un_fao postprocessor exited non-zero — read the message before assuming the worst; DeliveryNotFindableError means 1.4.0's findability guard fired and it names every object it checked" + +# ── 5. what actually landed ─────────────────────────────────────────────────────────── +stage report +# By name, from the live surface, not inferred from an exit code. `python -m tools.liveness` +# is the six-surface dashboard; it answers "is it there?" rather than "did we send it?". +"$VENV/bin/python" -m tools.liveness > "$OUT/PUBLISHED" 2>&1 \ + || echo "(liveness reported non-zero — read $OUT/PUBLISHED)" >> "$OUT/PUBLISHED" +tail -40 "$OUT/PUBLISHED" + +stage manifest +{ + echo "delivery: un_fao via $ENSEMBLE" + echo "finished: $(date -u +%Y-%m-%dT%H:%M:%SZ)" + echo "models: $MODELS" + echo "git: $(git -C "$REPO" rev-parse --short HEAD)" + if [ -n "$REHEARSAL_LESSONS" ]; then + echo "mode: REHEARSAL — NOT FIT TO DELIVER" + echo "lessons: $REHEARSAL_LESSONS (pod clones patched; tracked configs untouched)" + else + echo "mode: production" + echo "lessons: as committed at the git sha above" + fi +} > "$OUT/MANIFEST" +cat "$OUT/MANIFEST" + +if [ -n "$REHEARSAL_LESSONS" ]; then + { + echo "THIS DELIVERY IS A REHEARSAL. THE FORECASTS ON THE FAO SHELF ARE UNDERTRAINED." + echo + echo "lessons: $REHEARSAL_LESSONS" + echo "finished: $(date -u +%Y-%m-%dT%H:%M:%SZ)" + echo + echo "It ran the whole chain to prove the hops connect. Nothing downstream refuses a" + echo "marked rehearsal (views-models#523), so these forecasts ARE on the shelf and the" + echo "FAO API can serve them. They are structurally indistinguishable from real ones." + echo "Supersede or remove them before anyone reads them as a forecast." + } > "$OUT/REHEARSAL" + echo + echo "############################################################################" + echo "### REHEARSAL COMPLETE — undertrained forecasts are NOW ON THE FAO SHELF." + echo "### They are indistinguishable from real ones. Supersede them." + echo "### $OUT/REHEARSAL" + echo "############################################################################" +fi + +echo OK > "$OUT/STATUS" +stage done +echo "### FAO delivery — COMPLETE" diff --git a/tools/podrun/pod_run_model.sh b/tools/podrun/pod_run_model.sh index ccd63040..9dc71f4f 100755 --- a/tools/podrun/pod_run_model.sh +++ b/tools/podrun/pod_run_model.sh @@ -1,5 +1,5 @@ #!/usr/bin/env bash -# pod_run_model.sh +# pod_run_model.sh [--rehearsal ] [--forecast] # # ┌──────────────────────────────────────────────────────────────────────────────────────┐ # │ VERSION 0.1.0 — PROVISIONAL. NOT part of the monthly production run. │ @@ -21,13 +21,68 @@ # /workspace/deliver//STATUS OK or FAILED: # /workspace/deliver//FAILURE the reason, on failure only (written directly) # /workspace/deliver//run.log the whole transcript +# /workspace/deliver//REHEARSAL present ONLY on a --rehearsal run +# +# TWO MODES. +# (default) production. total_lessons must be >= 300, and the run refuses otherwise. +# --rehearsal exercises the whole chain on a deliberately undertrained model at +# lessons. Patches the POD's clone of the config — never a tracked file +# — and marks the output as unfit to deliver. +# +# The lesson floor exists because a 300-lesson budget can be spent by ACCIDENT — a config +# left at a sweep value, a stale checkout. It was never meant to forbid a deliberate cheap +# end-to-end test, which is a routine and necessary thing to want, especially after a run +# that failed late. Before --rehearsal existed the only way to get one was to edit the guard +# out, which produced output indistinguishable from a real run: the cheap test and the +# accident looked the same on disk. That is the actual hazard, and it is what REHEARSAL and +# the MANIFEST `mode:` line address. Wasted GPU time is recoverable; an undertrained +# forecast reaching a partner as a real one is not. # # Progress is readable from outside at any time: cat /workspace/deliver//STAGE set -uo pipefail -MODEL="${1:?usage: pod_run_model.sh }" -ROOT=/workspace +# --rehearsal takes the lesson count rather than reading it from the config, so a rehearsal +# needs NO edit to a tracked file. main.py has no hyperparameter override, so the count has +# to reach the model through its config; this script patches the POD's ephemeral clone and +# verifies the patch (see the config check). The count is REQUIRED — there is exactly one way +# to ask for a rehearsal, and it states the number out loud in the command that starts it. +USAGE='usage: pod_run_model.sh [--rehearsal ] [--forecast] ' +REHEARSAL_LESSONS="" +# --forecast selects the FAO leg: train on the forecasting partition and PRODUCE a forecast +# (`-r forecasting -t -f`) instead of train-and-evaluate on calibration. The two legs want +# different deliverables, so sections 4 and 5 — the 13-origin collapse and the posterior +# archive — are CALIBRATION-only and are skipped. A forecast has one origin, and what the +# FAO chain consumes is the pooled ensemble output, not per-model parquets. +# +# Kept in this script rather than a copy of it because everything before section 3 is +# identical and already exercised: the preflight, the venv, the appwrite check, the config +# patch-and-verify. A second script would have been a second place for C-151 to rot. +FORECAST=0 +while [ $# -gt 0 ]; do + case "$1" in + --rehearsal) + [ $# -ge 2 ] || { echo "!!! --rehearsal needs a lesson count, e.g. --rehearsal 40" >&2 + echo "!!! $USAGE" >&2; exit 1; } + REHEARSAL_LESSONS="$2" + case "$REHEARSAL_LESSONS" in + ''|*[!0-9]*) echo "!!! --rehearsal needs a positive integer, got: $REHEARSAL_LESSONS" >&2 + exit 1 ;; + esac + [ "$REHEARSAL_LESSONS" -ge 1 ] || { echo "!!! --rehearsal 0 has nothing to rehearse" >&2; exit 1; } + shift 2 ;; + --forecast) FORECAST=1; shift ;; + --*) echo "!!! unknown option: $1" >&2; echo "!!! $USAGE" >&2; exit 1 ;; + *) break ;; + esac +done +MODEL="${1:?$USAGE}" +# Honours PODRUN_ROOT for the same reason pod_run_fao_delivery.sh does, and — more +# importantly — so the two AGREE. The delivery script reads $ROOT/deliver//STATUS to +# decide whether a model may be pooled; if it relocated its workspace and this script did not, +# it would read a STATUS this script never wrote. On a pod both are /workspace and nothing +# changes. +ROOT="${PODRUN_ROOT:-/workspace}" REPO=$ROOT/views-models VENV=$ROOT/venv OUT=$ROOT/deliver/$MODEL @@ -38,6 +93,10 @@ mkdir -p "$OUT" # or `cat STATUS` reports that old failure for the whole of this run -- and the header # advertises STATUS/STAGE as the way to watch from outside, so a stale one actively misleads. rm -f "$OUT/STATUS" +# Same reasoning for the rehearsal marker: a production run in a directory left behind by an +# earlier rehearsal must not inherit its "undeliverable" mark, and — far worse — a rehearsal +# must not inherit a previous production run's ABSENCE of one. +rm -f "$OUT/REHEARSAL" "$OUT/.lessons" exec > >(tee -a "$LOG") 2>&1 stage() { echo "$1" > "$OUT/STAGE"; echo "=== [$(date +%H:%M:%S)] $1 ==="; } @@ -76,10 +135,22 @@ if [ ! -x "$VENV/bin/python" ]; then stage install_python uv venv --python 3.11 "$VENV" || die "venv creation failed" + # views-pipeline-core[appwrite] is requested EXPLICITLY, with no version, so + # views-hydranet's own range still decides which pipeline-core is installed. The extra is + # what carries the Appwrite SDK, and without it `_build_datastore` raises at publish time + # — which is AFTER the full training run. That is exactly how the first FAO delivery + # attempt failed on 2026-09-29 (views-models#517); it was fixed by hand on a pod that no + # longer exists, so the repository never learned it. uv pip install --python "$VENV/bin/python" \ - "views-hydranet~=0.1.1" "views-datafactory>=1.9.0,<2.0.0" || die "pip install failed" + "views-hydranet~=0.1.1" "views-datafactory>=1.13.0,<2.0.0" \ + "views-pipeline-core[appwrite]" || die "pip install failed" # register C-151: viewser pins toolz<0.12, which cannot import tlz submodules on # Python 3.11 and breaks EVERY datafactory fetch. Override after resolution. + # + # This MUST stay the last install in this block. Any pip install appended below it + # re-resolves this prefix and can silently pull toolz back under 0.12 — which happened on + # 2026-09-29, when installing the appwrite extra by hand reverted it 1.1.0 -> 0.11.2 with + # no error. Pinned by tests/test_falsification_40_lesson_run_readiness.py. uv pip install --python "$VENV/bin/python" "toolz>=0.12.1" || die "toolz override failed" else echo "venv already present — reusing" @@ -90,6 +161,18 @@ stage verify_env import torch, tlz.curried, views_hydranet, views_pipeline_core, datafactory_query assert torch.cuda.is_available(), "CUDA not available" print("torch", torch.__version__, "cap", torch.cuda.get_device_capability()) + +# The publish path is verified HERE, in preflight, not discovered at publish time. Without +# the appwrite extra this import is the only thing between a green-looking pod and a run +# that trains for hours and then cannot hand over its forecasts (#517). Importing the +# client is a weaker check than publishing, but it is the strongest one available before +# there is anything to publish — and it is the check whose absence cost the first delivery. +import appwrite # noqa: F401 — the SDK itself; views-pipeline-core[appwrite] provides it +from views_pipeline_core.modules.appwrite import file as _appwrite_file # noqa: F401 +print("appwrite client importable — the publish path exists") + +import pandas, numpy +print("pandas", pandas.__version__, "numpy", numpy.__version__) PY # ── 2. repo ─────────────────────────────────────────────────────────────────────── @@ -108,8 +191,12 @@ stage check_config # the same defect this repo has already shipped once (#501, "the guard that was not one"): a # substring assertion satisfied by a COMMENT recording the value's history, so the guard passed # on the wrong region. A comment cannot satisfy this one. +# REHEARSAL/OUT/MODEL reach the script through the environment: the heredoc is quoted, so +# the shell does not interpolate into it, and that is deliberate — the config values must +# come from the config file, not from string substitution. +REHEARSAL_LESSONS="$REHEARSAL_LESSONS" OUT="$OUT" MODEL="$MODEL" \ "$VENV/bin/python" - "$REPO/models/$MODEL" <<'CFGCHECK' || die "config check failed" -import importlib.util, sys +import importlib, importlib.util, os, re, subprocess, sys from pathlib import Path model = Path(sys.argv[1]) @@ -121,11 +208,73 @@ def load(name): spec.loader.exec_module(mod) return mod -hp = load("config_hyperparameters").get_hp_config() -lessons = hp["total_lessons"] +cfg_path = model / "configs" / "config_hyperparameters.py" +requested = os.environ.get("REHEARSAL_LESSONS") or "" + +lessons = load("config_hyperparameters").get_hp_config()["total_lessons"] print("total_lessons:", lessons, "(read from the config, not the file text)") -if lessons < 300: - sys.exit("total_lessons is %s, expected >= 300 - this pod would train a throwaway model" % lessons) + +if not requested: + # A rehearsal patches this pod's clone, and section 2 does NOT re-clone when .git already + # exists — so a production run started on a pod that has rehearsed reads the LEFTOVER + # patch. It would refuse (the floor holds), but it would blame the committed config and + # tell the operator to use --rehearsal, which is the opposite of what they want. Name the + # real cause instead. Checked before the floor so the accurate message wins. + dirty = subprocess.run( + ["git", "-C", str(model), "status", "--porcelain", "--", str(cfg_path)], + capture_output=True, text=True, + ) + if dirty.returncode == 0 and dirty.stdout.strip(): + sys.exit( + "%s is MODIFIED in this checkout, so total_lessons=%s is not what the committed\n" + " config says. This pod has almost certainly run --rehearsal already, and that\n" + " patch is still in place. A production run must start from the committed config:\n" + " git -C %s checkout -- %s\n" + " Refusing rather than training on a config neither of us chose." + % (cfg_path, lessons, model, cfg_path.relative_to(model)) + ) + +if requested: + # Patch the POD's clone. This is a throwaway checkout on rented hardware; the tracked + # config keeps saying 300, which is the production truth and must not be edited to get a + # cheap test. Substitution is anchored on the same literal the file is known to contain. + target = int(requested) + text = cfg_path.read_text() + patched, n = re.subn(r"('total_lessons'\s*:\s*)\d+", r"\g<1>%d" % target, text) + if n != 1: + sys.exit( + "--rehearsal %d: expected exactly one 'total_lessons': in %s, found %d.\n" + " Refusing rather than guessing which one to patch." + % (target, cfg_path, n) + ) + cfg_path.write_text(patched) + + # VERIFY by re-importing, not by trusting the substitution. A patch that silently failed + # would otherwise produce a 300-lesson run wearing a rehearsal label, or the reverse. + importlib.invalidate_caches() + lessons = load("config_hyperparameters").get_hp_config()["total_lessons"] + if lessons != target: + sys.exit( + "--rehearsal %d: patched %s but it still reports total_lessons=%s. Not proceeding." + % (target, cfg_path, lessons) + ) + print("REHEARSAL: patched the pod's config to %d lessons and re-read it to confirm." % lessons) + print(" Output will be marked NOT FIT TO DELIVER.") + +# Recorded for the MANIFEST, so the manifest reports the value the run was GATED on rather +# than re-deriving it by grepping the file text. Two readings of one number can disagree. +Path(os.environ["OUT"], ".lessons").write_text(str(lessons)) + +if not requested and lessons < 300: + sys.exit( + "total_lessons is %s, expected >= 300 - this pod would train a throwaway model.\n" + " If you MEANT a cheap end-to-end test, that is what --rehearsal is for:\n" + " pod_run_model.sh --rehearsal %s %s\n" + " It takes the lesson count on the command line, patches only the pod's clone, and\n" + " marks the output as unfit to deliver — so a deliberate cheap run cannot be\n" + " confused with an accidental one, and no tracked config has to be edited." + % (lessons, lessons, os.environ.get("MODEL", "")) + ) region = getattr(load("config_queryset"), "REGION", None) print("REGION:", region) @@ -134,14 +283,75 @@ if region != "land": CFGCHECK # ── 3. the run ──────────────────────────────────────────────────────────────────── -stage train_and_evaluate +if [ "$FORECAST" = "1" ]; then + stage train_and_forecast +else + stage train_and_evaluate +fi cd "$REPO/models/$MODEL" || die "cannot enter model dir" +# WANDB_MODE=offline is NOT optional. Without it main.py calls wandb.login(), which blocks +# on an interactive prompt no one is watching, and the first forecasting run on a pod died +# there after the environment was already built. export WANDB_MODE=offline WANDB_SILENT=true START=$(date +%s) -"$VENV/bin/python" main.py -r calibration -t -e || die "main.py exited non-zero" +if [ "$FORECAST" = "1" ]; then + "$VENV/bin/python" main.py -r forecasting -t -f || die "main.py exited non-zero" +else + "$VENV/bin/python" main.py -r calibration -t -e || die "main.py exited non-zero" +fi echo "run took $(( ($(date +%s) - START) / 60 )) minutes" -# ── 4. collapse to the deliverable ──────────────────────────────────────────────── +# ── 4-5. the CALIBRATION deliverable (13-origin parquets + posterior archive) ───── +# Skipped on the forecast leg: a forecast has one origin, so the 13-parquet count check +# would refuse it, and the FAO chain consumes the pooled ensemble output rather than these. +if [ "$FORECAST" = "1" ]; then + echo "### forecast leg — skipping the calibration collapse and posterior archive" + # But CLEAR them, rather than merely not writing them. A previous calibration run on this pod + # leaves parquet/ and draws/ in this same directory, and the forecast MANIFEST written below + # would then sit beside 13 parquets belonging to a different run type. The rsync in the guide + # copies the directory, so they would come home as this run's output. This is the same + # reasoning as the `rm -rf` guards in sections 4 and 5 — stale artefacts must not be + # countable as the current run's — applied to the case where the current run produces none. + rm -rf "$OUT/parquet" "$OUT/draws" + stage manifest + { + echo "model: $MODEL" + echo "finished: $(date -u +%Y-%m-%dT%H:%M:%SZ)" + echo "run_type: forecasting" + echo "git: $(git -C "$REPO" rev-parse --short HEAD)" + echo "lessons: $(cat "$OUT/.lessons" 2>/dev/null || echo unknown)" + if [ -n "$REHEARSAL_LESSONS" ]; then + echo "mode: REHEARSAL — NOT FIT TO DELIVER" + echo "config: PATCHED after checkout — total_lessons forced to $REHEARSAL_LESSONS" + else + echo "mode: production" + echo "config: as committed at the git sha above" + fi + echo "forecast: under $REPO/models/$MODEL/data/generated/ (pooled by rusty_bucket)" + } > "$OUT/MANIFEST" + cat "$OUT/MANIFEST" + if [ -n "$REHEARSAL_LESSONS" ]; then + { + echo "THIS OUTPUT IS A REHEARSAL. DO NOT DELIVER IT." + echo + echo "model: $MODEL" + echo "lessons: $(cat "$OUT/.lessons" 2>/dev/null || echo unknown)" + echo "run_type: forecasting" + echo "finished: $(date -u +%Y-%m-%dT%H:%M:%SZ)" + echo + echo "Produced with --rehearsal to exercise the delivery chain end to end. The model is" + echo "deliberately undertrained. The forecast is well-formed and will pass every" + echo "structural check downstream — that is precisely why this file exists. Nothing in" + echo "the data itself will tell you." + } > "$OUT/REHEARSAL" + echo "### WROTE $OUT/REHEARSAL — this forecast is NOT fit to deliver" + fi + echo OK > "$OUT/STATUS" + stage done + echo "### $MODEL — COMPLETE (forecast leg)" + exit 0 +fi + stage collapse # Clear a previous attempt first. Parquets are named from the SOURCE run's timestamp, so an # old set and a new set can coexist; if they happened to sum to 13 the count check below @@ -210,10 +420,50 @@ stage manifest echo "compressed: $(du -sh "$OUT/draws" | cut -f1)" echo "parquets: $(du -sh "$OUT/parquet" | cut -f1)" echo "git: $(git -C "$REPO" rev-parse --short HEAD)" - echo "lessons: $(grep -oE "'total_lessons': [0-9]+" "$REPO/models/$MODEL/configs/config_hyperparameters.py" | head -1)" + # The value the run was GATED on, written by the config check — not a second, independent + # grep of the file text, which could disagree with it and be believed. + echo "lessons: $(cat "$OUT/.lessons" 2>/dev/null || echo unknown)" + if [ -n "$REHEARSAL_LESSONS" ]; then + echo "mode: REHEARSAL — NOT FIT TO DELIVER" + # Stated explicitly because the `git:` line above no longer fully describes the run: the + # pod's config_hyperparameters.py was patched after checkout, so that sha alone would + # imply 300 lessons. A manifest that has to be cross-read with a flag is a manifest that + # will be misread. + echo "config: PATCHED after checkout — total_lessons forced to $REHEARSAL_LESSONS" + else + echo "mode: production" + echo "config: as committed at the git sha above" + fi } > "$OUT/MANIFEST" cat "$OUT/MANIFEST" +# ── the mark that makes a rehearsal unmistakable downstream ─────────────────────── +# A rehearsal's parquets are structurally identical to a production run's: same columns, +# same row counts, same names, same finite non-negative values. Every check in section 4 +# passes. Nothing about the FILES says the model behind them is undertrained, which is why +# this has to be a separate artefact that travels with them. +# +# This MARKS; it cannot REFUSE. The publish step is not in this script (the header is +# accurate: this runner uploads nothing), so the refusal has to live wherever the forecast +# is handed to a store. Until it does, this file is the only thing standing between a +# rehearsal and a partner, and that is a weaker guarantee than it should be — tracked as +# the second half of the rehearsal work, not as done. +if [ -n "$REHEARSAL_LESSONS" ]; then + { + echo "THIS OUTPUT IS A REHEARSAL. DO NOT DELIVER IT." + echo + echo "model: $MODEL" + echo "lessons: $(cat "$OUT/.lessons" 2>/dev/null || echo unknown)" + echo "finished: $(date -u +%Y-%m-%dT%H:%M:%SZ)" + echo + echo "It was produced with --rehearsal to exercise the pipeline end to end. The model is" + echo "deliberately undertrained. The parquets are well-formed and will pass every" + echo "structural check, including this runner's own — that is precisely why this file" + echo "exists. Nothing in the data itself will tell you." + } > "$OUT/REHEARSAL" + echo "### WROTE $OUT/REHEARSAL — this output is NOT fit to deliver" +fi + echo OK > "$OUT/STATUS" stage done echo "### $MODEL — COMPLETE"