fix(#516,#517,#509,#518,#523): a fresh pod can reproduce the run that delivered — and a cheap rehearsal that cannot be mistaken for one

docs/runpod_run_guide.md

@@ -180,6 +180,22 @@ grep -A2 '<data server IP>' ~/.netrc | ssh root@<ip> -p <port> -i ~/.ssh/id_ed25
     'umask 077; cat > /root/.netrc; chmod 600 /root/.netrc'
 ```
 
+> **Put credentials on `/root`, never on `/workspace` — `/workspace` silently ignores
+> `chmod`.**
+>
+> It is a network filesystem. `chmod 600` there **returns success and does nothing**: the
+> file stays mode `666`, readable by every process on the machine, and there is no error to
+> notice. `stat -c %a` is the only way to find out, and only if you think to look.
+>
+> Found on 2026-09-29 while placing the Appwrite publish credentials, which sat
+> world-readable on rented hardware until they were moved to `/root/.secrets`. Registered as
+> **views-models C-154**; see **#518**.
+>
+> This is why the command above writes to `/root/.netrc` and why the runner checks
+> `stat -c %a /root/.netrc` rather than trusting its own `chmod`. Both are deliberate. The
+> `umask 077` is what actually protects the file in transit — the `chmod` is a belt-and-braces
+> check that happens to work *here* because `/root` is local disk.
+
 Check it before trusting it — the pipe can silently deliver nothing:
 
 ```bash
@@ -269,7 +285,38 @@ nohup setsid bash tools/podrun/pod_run_model.sh <model> \
 `setsid` matters: the run must survive your SSH session closing and your laptop sleeping.
 
 It refuses before spending GPU time if `.netrc` is missing, if `total_lessons` is still a
-throwaway value, if `REGION` is not `"land"`, or if the converter is absent.
+throwaway value, if `REGION` is not `"land"`, if the Appwrite client is not importable, or
+if the converter is absent.
+
+### Rehearsing the chain first
+
+After a run that failed late, you usually want the whole chain exercised cheaply before you
+commit to a full one. That is `--rehearsal`, and it takes the lesson count:
+
+```bash
+nohup setsid bash tools/podrun/pod_run_model.sh --rehearsal 40 <model> \
+    > /workspace/<model>.nohup 2>&1 < /dev/null &
+```
+
+It patches **the pod's clone** of `config_hyperparameters.py` to 40 lessons, re-reads the
+config to confirm the patch took, and marks the output. Nothing tracked in git is edited —
+the committed configs stay at their production value, which is the point: a rehearsal
+obtained by committing a low lesson count is the failure mode this flag removes.
+
+A rehearsal leaves an extra file, **`REHEARSAL`**, beside the parquets, and `MANIFEST` gains
+`mode: REHEARSAL — NOT FIT TO DELIVER`. You need them, because the parquets themselves are
+indistinguishable from a real run's: same columns, same row counts, all finite and
+non-negative, every structural check green. Nothing in the data will tell you the model
+behind it is undertrained.
+
+**What a rehearsal does NOT prove.** It exercises the chain, not the capacity. A 40-lesson
+run has a different duration and memory profile from a 300-lesson one, and memory is where
+this platform has failed before. A green rehearsal means the hops connect; it does not mean
+the full run will fit.
+
+**It marks, it does not refuse.** This runner uploads nothing, so it cannot stop a rehearsal
+being published downstream — the `REHEARSAL` file is a warning to a person, not a guard. Do
+not publish a directory that contains one.
 
 Monitor without disturbing it:
 
@@ -296,8 +343,9 @@ The runner already does both. Its output is in `/workspace/deliver/<model>/`:
 |---|---|
 | `parquet/` | 13 files, one per origin, 2,333,448 rows each |
 | `draws/` | the full posterior, zstd — ~236× smaller than raw |
-| `MANIFEST` | sizes, timestamps, git sha, lesson count |
+| `MANIFEST` | sizes, timestamps, git sha, lesson count, and `mode:` |
 | `STATUS` | `OK`, or `FAILED:<stage>` |
+| `REHEARSAL` | present **only** after `--rehearsal` — this output is not fit to deliver |
 
 From your laptop:
 
@@ -323,6 +371,48 @@ stopped looking like conflict.
 
 ---
 
+## Phase 4b — The FAO delivery (Track B)
+
+Phases 4 and 5 are **Track A**: calibration predictions for research. The FAO delivery is a
+different chain and a different script.
+
+```bash
+cd /workspace/views-models
+
+# Seconds, costs nothing, checks everything that can be known before GPU time:
+bash tools/podrun/pod_run_fao_delivery.sh --preflight
+
+# Then, once preflight is clean:
+nohup setsid bash tools/podrun/pod_run_fao_delivery.sh --rehearsal 40 \
+    > /workspace/fao.nohup 2>&1 < /dev/null &
+```
+
+Drop `--rehearsal 40` for a production delivery.
+
+It runs: the eight HydraNets on the forecasting partition → `rusty_bucket` pools them from
+the saved member forecasts → publish → the `un_fao` postprocessor → a read-back of what
+actually landed via `python -m tools.liveness`.
+
+Watch it with `cat /workspace/deliver/_fao/STAGE` or `tail -f /workspace/deliver/_fao/run.log`.
+
+**Run `--preflight` on every fresh pod.** At 300 lessons the eight forecasts alone are ~16 GPU
+hours, and the two things most likely to stop the delivery are invisible until the end:
+the three Appwrite publish secrets, and **`conda`** — the postprocessor launcher requires it
+while this pod builds a `uv` venv, so a pod can satisfy the training leg and not the delivery
+leg. That failure lands *after* all the training.
+
+### Reading the outcome
+
+- **`DeliveryNotFindableError`** is views-postprocessing 1.4.0 **working**. That build verifies
+  a delivery by what it refuses, and the message names every object it checked. Do not read it
+  as the script failing.
+- A **rehearsal's forecasts reach the FAO shelf and are servable.** Nothing downstream refuses
+  a marked rehearsal (#523), and they are structurally indistinguishable from real forecasts —
+  same columns, same coverage, all finite. `/workspace/deliver/_fao/REHEARSAL` says so. **Do
+  not leave them there**: supersede or remove them before anyone reads them as a forecast.
+- A rehearsal proves the **chain**, not the **capacity**. 40 lessons has a different duration
+  and memory profile from 300, and memory is where this platform has failed before.
+
 ## Phase 6 — Teardown
 
 Confirm `STATUS` is `OK` and the files are on your laptop, then **terminate** the pod — not stop

postprocessors/un_crafd/requirements.txt

@@ -1,5 +1,16 @@
-views-datafactory>=1.9.0,<2.0.0
+# views-datafactory floored at 1.13.0 — see the note in un_fao/requirements.txt. Declared
+# identically here for the same C-116 reason as the pins below: one shared prefix, so whichever
+# postprocessor runs last decides the version for both, and flooring one is flooring neither.
+views-datafactory>=1.13.0,<2.0.0
 # numpy stays on 1.x — see the note in un_fao/requirements.txt. Both postprocessors
 # install into the same envs/views-postprocessing prefix (C-116), so this pin has to be
 # declared identically in both or whichever runs last decides numpy for both.
 numpy>=1.26.4,<2.0.0
+
+# xarray and pandas pinned — see the note in un_fao/requirements.txt for the measurement
+# and for why the bound stops at xarray 2024.5.0. Declared identically here because both
+# postprocessors install into the same envs/views-postprocessing prefix (C-116): pinning
+# only one lets whichever runs last decide pandas for both, which is the same trap the
+# numpy pin above already documents.
+xarray>=2024.1,<2024.5
+pandas>=1.5.3,<2.0.0

postprocessors/un_fao/requirements.txt

@@ -1,4 +1,11 @@
-views-datafactory>=1.9.0,<2.0.0
+# views-datafactory is floored at 1.13.0, not the >=1.9.0 this file carried (#509). The
+# credential-handling fixes landed there: before 1.13.0 the client could carry a netrc
+# credential across a redirect to another host, and could embed it in error messages. The
+# postprocessor runs on the same rented hardware as the training legs and fetches the actuals
+# it curates land -> land_gaul against, so it holds that credential too. Flooring only the pod
+# install (tools/podrun/pod_run_model.sh) left this leg — the LAST one, the one that reaches
+# the partner — declaring a version without the fix.
+views-datafactory>=1.13.0,<2.0.0
 # numpy stays on 1.x. views-datafactory allows <3, so pip resolves 2.x — and the pandas
 # wheel in envs/views-postprocessing is built against the numpy 1.x C ABI. Importing the
 # delivery manager then dies at views_pipeline_core/modules/datastore/datastore.py:54:
@@ -8,3 +15,28 @@ views-datafactory>=1.9.0,<2.0.0
 # Declared identically in un_crafd/requirements.txt — one shared prefix (C-116), so
 # pinning only one lets the other upgrade numpy out from under it on its next run.
 numpy>=1.26.4,<2.0.0
+
+# xarray and pandas are pinned because xarray is the CARRIER, and it was the only thing
+# in this file's dependency closure free to move. views-datafactory declares
+# `xarray>=2024.1,<2026` as a required dependency and `pandas>=2.0` only in its optional
+# `pandas` extra — which is not installed — so pandas arrives transitively THROUGH xarray
+# and nothing here constrained either one.
+#
+# Measured 2026-09-29, resolving this file as it stood:
+#     declared:  views-datafactory>=1.9.0,<2.0.0 + numpy>=1.26.4,<2.0.0
+#     resolved:  numpy 1.26.4, xarray 2025.12.0, pandas 3.0.6
+#     the pod that produced the first FAO delivery ran: xarray 2024.3.0, pandas 1.5.3
+#
+# So this file did not fail to build — it built, on a different pandas MAJOR, silently.
+# views-models#516 originally called the environment "unbuildable"; that was wrong and the
+# wrong version is the more dangerous one, because a build failure is loud. The working
+# versions existed only as hand-pins on a pod that has since been destroyed.
+#
+# The bound is measured, not guessed. xarray 2024.3.0 is the LAST release that accepts
+# pandas 1.x; 2024.5.0 moved to `pandas>=2.0` and 2024.9.0 to `pandas>=2.1`. A tidy-looking
+# `<2025` cap would therefore have been wrong — the cliff is inside the 2024 line, not at
+# the year boundary. Lifting these means moving the platform off pandas 1.x deliberately,
+# which views-pipeline-core already permits (`pandas>=1.5.3,<3.0`) and this codebase does
+# not yet assume.
+xarray>=2024.1,<2024.5
+pandas>=1.5.3,<2.0.0