Add a check mode and a relocate command for legacy data trees

docs/Explanations/design.md

@@ -38,6 +38,10 @@ The data root comes from an explicit path or the `FWL_DATA` environment variable
 
 *Why*: production runs happen on compute nodes without internet. Anything that only works online is not usable for the campaigns this ecosystem runs.
 
+Beside those two there is a third, read-only mode: `fwl-io check` reports whether a tree is complete and matches its registries without downloading anything or writing to it. Offline mode stops at the first thing it cannot serve and online mode repairs what it finds, so neither can answer what a diagnostic asks, which is the state of the whole tree at once.
+
+*Why the report says what it could not establish*: anything that stopped a file being looked at counts against the verdict, so a manifest that failed to load, a dataset whose registry has never been generated, a file that cannot be read, and a model no manifest declares are all faults rather than silence. A dataset whose members carry no digests to check against is reported as `presence only`; that is not a fault, because presence is all the archive-only checksum policy makes checkable, but it is said plainly rather than reported as verification.
+
 ## Atomic placement, distinct failure classes
 
 Every write is staged on the destination filesystem and moved into place with an atomic rename. Network or checksum failures raise a download error listing every mirror attempt; local problems (read-only tree, full disk) raise their own OSError.

docs/Explanations/manifests.md

@@ -67,7 +67,7 @@ For every requested file:
 
 ## The FWL_DATA layout
 
-This section is the target layout specification: new datasets and migrating models use it; existing trees keep their legacy directory names until their consumers migrate, so both forms coexist during the transition. A flat copy left by a pre-versioning fetch is re-fetched rather than adopted; a command that relocates such trees in place is tracked in [#13](https://github.com/FormingWorlds/fwl-io/issues/13).
+This section is the target layout specification: new datasets and migrating models use it; existing trees keep their legacy directory names until their consumers migrate, so both forms coexist during the transition. A flat copy left by a pre-versioning fetch is re-fetched rather than adopted; `fwl-io relocate` moves such a tree into its current location instead, once its files have been checked against the registry.
 
 The target tree is organized by physical domain, mirroring the package structure of the PROTEUS source tree (`src/proteus/`), with one deliberate exception: the two interior packages (`interior_struct`, `interior_energetics`) share a single `interior/` data domain, because the equation-of-state tables serve both.
 
@@ -95,7 +95,7 @@ FWL_DATA/
 
 The tree holds **immutable fetched reference data only**: anything generated at runtime (derived tables, interpolation caches, solver caches) belongs in run output or cache directories, never below `FWL_DATA`. This keeps a shared read-only cache trustworthy as a whole.
 
-Models adopt this layout when they migrate to fwl-io; legacy directories from the previous layout remain readable by unmigrated code and age out when their last consumer migrates (a relocate command for cleaning local trees immediately is tracked in [#13](https://github.com/FormingWorlds/fwl-io/issues/13)). The mapping from the legacy locations:
+Models adopt this layout when they migrate to fwl-io; legacy directories from the previous layout remain readable by unmigrated code and age out when their last consumer migrates. `fwl-io relocate` cleans a local tree up straight away instead, moving each dataset whose files check out against its registry. It acts on the datasets listed in the package's `legacy_layout.toml`, which grows as each model migrates and today names three of the families below; the rest are the historical mapping, and a tree holding one of them is left alone until its dataset is declared. The mapping from the legacy locations:
 
 | Legacy location (live today) | Target location |
 |---|---|

docs/Reference/api/check.md

@@ -0,0 +1,3 @@
+# Check
+
+::: fwl_io.check

docs/Reference/api/index.md

@@ -7,6 +7,9 @@ from fwl_io import (
     create_fetcher, Fetcher,            # fetching
     load_manifest, discover_manifests,  # manifests
     fetch_for, Dataset,
+    check_for, check_dataset,           # validate-only checking
+    relocate_all, plan_relocations,         # moving a legacy tree into the current layout
+    CheckReport, DatasetCheck, FileCheck,
     resolve_data_root, resolve_cache_root,
     DownloadError, OfflineDataError, MissingDataRootError,
     ManifestSchemaError,
@@ -16,6 +19,8 @@ from fwl_io import (
 Per-module reference pages:
 
 - [Fetching](fetch.md): `Fetcher`, `create_fetcher`, error types
+- [Checking](check.md): `check_for`, `check_dataset`, `CheckReport`, `DatasetCheck`, `FileCheck`
+- [Relocating](relocate.md): `relocate_all`, `plan_relocations`, `RelocationReport`, `Relocation`
 - [Manifests](manifest.md): `Dataset`, `load_manifest`, `discover_manifests`, `fetch_for`, `ManifestSchemaError`
 - [Registries](registry.md): registry file reading and writing
 - [Sync](sync.md): registry generation from the Zenodo API

docs/Reference/api/relocate.md

@@ -0,0 +1,3 @@
+# Relocate
+
+::: fwl_io.relocate

docs/Reference/cli.md

@@ -1,6 +1,6 @@
 # CLI reference
 
-The `fwl-io` command has four subcommands. Failures are reported as concise messages on stderr (never a traceback) and exit with status 1; success exits 0. `sync` and `fetch` aggregate per-dataset failures into a multi-line report, and a download failure lists every mirror attempt.
+The `fwl-io` command has six subcommands. Failures are reported as concise messages on stderr (never a traceback) and exit with status 1; success exits 0. `sync` and `fetch` aggregate per-dataset failures into a multi-line report, and a download failure lists every mirror attempt.
 
 ## fwl-io sync
 
@@ -26,6 +26,38 @@ fwl-io fetch <model> [--data-root PATH]
 
 Fetches every dataset that lists `<model>` in its `required_by`. All datasets are attempted; failures are aggregated into one report. `--data-root` overrides the `FWL_DATA` tree.
 
+## fwl-io check
+
+```bash
+fwl-io check <model> [--data-root PATH]
+```
+
+Reports whether every dataset that lists `<model>` in its `required_by` is present and matches its registry, without downloading anything. Each file is reported in one of five states: `ok` (present, checksum matches), `missing`, `mismatch` (present, contents differ), `unreadable` (present, could not be read to be checked), or `present`. The last means the file is there and nothing was available to verify it against, which is the case for the members of an archive dataset: the registry pins the checksum of the archive, not of the files extracted from it, so such a dataset is reported `presence only`. That is not counted as a fault, since presence is all that is checkable there, but it is never reported as verification.
+
+Two kinds of failure are reported apart from the datasets, because they call for different repairs. `MANIFEST UNREADABLE` means an installed package's manifest could not be read at all, so nothing it declares was inspected. `NOT CHECKED` means the manifest was fine but one dataset could not be resolved, most often because its registry has not been generated yet; run `fwl-io sync` for it. Either is on its own enough to fail the check.
+
+The report goes to stdout whatever the verdict, so a caller running this to find out what is wrong gets the detail and not only the exit status. Exit is 1 on any missing, corrupt or unreadable file, any unreadable manifest, any unresolvable dataset, or a model no manifest declares. The closing line says `all data present and verified` only when every file was compared against a digest; a sound tree holding a presence-only dataset closes with `all data present, N dataset(s) by presence only` instead, and still exits 0.
+
+Checking reads and hashes every file a plain dataset declares, so for those the cost is one full pass over the data: on a multi-gigabyte tree, or a shared cluster filesystem, expect it to take as long as reading that data once. An archive dataset costs far less, since its members have no digests to check and are only tested for presence.
+
+Nothing is downloaded and no dataset directory or file is written, which makes this safe to run against a tree another process is reading. Resolving the data root creates that root if it does not exist, as it does for every other subcommand. The equivalent Python entry point is `fwl_io.check_for`, whose `CheckReport.ok` is false when nothing was checked, so a model that matches no dataset can never read as a clean tree. `CheckReport.verified` is the stricter question, false whenever any part of the tree was checked by presence alone.
+
+## fwl-io relocate
+
+```bash
+fwl-io relocate [--data-root PATH] [--dry-run]
+```
+
+Moves data left by the previous layout into the place it belongs now, for a tree fetched before the current layout existed. Unmigrated code still reads the old directories, so they are otherwise left alone and age out as their consumers migrate; this is for cleaning a tree up straight away instead.
+
+A dataset moves only when every file its registry declares is present in the old location and matches its recorded digest. Anything else is reported and left exactly where it is: an incomplete tree, a file whose contents differ, or a dataset whose registry has not been generated. Verifying first is the point, since moving a stale copy would put it where the fetcher then trusts it. Once a dataset's files have moved, the emptied directories are removed, and the walk upward stops at the data root.
+
+Two kinds of dataset cannot be verified at all and are refused rather than moved, each named with its reason. An archive dataset's registry pins the packed archive, while an old tree holds the files extracted from it, so there is nothing to hash the tree against; move such a tree by hand, or delete it and let the fetcher rebuild it at the current location. A dataset whose registry is empty offers no files to compare, so every check over it would pass for want of anything to fail; run `fwl-io sync` for it. Both are reported only when an old directory is actually there, so a machine that never had the previous layout is unaffected.
+
+A dataset already at its current location is not a fault, and a copy still sitting at the old location beside it is named rather than deleted. Nothing here removes data: the only directories it removes are ones it has just emptied itself.
+
+Exit is 1 when a legacy tree was found and could not be moved, when an installed manifest could not be read, since that manifest may be the one declaring the dataset a tree still holds, or when the shipped table of old locations could not be read, since without it no dataset has an old location to look at and a run that reported nothing would read like a tidy tree. A tree that was already tidy exits 0. `--dry-run` reports the same plan without moving anything. The equivalent Python entry points are `fwl_io.relocate_all` and `fwl_io.plan_relocations`.
+
 ## fwl-io mirror
 
 ```bash

mkdocs.yml

@@ -27,6 +27,8 @@ nav:
       - API reference:
           - Overview: Reference/api/index.md
           - Fetching: Reference/api/fetch.md
+          - Checking: Reference/api/check.md
+          - Relocating: Reference/api/relocate.md
           - Manifests: Reference/api/manifest.md
           - Registries: Reference/api/registry.md
           - Sync: Reference/api/sync.md

pyproject.toml

@@ -44,6 +44,7 @@ Issues = "https://github.com/FormingWorlds/fwl-io/issues"
 develop = [
     "pytest>=8.0",
     "pytest-cov",
+    "pytest-timeout",
     "ruff",
 ]
 docs = [

src/fwl_io/__init__.py

@@ -9,6 +9,7 @@
 
 from importlib.metadata import PackageNotFoundError, version
 
+from fwl_io.check import CheckReport, DatasetCheck, FileCheck, check_dataset, check_for
 from fwl_io.fetch import DownloadError, Fetcher, OfflineDataError, create_fetcher
 from fwl_io.manifest import (
     Dataset,
@@ -18,24 +19,34 @@
     load_manifest,
 )
 from fwl_io.paths import MissingDataRootError, resolve_cache_root, resolve_data_root
+from fwl_io.relocate import Relocation, RelocationReport, plan_relocations, relocate_all
 
 try:
     __version__ = version('fwl-io')
 except PackageNotFoundError:  # editable checkout without installed metadata
     __version__ = '0.0.0'
 
 __all__ = [
+    'CheckReport',
     'Dataset',
+    'DatasetCheck',
     'DownloadError',
+    'FileCheck',
     'Fetcher',
     'ManifestSchemaError',
     'MissingDataRootError',
     'OfflineDataError',
+    'Relocation',
+    'RelocationReport',
     '__version__',
+    'check_dataset',
+    'check_for',
     'create_fetcher',
     'discover_manifests',
     'fetch_for',
     'load_manifest',
+    'plan_relocations',
+    'relocate_all',
     'resolve_cache_root',
     'resolve_data_root',
 ]