Enable ensemble creation and execution through a new yaml config file

.github/workflows/tests.yml

@@ -10,7 +10,7 @@ jobs:
   test:
     strategy:
       matrix:
-        python-version: ["3.12", "3.13"]
+        python-version: ["3.12", "3.13", "3.14"]
 
     runs-on: ubuntu-latest
     container: osgeo/grass-gis:releasebranch_8_4-ubuntu
@@ -31,11 +31,12 @@ jobs:
 
     - name: Pytest
       run: |
-        $HOME/.local/bin/uv run --all-extras --python ${{ matrix.python-version }} pytest --cov=src --junitxml=junit/test-cli-results.xml --cov-report=xml --cov-report=html tests/cli tests/test_grass_session.py
-        $HOME/.local/bin/uv run --all-extras --python ${{ matrix.python-version }} pytest --cov=src --junitxml=junit/test-itzi-results.xml --cov-report=xml --cov-report=html tests/grass/test_itzi.py
-        $HOME/.local/bin/uv run --all-extras --python ${{ matrix.python-version }} pytest --cov=src --junitxml=junit/test-bmi-results.xml --cov-report=xml --cov-report=html tests/grass/test_bmi.py
-        $HOME/.local/bin/uv run --all-extras --python ${{ matrix.python-version }} pytest --cov=src --junitxml=junit/test-tutorial-results.xml --cov-report=xml --cov-report=html tests/grass/test_tutorial.py
-        $HOME/.local/bin/uv run --all-extras --python ${{ matrix.python-version }} pytest --cov=src --junitxml=junit/test-temporal-overwrite-results.xml --cov-report=xml --cov-report=html tests/grass/test_temporal_overwrite.py
+        $HOME/.local/bin/uv run --all-extras --python ${{ matrix.python-version }} pytest --cov=src --cov-report= --junitxml=junit/test-cli-results.xml tests/cli tests/test_grass_session.py
+        for test_file in tests/grass/test_*.py; do
+          $HOME/.local/bin/uv run --all-extras --python ${{ matrix.python-version }} pytest --cov=src --cov-append --cov-report= --junitxml="junit/$(basename "$test_file" .py)-results.xml" "$test_file"
+        done
+        $HOME/.local/bin/uv run --all-extras --python ${{ matrix.python-version }} coverage xml
+        $HOME/.local/bin/uv run --all-extras --python ${{ matrix.python-version }} coverage html
 
     - name: Upload coverage
       uses: actions/upload-artifact@v7

AGENTS.md

@@ -2,15 +2,19 @@
 
 ## Common commands
 - Run a single test: `uv run pytest tests/my_test.py`
-- Due to a bug in GRASS, tests will fails if the mapset is changed between tests. Run each test file independently to prevent this. Tests that need to run in a separate process are marked with `@pytest.mark.forked`.
-- Enforce code formatting: `uvx ruff format .`
+- Due to a bug in GRASS<8.5, a change of mapset cannot occur within the same process when using the temporal framework. Each test file must therefore be run independently. Tests that need to run in a separate process are marked with `@pytest.mark.forked`.
+- Enforce code formatting: `uv run ruff format .`
 
 ## Code style
-- Use python type hints. When a function that does not yet use hints is substantially edited, take the opportunity to add type hints.
-- Do not quote class names in hints. Use `from __future__ import annotations` when necessary.
-- Since the arguments types and return types are already documented by the hints, there's no need to duplicate this information in the docstrings.
-- Apart from particular cases, use pydantic BaseModel instead of dataclass
+- Use python type annotation. When a function that does not yet use hints is substantially edited, take the opportunity to add type hints.
+- Do not quote class names in hints. Use `from __futue__ import annotations` when necessary.
+- Do not use the `Any` type, and avoid the use of the general `object` type. Define the actual type instead.
+- Since the arguments types and return types are already documented by the annotations, there's no need to duplicate this information in the docstrings.
+- Use pydantic BaseModel when validation is needed, and dataclass for internal data exchange.
+- No need to write tests to check if declarative Pydantic validation is working.
 - Place imports at the top of the file. Only break this rule to prevent heavy imports in a rarely used function (for example, CLI options).
+- When passing data structures between functions, prefer immutable objects instead of equivalent mutable ones, unless immutability is desired.
+- For GRASS interactions, prefer C functions calls through `pygrass` when available, instead of sub-process tool calls through `grass.script`
 
 ## General comments
 - The project uses `uv`. To run a command in the correct environment, use `uv run`

docs/cli.rst

@@ -11,6 +11,28 @@ Run a simulation
    :path: run
    :nodefault:
 
+YAML ensembles
+~~~~~~~~~~~~~~
+
+Each YAML document is one ensemble. Use ``--dry-run`` (or ``-d``) to perform parsing,
+expansion, GRASS input resolution, member-ID calculation, and artifact
+validation without writing manifests or model outputs:
+
+.. code-block:: bash
+
+   itzi run --dry-run studies.yaml
+
+Select resolved members with a qualified ensemble and simulation ID. The
+unqualified ID form is accepted only when the batch contains one ensemble.
+
+.. code-block:: bash
+
+   itzi run studies.yaml --member central-city#sim-c1f52e9a
+
+Stage 1 provides basic YAML ensemble execution. YAML hotstart resume mapping
+and checkpoint provenance are introduced in Stage 2; the retained INI resume
+syntax below continues to apply to legacy INI-only invocations.
+
 
 Hotstart usage
 ~~~~~~~~~~~~~~

docs/conf_file.rst

@@ -1,6 +1,6 @@
 
-Configuration file
-==================
+INI configuration file
+======================
 
 The parameters of a simulation are given through a configuration file in
 a format similar to Microsoft Windows INI files.

docs/index.rst

@@ -17,5 +17,7 @@ Please see the Itzï's `home page <https://www.itzi.org/>`__.
    cli
    tutorial
    conf_file
+   yaml_file
+   manifest
    faq
    prog_manual

docs/manifest.rst

@@ -0,0 +1,102 @@
+Ensemble manifest
+=================
+
+Each YAML ensemble run writes a YAML manifest describing its resolved members,
+their planned outputs, and their run status. By default, it is saved
+as ``results/<ensemble-id>.manifest.yaml`` beside the source YAML file. Set
+``outputs.manifest.file`` in the ensemble configuration to choose another file:
+
+.. code-block:: yaml
+
+   outputs:
+     manifest:
+       file: "results/{ensemble}.yaml"
+
+The manifest path may be absolute or relative to the source YAML file; only
+``{ensemble}`` is allowed as a template placeholder (not ``{simulation}``).
+An existing manifest requires ``itzi run -o`` to be replaced; if the manifest
+destination fails validation, no manifest is written. A dry run validates the
+destination but does not write a manifest. An ensemble with no selected resolved
+members does not write one, unless it contains unresolved candidates in a run
+without ``--member``.
+
+Format
+------
+
+The manifest is a YAML mapping with these top-level fields:
+
+* ``manifest_version``: manifest format version, currently ``1``. This is
+  distinct from the configuration's ``schema_version``.
+* ``last_updated_at``: last write time as an ISO 8601 string with a local
+  timezone offset. It changes when the manifest is updated.
+* ``ensemble``: ``id`` and ``description`` from the configuration;
+  ``description`` is ``null`` when omitted.
+* ``source``: absolute ``path`` to the source YAML file and zero-based
+  ``document_index`` within that YAML stream.
+* ``members``: resolved members in sweep expansion order, followed by
+  unresolved candidates in sweep expansion order.
+
+For example, a completed member might appear as follows (names and timings
+are illustrative):
+
+.. code-block:: yaml
+
+   manifest_version: 1
+   last_updated_at: '2026-09-30T10:20:30+02:00'
+   ensemble:
+     id: central-city
+     description: Central city rainfall study
+   source:
+     path: /srv/studies/city.yaml
+     document_index: 0
+   members:
+   - simulation_id: sim-1234abcd
+     selected: true
+     status: completed
+     elapsed_seconds: 12.5
+     coordinates:
+       parameters.cfl: 0.5
+     artifacts:
+       rasters:
+         water_depth: central_city_sim-1234abcd_water_depth@itzi
+       drainage: null
+       statistics: /srv/studies/results/sim-1234abcd.csv
+
+Member fields
+-------------
+
+``simulation_id`` is the resolved member ID (``sim-...``), or ``null`` if
+input resolution failed before an ID could be assigned. ``selected`` indicates
+whether this member was selected to run. ``coordinates`` maps swept configuration
+paths (for example, ``parameters.cfl`` or ``input.rainfall_rate``) to their
+chosen values; it is ``{}`` without sweeps. An ``input.infiltration`` sweep
+value is a mapping containing its ``type`` and applicable fields.
+
+Resolved members also have ``artifacts``: ``rasters`` maps requested output
+variables to GRASS space-time raster dataset IDs (``name@mapset``),
+``drainage`` is a GRASS space-time vector dataset ID or ``null``, and
+``statistics`` is an absolute CSV file path or ``null``. ``rasters`` is ``{}``
+when no raster outputs are requested. These are intended destinations; a
+failed or unfinished member may not have produced them. Unresolved candidates
+have no ``artifacts`` field.
+
+``elapsed_seconds`` is ``null`` until execution finishes, then records the
+elapsed wall-clock time in seconds, including when execution fails. The
+``status`` field can be:
+
+* ``not_selected``: excluded by ``--member``.
+* ``planned``: selected, but not yet started.
+* ``running``: execution started; the manifest is updated before and after
+  each member runs.
+* ``completed``: execution finished successfully.
+* ``validation_failed``: the member could not be run.
+* ``execution_failed``: execution raised an error.
+
+Failures also have a ``failure`` mapping with ``phase`` and ``detail``.
+Possible phases are ``input_resolution`` (no resolved ID or artifacts),
+``artifact_validation`` (conflicting or invalid outputs),
+``run_validation`` (pre-run checks), and ``execution``. Unresolved candidates
+are marked ``validation_failed`` when running without ``--member``; with
+selectors they appear as ``not_selected`` but still carry their failure
+detail. A failed YAML document (parse, schema, or expansion error) has no
+manifest entry because no ensemble was expanded from it.