Update main to reflect CEPM workflow

.gitignore

@@ -66,3 +66,6 @@ slurm_parallel_runs/
 
 # documentation files 
 docs/build/*
+
+# uv stuff
+ReEDS.egg-info/

.python-version

@@ -0,0 +1 @@
+3.11

CEPM/UV_MAMBA_GUIDE.md

@@ -0,0 +1,77 @@
+# Translating between `environment.yml` (mamba/conda) and `pyproject.toml`/`uv.lock` (uv)
+
+This repo keeps both, per the decision to keep `environment.yml` as the
+upstream-compatible fallback and `pyproject.toml`/`uv.lock` as RMI's primary
+path (see `pyproject.toml` at repo root and `dev`'s `environment.yml`).
+Whenever one changes, the other should be updated by hand — there's no
+automatic converter, so this is a manual mapping.
+
+## Automated drift check
+
+`CEPM/check_env_sync.py` compares the two files and warns when they drift beyond
+a hard-coded allowlist of intentional exceptions (the "known-accepted" cases
+below). `run_cepm.ps1` runs it as a non-fatal step, or run it directly:
+
+```bash
+uv run python CEPM/check_env_sync.py
+```
+
+It reports three things: packages only in one file and packages pinned to
+incompatible versions. Exit code `0` means aligned (all differences are on the
+allowlist); `1` means new/unexpected drift. When you intentionally add an
+exception, update both the allowlist in `CEPM/check_env_sync.py` and the
+"Known current drift" / "Things that don't map 1:1" sections below so the two
+stay in agreement.
+
+## Structural mapping
+
+| environment.yml | pyproject.toml | Notes |
+|---|---|---|
+| `name: reeds2` | `[project] name = "ReEDS"` | Names don't have to match; conda's is the env name, uv's is the package/distribution name. |
+| `dependencies: - python=3.11` | `[project] requires-python = "==3.11.*"` | Also mirrored in `.python-version` (`3.11`). |
+| `dependencies:` (conda channel packages, no `pip:` block) | `[project] dependencies = [...]` | Top-level conda deps → top-level uv deps. |
+| `dependencies: - pip: - ...` (the nested pip list) | `[project] dependencies = [...]` | Conda draws a line between conda-channel and pip-installed packages; uv doesn't — everything just goes in one `dependencies` list. |
+| Comment-delimited "optional" blocks (`## vvv ... ## ^^^`) | `[project.optional-dependencies]` groups (`interactive`, `maps`, `docs`, `reporting`, `network`, `testing`, `dev`) | conda has no native optional-group syntax, so mvp used a comment convention. uv has real extras — prefer putting new optional packages in the right extras group instead of a top-level comment block. |
+| version pins like `bokeh=3.2` | `"bokeh==3.2.*"` | conda's single `=` is a "starts with" match; uv/pip needs `==` plus an explicit `.*` to get the same "any patch version" behavior. |
+| N/A | `dev = [...]` group | uv's `dev` extra is just a bundle of the others (currently duplicates `interactive` + `maps` + `docs` + `reporting` + `network` + `testing`). Keep it in sync manually — there's no "include another extra" syntax in this pyproject's toml version. |
+
+## Adding a new dependency
+
+1. **Figure out where it's used.** Is it a hard runtime dependency (imported directly in `reeds/`, `hourlize/`, or `runreeds.py`) or something only needed for docs/interactive/testing?
+2. **Add it to `environment.yml`:**
+   - If it's on conda-forge/defaults, add it as a top-level line under `dependencies:` with a conda-style pin (`package=X.Y`).
+   - If it's pip-only, add it under the nested `pip:` list with a `==X.Y.Z` pin.
+   - If it's optional, put it inside the commented `## vvv ... ## ^^^` block (top-level optional) or the nested pip optional block.
+3. **Add it to `pyproject.toml`:**
+   - Hard dependency → `[project.dependencies]`, using `==X.Y.*` (or an exact pin if you want to match conda's pin precisely).
+   - Optional → the matching `[project.optional-dependencies]` group, **and** add it to the `dev` group too if `dev` is supposed to be "everything."
+4. **Regenerate the lockfile:** `uv lock` (or `uv sync --extra dev` if you also want your local venv updated). Don't hand-edit `uv.lock`.
+5. **Sanity-check both paths still work:**
+   - `uv sync --extra dev && uv run python runreeds.py -h`
+   - `mamba env update -f environment.yml` (or recreate the env) if you're validating the conda side too.
+
+## Removing a dependency
+
+Same as above in reverse — pull it from both files, then `uv lock` to drop it (and anything only it depended on) from `uv.lock`.
+
+## Things that don't map 1:1
+
+- **Non-Python packages.** `environment.yml` has `git-lfs=2.13` and `mscorefonts=0.0` — these aren't pip-installable and have no uv equivalent. They just don't appear in `pyproject.toml`; document them in setup docs instead.
+- **`pip` itself and its version.** conda pins `pip=23.2` as a bootstrap tool; uv manages its own resolver, so this has no uv equivalent either.
+- **Exact vs. fuzzy pins.** conda's `package=X.Y` conventionally means "X.Y.* is fine"; the uv side in this repo mostly uses explicit `==X.Y.*` to match that intent, but a few packages (`tables`, `gdxpds`, `geopandas`, `pulp`, `shapely`, `cmocean`) are pinned to an exact patch version in `pyproject.toml` where conda only pins minor. If you tighten/loosen a pin on one side, consider whether the other side should match.
+- **PyTables naming and version drift.** conda's package is `pytables`, but its importable/pip name is `tables`, so it appears as `pytables=3.8` in `environment.yml` and `tables==3.11.1` in `pyproject.toml`. These are currently different minor versions — a real drift, not just a naming difference. Reconcile the version if the two paths are meant to match.
+- **Git-sourced packages.** `rmi.etoolbox @ git+https://github.com/rmi/etoolbox.git` in `pyproject.toml` has no equivalent line in `environment.yml` yet — conda's `pip:` block *can* take a `git+https://...` URL the same way, so if `environment.yml` needs to stay fully equivalent, add a matching line there.
+
+## Known pre-existing gaps (not introduced by this cleanup, not yet fixed)
+
+A few packages are imported directly in `postprocessing/` (`jinja2`, `loguru`, `lxml`, `seaborn`, `six`) but aren't declared as explicit dependencies in *either* file — they currently work only because something else pulls them in transitively. This gap already exists on the `dev`/upstream base itself, independent of the RMI rebase, so it wasn't in scope for the pyproject.toml restoration. Worth its own follow-up if these ever stop being transitively satisfied.
+
+## Known current drift between the two files
+
+Both files now exist at the repo root, and a few packages are out of sync
+(worth reconciling next time either file is touched):
+
+- `pyyaml` is declared in `pyproject.toml` but missing from `environment.yml`.
+- `proj==0.2.0` is in `environment.yml`'s `pip:` block but has no entry in
+  `pyproject.toml`.
+- `pytables=3.8` (conda) vs `tables==3.11.1` (uv) — see the PyTables note above.

CEPM/check_env_sync.py

@@ -0,0 +1,218 @@
+"""Check that environment.yml (mamba/conda) and pyproject.toml (uv) stay aligned.
+
+RMI/CEPM keeps `environment.yml` as an upstream-compatible conda/mamba fallback
+and `pyproject.toml`/`uv.lock` as the primary uv path (see
+CEPM/UV_MAMBA_GUIDE.md). There is no automatic converter, so the two are kept in
+sync by hand and drift easily. This script reports drift and warns only when it
+finds a difference that is NOT already on the known-accepted allowlist below.
+
+It is intentionally stdlib-only (tomllib is in Python 3.11) so it can run as a
+bootstrap step without depending on a fully synced environment.
+
+Exit codes:
+    0  aligned (all differences are on the allowlist)
+    1  unexpected drift found (bootstrap turns this into a warning, not a failure)
+    2  could not run the check (e.g. a file is missing or unparseable)
+
+Keeping the allowlist current: as drift is reconciled, remove entries here. The
+script also notes allowlist entries that no longer apply, so the list does not
+silently rot.
+"""
+
+import os
+import re
+import sys
+import tomllib
+
+# --- Known-accepted exceptions (see CEPM/UV_MAMBA_GUIDE.md) ------------------
+
+# Packages expected to live ONLY in environment.yml -- no uv/pip equivalent by
+# nature (non-Python packages, conda bootstrap tooling, the interpreter itself).
+CONDA_ONLY_OK = {"git-lfs", "mscorefonts", "pip", "python"}
+
+# Packages expected to live ONLY in pyproject.toml (e.g. git-sourced deps that
+# have no conda-channel equivalent line yet).
+UV_ONLY_OK = {"rmi-etoolbox"}
+
+# Currently-accepted drift, documented in CEPM/UV_MAMBA_GUIDE.md. Trim as fixed.
+KNOWN_DRIFT_CONDA_ONLY = {"proj"}          # in environment.yml, not in pyproject
+KNOWN_DRIFT_UV_ONLY = {"pyyaml"}           # in pyproject, not in environment.yml
+KNOWN_VERSION_MISMATCH = {"tables"}        # pytables 3.8 (conda) vs tables 3.11.1 (uv)
+
+# conda package name -> pip/import name, so the same package matches across files.
+NAME_ALIASES = {"pytables": "tables"}
+
+
+def normalize(name):
+    """PEP 503-style normalization (collapse runs of . _ - to a single -) plus
+    conda->pip aliasing."""
+    n = re.sub(r"[-_.]+", "-", name.strip().lower())
+    return NAME_ALIASES.get(n, n)
+
+
+def _clean_version(ver):
+    """Strip a trailing '.*' / '*' wildcard and surrounding dots/space."""
+    ver = ver.strip().rstrip("*").rstrip(".").strip()
+    return ver or None
+
+
+def split_conda_spec(item):
+    """Parse a conda/pip dependency line ('bokeh=3.2', 'pulp==2.7.0', 'python')."""
+    for sep in ("==", "="):
+        if sep in item:
+            name, _, ver = item.partition(sep)
+            return name.strip(), _clean_version(ver)
+    return item.strip(), None
+
+
+def parse_requirement(spec):
+    """Parse a PEP 508 requirement from pyproject ('bokeh==3.2.*', 'x @ git+...')."""
+    spec = spec.strip()
+    if " @ " in spec:  # url/git form: 'name @ https://...'
+        return spec.split(" @ ", 1)[0].strip(), None
+    match = re.match(r"^([A-Za-z0-9_.\-]+)\s*(.*)$", spec)
+    if not match:
+        return spec, None
+    name, rest = match.group(1), match.group(2)
+    ver_match = re.search(r"==\s*([0-9][0-9A-Za-z.\-]*)", rest)
+    ver = _clean_version(ver_match.group(1)) if ver_match else None
+    return name, ver
+
+
+def parse_environment_yml(path):
+    """Minimal parser for our environment.yml -> {normalized_name: version|None}.
+
+    Tailored to this file's shape (a top-level `dependencies:` list plus a nested
+    `- pip:` list); not a general YAML parser. Both conda-channel and pip entries
+    are collected into one map since the comparison is by package, not channel.
+    """
+    deps = {}
+    in_deps = False
+    with open(path, encoding="utf-8") as handle:
+        for raw in handle:
+            line = raw.rstrip("\n")
+            if "#" in line:  # drop inline/full-line comments (incl. '## vvv' blocks)
+                line = line[: line.index("#")]
+            stripped = line.strip()
+            if not stripped:
+                continue
+            if stripped == "dependencies:":
+                in_deps = True
+                continue
+            if not in_deps:
+                continue
+            indent = len(line) - len(line.lstrip(" "))
+            # A new top-level key (e.g. `channels:`) ends the dependencies block.
+            if indent == 0 and not stripped.startswith("-"):
+                in_deps = False
+                continue
+            if not stripped.startswith("-"):
+                continue
+            item = stripped[1:].strip()  # drop the leading '- '
+            if not item or item.rstrip(":") == "pip":  # the '- pip:' sub-list header
+                continue
+            name, version = split_conda_spec(item)
+            if name:
+                deps[normalize(name)] = version
+    return deps
+
+
+def parse_pyproject(path):
+    """Return {normalized_name: version|None} across dependencies + all extras."""
+    with open(path, "rb") as handle:
+        data = tomllib.load(handle)
+    project = data.get("project", {})
+    specs = list(project.get("dependencies", []))
+    for group in project.get("optional-dependencies", {}).values():
+        specs.extend(group)
+    deps = {}
+    for spec in specs:
+        name, version = parse_requirement(spec)
+        if name:
+            deps[normalize(name)] = version
+    return deps
+
+
+def versions_align(conda_ver, uv_ver):
+    """True if versions are compatible. conda's 'X.Y' means 'X.Y.*', so treat a
+    shorter version as a prefix match of the longer one (e.g. 3.2 ~ 3.2.*)."""
+    if conda_ver is None or uv_ver is None:
+        return True  # nothing to compare (e.g. a git dep with no pinned version)
+    left = conda_ver.split(".")
+    right = uv_ver.split(".")
+    shared = min(len(left), len(right))
+    return left[:shared] == right[:shared]
+
+
+def main():
+    repo_root = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
+    if len(sys.argv) == 3:  # optional explicit paths, mostly for testing
+        env_path, pyproject_path = sys.argv[1], sys.argv[2]
+    else:
+        env_path = os.path.join(repo_root, "environment.yml")
+        pyproject_path = os.path.join(repo_root, "pyproject.toml")
+
+    try:
+        env_deps = parse_environment_yml(env_path)
+        uv_deps = parse_pyproject(pyproject_path)
+    except FileNotFoundError as exc:
+        print(f"[check_env_sync] could not read a dependency file: {exc}")
+        return 2
+    except Exception as exc:  # never let this check crash the bootstrap
+        print(f"[check_env_sync] failed to parse dependency files: {exc}")
+        return 2
+
+    env_names = set(env_deps)
+    uv_names = set(uv_deps)
+
+    conda_only = env_names - uv_names
+    uv_only = uv_names - env_names
+    mismatches = {
+        name
+        for name in (env_names & uv_names)
+        if not versions_align(env_deps[name], uv_deps[name])
+    }
+
+    unexpected_conda_only = sorted(conda_only - CONDA_ONLY_OK - KNOWN_DRIFT_CONDA_ONLY)
+    unexpected_uv_only = sorted(uv_only - UV_ONLY_OK - KNOWN_DRIFT_UV_ONLY)
+    unexpected_mismatch = sorted(mismatches - KNOWN_VERSION_MISMATCH)
+
+    # Note allowlist entries that no longer apply, so the list does not rot.
+    stale_allowlist = sorted(
+        (KNOWN_DRIFT_CONDA_ONLY - conda_only)
+        | (KNOWN_DRIFT_UV_ONLY - uv_only)
+        | (KNOWN_VERSION_MISMATCH - mismatches)
+    )
+
+    has_drift = bool(unexpected_conda_only or unexpected_uv_only or unexpected_mismatch)
+
+    if has_drift:
+        print("[check_env_sync] Unexpected drift between environment.yml and pyproject.toml:")
+        for name in unexpected_conda_only:
+            print(f"  - only in environment.yml: {name} ({env_deps[name] or 'no pin'})")
+        for name in unexpected_uv_only:
+            print(f"  - only in pyproject.toml:  {name} ({uv_deps[name] or 'no pin'})")
+        for name in unexpected_mismatch:
+            print(
+                f"  - version mismatch: {name} "
+                f"(environment.yml={env_deps[name]} vs pyproject.toml={uv_deps[name]})"
+            )
+        print(
+            "  Update both files, or add an intentional exception to the allowlist "
+            "in CEPM/check_env_sync.py (and note it in CEPM/UV_MAMBA_GUIDE.md)."
+        )
+    else:
+        print("[check_env_sync] environment.yml and pyproject.toml are aligned "
+              "(all differences are on the known-accepted allowlist).")
+
+    if stale_allowlist:
+        print(
+            "[check_env_sync] note: these allowlist entries no longer drift and can "
+            f"be removed from CEPM/check_env_sync.py: {', '.join(stale_allowlist)}"
+        )
+
+    return 1 if has_drift else 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

docs/internal-ci-testing.md → CEPM/internal-ci-testing.md

@@ -76,9 +76,9 @@ Expected directories include:
 Run each scenario exactly as CI does:
 
 ```bash
-python runbatch.py -b test -c test -s github_Pacific
-python runbatch.py -b test -c test -s github_Everything
-python runbatch.py -b test -c test -s github_MA_county_CC
+python runreeds.py -b test -c test -s github_Pacific
+python runreeds.py -b test -c test -s github_Everything
+python runreeds.py -b test -c test -s github_MA_county_CC
 ```
 
 Run folders should be created at:
@@ -115,12 +115,12 @@ Also inspect diagnostics if needed:
 
 ## 7) Run R2X compatibility checks (optional but recommended parity)
 
-CI executes `scripts/run_r2x.py` for each scenario. Reproduce locally after successful ReEDS runs.
+CI executes `postprocessing/run_r2x.py` for each scenario. Reproduce locally after successful ReEDS runs.
 
 If using `uvx` as in CI:
 
 ```bash
-uvx --from "r2x-reeds>=0.3.5" python scripts/run_r2x.py \
+uvx --from "r2x-reeds>=0.3.5" python postprocessing/run_r2x.py \
   --reeds-run-path "github_Pacific-2029" \
   --scenario "github_Pacific" \
   --solve-year "2029" \