Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -124,7 +124,7 @@ jobs:
working-directory: modflow-devtools/autotest
env:
REPOS_PATH: ${{ github.workspace }}
MODFLOW_DEVTOOLS_NO_AUTO_SYNC: 1
MODFLOW_DEVTOOLS_AUTO_SYNC: 0
TEST_DFN_PATH: ${{ github.workspace }}/modflow6/doc/mf6io/mf6ivar/dfn
# use --dist loadfile to so tests requiring pytest-virtualenv run on the same worker
run: uv run pytest -v -n auto --dist loadfile --durations 0 --ignore test_download.py --ignore test_models.py --ignore test_dfns_registry.py
Expand Down
4 changes: 2 additions & 2 deletions docs/md/dev/dfns.md
Original file line number Diff line number Diff line change
Expand Up @@ -476,7 +476,7 @@ status = get_sync_status()
- **At install time**: Best-effort sync to default refs during package installation (fail silently on network errors)
- **On first use**: If registry cache is empty for requested ref, attempt to sync before raising errors
- **Lazy loading**: Don't sync until DFN access is actually requested
- **Configurable**: Users can disable auto-sync via environment variable: `MODFLOW_DEVTOOLS_NO_AUTO_SYNC=1`
- **Configurable (Experimental)**: Auto-sync is opt-in via environment variable: `MODFLOW_DEVTOOLS_AUTO_SYNC=1` (set to "1", "true", or "yes")

### Source repository integration

Expand Down Expand Up @@ -1295,7 +1295,7 @@ dfn.name # attribute access
5. Implement `sync_dfns()` function
6. Add registry metadata caching with hash verification
7. Implement version-controlled registry discovery
8. Add auto-sync on first use (with opt-out via `MODFLOW_DEVTOOLS_NO_AUTO_SYNC`)
8. Add auto-sync on first use (opt-in via `MODFLOW_DEVTOOLS_AUTO_SYNC` while experimental)
9. **Implement `DfnSpec` dataclass** with `Mapping` protocol for single canonical hierarchical representation with flat dict access

**CLI and module API** (depends on Registry infrastructure):
Expand Down
2 changes: 1 addition & 1 deletion docs/md/dev/programs.md
Original file line number Diff line number Diff line change
Expand Up @@ -509,7 +509,7 @@ status = get_sync_status()

- **At install time**: Best-effort sync during package installation (fail silently on network errors)
- **On first use**: If registry cache is empty, attempt to sync before raising errors
- **Configurable**: Users can disable auto-sync via environment variable: `MODFLOW_DEVTOOLS_NO_AUTO_SYNC=1`
- **Configurable (Experimental)**: Auto-sync is opt-in via environment variable: `MODFLOW_DEVTOOLS_AUTO_SYNC=1` (set to "1", "true", or "yes")

#### Force semantics

Expand Down
14 changes: 7 additions & 7 deletions docs/md/models.md
Original file line number Diff line number Diff line change
Expand Up @@ -156,7 +156,7 @@ python -m modflow_devtools.models cp mf6/example/ex-gwf-twri01 /path/to/workspac
```

The copy command:
- Automatically attempts to sync registries before copying (unless `MODFLOW_DEVTOOLS_NO_AUTO_SYNC=1`)
- Automatically attempts to sync registries before copying (if `MODFLOW_DEVTOOLS_AUTO_SYNC=1`)
- Creates the workspace directory if it doesn't exist
- Copies all input files for the specified model
- Preserves subdirectory structure within the workspace
Expand Down Expand Up @@ -316,16 +316,16 @@ mf models clear --force

## Automatic Synchronization

By default, `modflow-devtools` attempts to sync registries:
- On first import (best-effort, fails silently on network errors)
- When accessing models (unless `MODFLOW_DEVTOOLS_NO_AUTO_SYNC=1`)

To disable auto-sync:
Auto-sync is **opt-in** (experimental). To enable:

```bash
export MODFLOW_DEVTOOLS_NO_AUTO_SYNC=1
export MODFLOW_DEVTOOLS_AUTO_SYNC=1 # or "true" or "yes"
```

When enabled, `modflow-devtools` attempts to sync registries:
- On first access (best-effort, fails silently on network errors)
- When accessing models via the API or CLI

Then manually sync when needed:

```bash
Expand Down
14 changes: 7 additions & 7 deletions docs/md/programs.md
Original file line number Diff line number Diff line change
Expand Up @@ -347,17 +347,17 @@ mf programs install mf6 --force

## Automatic Synchronization

By default, `modflow-devtools` attempts to sync registries:
- On first import (best-effort, fails silently on network errors)
- Before installation (unless `MODFLOW_DEVTOOLS_NO_AUTO_SYNC=1`)
- Before listing available programs

To disable auto-sync:
Auto-sync is **opt-in** (experimental). To enable:

```bash
export MODFLOW_DEVTOOLS_NO_AUTO_SYNC=1
export MODFLOW_DEVTOOLS_AUTO_SYNC=1 # or "true" or "yes"
```

When enabled, `modflow-devtools` attempts to sync registries:
- On first access (best-effort, fails silently on network errors)
- Before installation
- Before listing available programs

Then manually sync when needed:

```bash
Expand Down
70 changes: 69 additions & 1 deletion modflow_devtools/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,11 @@
Root CLI for modflow-devtools.

Usage:
mf sync
mf dfns sync
mf dfns info
mf dfns list
mf dfns clean
mf models sync
mf models info
mf models list
Expand All @@ -17,6 +22,56 @@

import argparse
import sys
import warnings


def _sync_all():
"""Sync all registries (dfns, models, programs)."""
print("Syncing all registries...")
print()

# Sync DFNs
print("=== DFNs ===")
try:
from modflow_devtools.dfns.registry import sync_dfns

registries = sync_dfns()
for registry in registries:
meta = registry.registry_meta
print(f" {registry.ref}: {len(meta.files)} files")
print(f"Synced {len(registries)} DFN registry(ies)")
except Exception as e:
print(f"Error syncing DFNs: {e}")
print()

# Sync Models
print("=== Models ===")
try:
from modflow_devtools.models import ModelSourceConfig

config = ModelSourceConfig.load()
config.sync()
print("Models synced successfully")
except Exception as e:
print(f"Error syncing models: {e}")
print()

# Sync Programs
print("=== Programs ===")
try:
# Suppress experimental warning
with warnings.catch_warnings():
warnings.filterwarnings("ignore", message=".*modflow_devtools.programs.*experimental.*")
from modflow_devtools.programs import ProgramSourceConfig

config = ProgramSourceConfig.load()
config.sync()
print("Programs synced successfully")
except Exception as e:
print(f"Error syncing programs: {e}")
print()

print("All registries synced!")


def main():
Expand All @@ -27,6 +82,12 @@ def main():
)
subparsers = parser.add_subparsers(dest="subcommand", help="Available commands")

# Sync subcommand (syncs all APIs)
subparsers.add_parser("sync", help="Sync all registries (dfns, models, programs)")

# DFNs subcommand
subparsers.add_parser("dfns", help="Manage MODFLOW 6 definition files")

# Models subcommand
subparsers.add_parser("models", help="Manage MODFLOW model registries")

Expand All @@ -41,7 +102,14 @@ def main():
sys.exit(1)

# Dispatch to the appropriate module CLI with remaining args
if args.subcommand == "models":
if args.subcommand == "sync":
_sync_all()
elif args.subcommand == "dfns":
from modflow_devtools.dfns.__main__ import main as dfns_main

sys.argv = ["mf dfns", *remaining]
sys.exit(dfns_main())
elif args.subcommand == "models":
from modflow_devtools.models.__main__ import main as models_main

# Replace sys.argv to make it look like we called the submodule directly
Expand Down
7 changes: 4 additions & 3 deletions modflow_devtools/dfns/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -575,11 +575,12 @@ def map(
schema_version: str | Version = "2",
) -> Dfn:
"""Map a MODFLOW 6 specification to another schema version."""
if dfn.schema_version == schema_version:
version = Version(str(schema_version))
if version == dfn.schema_version:
return dfn
elif Version(str(schema_version)) == Version("1"):
elif version == Version("1"):
raise NotImplementedError("Mapping to schema version 1 is not implemented yet.")
elif Version(str(schema_version)) == Version("2"):
elif version == Version("2"):
return MapV1To2().map(dfn)
raise ValueError(f"Unsupported schema version: {schema_version}. Expected 1 or 2.")

Expand Down
12 changes: 6 additions & 6 deletions modflow_devtools/dfns/__main__.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,10 +2,10 @@
Command-line interface for the DFNs API.

Usage:
python -m modflow_devtools.dfns sync [--ref REF] [--force]
python -m modflow_devtools.dfns info
python -m modflow_devtools.dfns list [--ref REF]
python -m modflow_devtools.dfns clean [--all]
mf dfns sync [--ref REF] [--force]
mf dfns info
mf dfns list [--ref REF]
mf dfns clean [--all]
"""

from __future__ import annotations
Expand Down Expand Up @@ -138,7 +138,7 @@ def cmd_list(args: argparse.Namespace) -> int:

except DfnRegistryNotFoundError as e:
print(f"Error: {e}", file=sys.stderr)
print("Try running 'python -m modflow_devtools.dfn sync' first.", file=sys.stderr)
print("Try running 'mf dfns sync' first.", file=sys.stderr)
return 1
except Exception as e:
print(f"Error: {e}", file=sys.stderr)
Expand Down Expand Up @@ -198,7 +198,7 @@ def _format_size(size_bytes: int) -> str:
def main(argv: list[str] | None = None) -> int:
"""Main entry point for the CLI."""
parser = argparse.ArgumentParser(
prog="python -m modflow_devtools.dfn",
prog="mf dfns",
description="MODFLOW 6 definition file tools",
)
parser.add_argument(
Expand Down
13 changes: 7 additions & 6 deletions modflow_devtools/dfns/registry.py
Original file line number Diff line number Diff line change
Expand Up @@ -734,7 +734,7 @@ def get_sync_status(source: str = "modflow6") -> dict[str, bool]:
def get_registry(
source: str = "modflow6",
ref: str = "develop",
auto_sync: bool = True,
auto_sync: bool = False,
path: str | PathLike | None = None,
) -> DfnRegistry:
"""
Expand All @@ -747,8 +747,9 @@ def get_registry(
ref : str, optional
Git ref (branch, tag, or commit hash). Default is "develop".
auto_sync : bool, optional
If True and registry is not cached, automatically sync. Default is True.
Can be disabled via MODFLOW_DEVTOOLS_NO_AUTO_SYNC environment variable.
If True and registry is not cached, automatically sync. Default is False
(opt-in while experimental). Can be enabled via MODFLOW_DEVTOOLS_AUTO_SYNC
environment variable (set to "1", "true", or "yes").
Ignored when path is provided.
path : str or PathLike, optional
Path to a local directory containing DFN files. If provided, returns
Expand All @@ -775,9 +776,9 @@ def get_registry(
if path is not None:
return LocalDfnRegistry(path=Path(path), source=source, ref=ref)

# Check for auto-sync opt-out
if os.environ.get("MODFLOW_DEVTOOLS_NO_AUTO_SYNC", "").lower() in ("1", "true", "yes"):
auto_sync = False
# Check for auto-sync opt-in (experimental - off by default)
if os.environ.get("MODFLOW_DEVTOOLS_AUTO_SYNC", "").lower() in ("1", "true", "yes"):
auto_sync = True

registry = RemoteDfnRegistry(source=source, ref=ref)

Expand Down
10 changes: 8 additions & 2 deletions modflow_devtools/models/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -1062,15 +1062,15 @@ def _load(self):
Load registry data from cache.

Raises an error if no cached registries are found.
Run 'python -m modflow_devtools.models sync' to populate the cache.
Run 'mf models sync' to populate the cache.
"""
# Try to load from cache
loaded_from_cache = self._try_load_from_cache()

if not loaded_from_cache:
raise RuntimeError(
"No model registries found in cache. "
"Run 'python -m modflow_devtools.models sync' to download registries, "
"Run 'mf models sync' to download registries, "
"or use ModelSourceConfig.load().sync() programmatically."
)

Expand Down Expand Up @@ -1322,13 +1322,19 @@ def get_default_registry():
This allows the module to import successfully even if the cache
is empty, with a clear error message on first use.

Auto-sync can be enabled via MODFLOW_DEVTOOLS_AUTO_SYNC environment variable
(currently opt-in while experimental). Set to "1", "true", or "yes" to enable.

Returns
-------
PoochRegistry
The default model registry
"""
global _default_registry_cache
if _default_registry_cache is None:
# Opt-in auto-sync (experimental - off by default)
if os.environ.get("MODFLOW_DEVTOOLS_AUTO_SYNC", "").lower() in ("1", "true", "yes"):
_try_best_effort_sync()
_default_registry_cache = PoochRegistry(base_url=_DEFAULT_BASE_URL, env=_DEFAULT_ENV)
return _default_registry_cache

Expand Down
6 changes: 3 additions & 3 deletions modflow_devtools/models/__main__.py
Original file line number Diff line number Diff line change
Expand Up @@ -108,7 +108,7 @@ def cmd_sync(args):
def cmd_info(args):
"""Info command handler."""
# Attempt auto-sync before showing info (unless disabled)
if not os.environ.get("MODFLOW_DEVTOOLS_NO_AUTO_SYNC"):
if os.environ.get("MODFLOW_DEVTOOLS_AUTO_SYNC", "").lower() in ("1", "true", "yes"):
_try_best_effort_sync()

config = ModelSourceConfig.load()
Expand Down Expand Up @@ -176,7 +176,7 @@ def cmd_info(args):
def cmd_list(args):
"""List command handler."""
# Attempt auto-sync before listing (unless disabled)
if not os.environ.get("MODFLOW_DEVTOOLS_NO_AUTO_SYNC"):
if os.environ.get("MODFLOW_DEVTOOLS_AUTO_SYNC", "").lower() in ("1", "true", "yes"):
_try_best_effort_sync()

cached = _DEFAULT_CACHE.list()
Expand Down Expand Up @@ -285,7 +285,7 @@ def cmd_clear(args):
def cmd_copy(args):
"""Copy command handler."""
# Attempt auto-sync before copying (unless disabled)
if not os.environ.get("MODFLOW_DEVTOOLS_NO_AUTO_SYNC"):
if os.environ.get("MODFLOW_DEVTOOLS_AUTO_SYNC", "").lower() in ("1", "true", "yes"):
_try_best_effort_sync()

from . import copy_to
Expand Down
8 changes: 5 additions & 3 deletions modflow_devtools/programs/__main__.py
Original file line number Diff line number Diff line change
Expand Up @@ -139,13 +139,15 @@ def cmd_info(args):
def cmd_list(args):
"""List command handler."""
# Attempt auto-sync before listing (unless disabled)
if not os.environ.get("MODFLOW_DEVTOOLS_NO_AUTO_SYNC"):
if os.environ.get("MODFLOW_DEVTOOLS_AUTO_SYNC", "").lower() in ("1", "true", "yes"):
_try_best_effort_sync()

cached = _DEFAULT_CACHE.list()

if not cached:
print("No cached program registries. Run 'sync' first.")
print(
"No program registries found in cache. Run 'mf programs sync' to download registries."
)
return

# Apply filters
Expand Down Expand Up @@ -193,7 +195,7 @@ def cmd_list(args):
def cmd_install(args):
"""Install command handler."""
# Attempt auto-sync before installation (unless disabled)
if not os.environ.get("MODFLOW_DEVTOOLS_NO_AUTO_SYNC"):
if os.environ.get("MODFLOW_DEVTOOLS_AUTO_SYNC", "").lower() in ("1", "true", "yes"):
_try_best_effort_sync()

# Parse program@version syntax if provided
Expand Down
Loading