feat: Add RegistryRequest for pooch registries and improve catalog handling #490
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ndling - Add RegistryRequest class for fetching datasets from pooch registries (pmp-climatology, obs4ref) instead of ESGF - Add hash-based change detection to skip writing unchanged catalogs - Support multi-file datasets (same instance_id, different files) - Improve CLI test-cases commands: - Add --only-missing and --force flags to fetch command - Add catalog/regression status columns to list command - Refactor run command to iterate over multiple test cases - Add --dry-run, --if-changed, and --clean flags to run command - Add comprehensive tests for new functionality
- Create _git_utils.py with git status utilities (get_repo_for_path, get_git_status, collect_regression_file_info) - Add format_size() to _utils.py for human-readable file sizes - Move catalog_changed_since_regression() to climate_ref_core/testing.py - Update test_cases.py to use extracted functions (729 lines, down from 808) - Add comprehensive tests for all new/moved functions (100% coverage) Previously untested functions now have dedicated tests: - format_size: 14 parametrized tests - get_git_status: 8 tests - get_repo_for_path: 2 tests - collect_regression_file_info: 6 tests - catalog_changed_since_regression: 6 tests
Mark sections that are impractical to unit test: - _print_regression_summary: Rich table display code - fetch command dry-run display and ESGF fetching loop - run command dry-run display table
lewisjared
deleted the
feat/registry-request-catalog-improvements-clean
branch
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Add
RegistryRequestclass for fetching datasets from pooch registries (pmp-climatology, obs4ref) instead of ESGF, along with catalog handling improvements and CLI enhancements.New RegistryRequest Class
ESGFRequestprotocol for compatibilityCatalog Improvements
get_catalog_hash()function for checking existing catalog statesave_datasets_to_yaml()returns bool indicating if writtenCLI Improvements
fetch: Added--only-missingand--forceflagslist: Added catalog/regression status columnsrun: Refactored to iterate over multiple test cases for a providerrun: Added--dry-run,--if-changed, and--cleanflagsChecklist
Please confirm that this pull request has done the following:
changelog/