Refactor data loading to modular data_utils structure#2
Merged
Conversation
- Import functions from data_utils.loaders, transformers, scalers - Remove duplicate function definitions - Reduced file size from 1200 to 631 lines (47% reduction) - Keep legacy functions for backward compatibility - All imports working correctly
- Add load_data_df to test assertions - All tests passing (4/4)
Changed stack_era5() to use explicit code matching via dictionary instead of assuming discharge and forcing TimeSeries lists are in the same order. - Create forcing_map dictionary for O(1) code lookups - Skip basins with missing forcing data (with warning) - Return filtered result list instead of modifying in place - Add comprehensive docstring explaining the matching logic This fixes the "error 15025.0 15022.0" messages caused by TimeSeries lists being in different orders. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Add factory functions for creating training components with full configurability: - RunConfig extended with scheduler, likelihood, callbacks, and checkpoint settings - New training_components.py with create_likelihood, create_scheduler_config, etc. - Unified initialize_model() using factory functions for all 6 model types - Configurable checkpoint_strategy (best/last) in training - Quantiles flow from config through model init Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Add TP, FP, TN, FN counts to flood_metrics.csv output - Add horizontal threshold lines (2, 5, 10-year) to forecast plots - Add confusion matrix visualization by return period with lead time subplots - Extend compute_classification_metrics, compute_basin_flood_metrics, and compute_all_basins_flood_metrics to include confusion matrix counts - Add load_flood_thresholds helper and RETURN_PERIOD_* constants - Add plot_confusion_matrices_by_return_period function - Update generate_visualizations to wire new features Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
…rohuni/DL-Short-Term-Forecasting into refactor/modular-data-utils
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Refactors
ST_Forecast/get_data.pyto use a cleaner, modulardata_utils/structure. Reduces code duplication and improves maintainability while preserving all functionality.Closes #1
Changes
New Module Structure
Key Improvements
1. Code Reduction: Reduced
get_data.pyfrom 1200 → 631 lines (47% reduction)2. Separation of Concerns:
3. Extracted Functions (16 total):
get_discharge_df(),get_era5_df(),get_static_df()load_snow_data(),load_forcing_data()time_shift_sla_data(),append_nir_to_sla(),load_sla_nir_data()process_sla_data(),add_moving_avg()compute_discharge_scalers(),apply_discharge_scalers()compute_era5_scalers(),apply_era5_scalers()scale_static_features(),save_scalers()4. Preserved Functionality:
load_data()Testing
✅ All import smoke tests pass (4/4)
Technical Details
Import Structure
Backward Compatibility
load_data()function signature unchangedscale_discharge,scale_era5) kept temporarilyCode Quality
ruff formatMigration Impact
Zero breaking changes - existing code using
load_data()will work without modification.Related Files
ST_Forecast/data_loading_example.py(functionality now in data_utils)tests/test_imports.py(smoke tests)scratchpads/issues/refactor-data-loading-modular-structure-1.md(implementation plan)🤖 Generated with Claude Code
Co-Authored-By: Claude noreply@anthropic.com