Trendy 1 ai restart creation - #66
Merged
Merged
Conversation
- Introduced `training_config_experiment_3_global.json` for a new global training setup with updated hyperparameters and variable weights. - Expanded `commands.txt` with additional commands for running inference and validation, including logging outputs for better tracking. - Enhanced `ai_model_comparison_plot.py` to support a broader range of variables and improved model comparison functionality. - Updated `run_inference_all.py` to resolve model paths more robustly, ensuring compatibility with different directory structures.
- Modified `commands.txt` to improve command formatting for better readability. - Added a new section in `CNP_pipeline_runbook.md` detailing the validation process for predictions in the `cnp_inference_entire_dataset` subdirectory, including commands for generating validation statistics and quality reports. - Enhanced `ai_model_comparison_plot.py` to clarify variable selection behavior based on command-line arguments.
- Added commands for navigating to the results directory and copying the configuration file for validation. - Included commands to generate validation statistics and prediction quality reports, enhancing the inference process documentation.
- Updated `CNP_IO_updated9_dev_dw.txt` to modify the file pattern for dataset matching and refined scalar variable listings. - Improved `commands.txt` with additional commands for model comparison and validation, including new options for running inference and generating quality reports. - Enhanced `training_config.py` to support multihead attention configurations for improved model performance. - Expanded `cnp_combined_model.py` to implement multihead attention and mode-specific feature extraction, optimizing the model's output capabilities. - Updated `ai_model_comparison_plot.py` to clarify model comparison functionality and improve plotting of results. - Added gridcell metadata handling in `cnp_result_validationplot.py` for better filtering of results based on natural vegetation presence.
…on_runs script - docs/RUN_COMPARISON_20260226_114546_vs_114659.md: summary of no-filter vs natveg-aligned comparison (test set alignment, validation stats, quality reports, conclusions) - scripts/compare_cnp_validation_runs.py: reusable script to compare two runs using validation_stats.csv and quality_summary_report.txt; supports --report FILE Made-with: Cursor
- Updated `train_cnp_model.py` to include new command-line arguments for optional output directory suffix and natveg-only filtering, allowing for more flexible model training configurations. - Introduced `training_config_experiment_3_global_natveg_improved.json` for improved training setup focusing on natural vegetation, with adjusted hyperparameters and filtering settings. - Enhanced `data_loader_individual.py` to support natveg-only filtering during data loading, ensuring that only relevant gridcells are used for training and validation. - Expanded documentation in `CNP_pipeline_runbook.md` and `NATVEG_VS_NOFILTER_COMPARISON.md` to clarify the implications of natveg filtering on model performance and validation metrics. - Updated `commands.txt` with additional commands for running new validation and comparison scripts, improving usability for users working with natveg configurations.
- Updated `commands.txt` to include a minimal pipeline for validation statistics, per-PFT npool/ppool validation, and quality report generation, streamlining the validation process. - Modified `cnp_result_validationplot.py` to ensure npool and ppool are always included in selections during restricted mode, enhancing scatter plot generation. - Refactored `generate_prediction_quality_report.py` to run the quality report generation and top bad plots more efficiently, ensuring consistent output paths. - Improved `validation_npool_ppool_per_pft.py` to create output directories as needed, enhancing usability for report generation.
…ipts - figured out the impact of npool/ppool background values to the quality report - identify the root problem of tropical region prediction is due to the values of solutionp_vr and occlp_vr - modify inference scripts to get full-grid, tropical range - add script to merge inference variable from tropical range training into an existing global restart file - Introduced `training_config_phase2_tropical_soilp_only.json` for a dedicated tropical model focusing on soil phosphorus variables, with specific hyperparameters and data filtering settings. - Updated `train_cnp_model.py` to handle tropical latitude range configurations, improving model training flexibility. - Enhanced `ai_predictions_to_restart.py` to support tropical-only updates during restart file generation, allowing for targeted variable updates in tropical grid cells. - Added new comparison scripts and documentation for validating predictions at the Amazon site, improving analysis of model performance across different configurations. - Created `AMAZON_SITE_SOILP_COMPARISON_REPORT.md` and `PHASE2_TROPICAL_SOILP_STRATEGY_AND_VALIDATION.md` to document methodologies and results for tropical soil-P variable comparisons.
- Updated `commands.txt` to include a new command for extracting the Amazon site from a full-grid 5P bias-corrected restart. - Expanded `INSTRUCTIONS_TRENDY_1_AI_RESTART_CREATION.md` to clarify prerequisites for new users and provide detailed steps for training two CNP models. - Revised `WORKFLOW_PHASE1_PHASE2_PHASE3_RESTARTS.md` to improve guidance for new users on model training and restart creation. - Added new scripts for running phases of the TRENDY-1 AI restart process, including `run_phase1_global.sh`, `run_phase2_tropical.sh`, and `run_phase3_tworegions.sh`, to streamline the workflow. - Enhanced `compare_5p_amazon_site_scatter.py` with new functions for loading 5P variables from restart files and generating profile and scatter plots for comparison.
…Pimprove add general site comparison code
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.