diff --git a/README.md b/README.md index 0f764ac13..c43286c09 100644 --- a/README.md +++ b/README.md @@ -69,7 +69,7 @@ Detailed dependency installation instructions are available on the ## Installation -After checking out HydgraGNN, we recommend to install HydraGNN in a +After checking out HydraGNN, we recommend installing HydraGNN in developer mode so that you can use the files in your current location and update them if needed: ```bash @@ -81,7 +81,7 @@ Or, simply type the following in the HydraGNN directory: export PYTHONPATH=$PWD:$PYTHONPATH ``` -Alternatively, if you have no plane to update, you can install +Alternatively, if you have no plans to update, you can install HydraGNN in your python tree as a static package: ```bash python setup.py install @@ -89,7 +89,7 @@ python setup.py install ## Quick Start -For detailed instructions, see the [**Comprehensive User Manual**](USER_MANUAL.md). +**For detailed instructions, configuration options, and advanced features, please refer to the [Comprehensive User Manual](USER_MANUAL.md).** Below are the four main functionalities for running the code. 1. Training a model, including continuing from a previously trained model using configuration options: @@ -114,80 +114,136 @@ hydragnn.load_existing_model(model, model_name, path="./logs/") import hydragnn hydragnn.run_prediction("examples/configuration.json", model) ``` -The `run_training` and `run_predictions` functions are convenient routines that encapsulate all the steps of the training process (data generation, data pre-processing, training of HydraGNN models, and use of trained HydraGNN models for inference) on toy problems, which are included in the CI test workflows. Both `run_training` and `run_predictions` require a JSON input file for configurable options. The `save_model` and `load_model` functions store and retrieve model checkpoints for continued training and subsequent inference. Ad-hoc example scripts where data pre-processing, training, and inference are done for specific datasets are provided in the `examples` folder. +The `run_training` and `run_predictions` functions are convenient routines that encapsulate all the steps of the training process (data generation, data pre-processing, training of HydraGNN models, and use of trained HydraGNN models for inference). Both `run_training` and `run_predictions` require a JSON input file for configurable options. The `save_model` and `load_model` functions store and retrieve model checkpoints for continued training and subsequent inference. + +Example scripts showing data pre-processing, training, and inference for specific datasets are provided in the `examples/` folder. See the [User Manual](USER_MANUAL.md) for comprehensive documentation. ### Datasets -Built in examples are provided for testing purposes only. One source of data to -create HydraGNN surrogate predictions is DFT output on the OLCF Constellation: -https://doi.ccs.ornl.gov/ - -Detailed instructions are available on the -[Wiki](https://github.com/ORNL/HydraGNN/wiki/Datasets) - -### Configurable settings - -HydraGNN uses a JSON configuration file (examples in `examples/`): - -There are many options for HydraGNN; the dataset and model type are particularly -important: - - `["Verbosity"]["level"]`: `0`, `1`, `2`, `3`, `4` (int) - - `["Dataset"]["name"]`: `CuAu_32atoms`, `FePt_32atoms`, `FeSi_1024atoms` (str) - -Additionally, many important arguments fall within the `["NeuralNetwork"]` section: - -- `["NeuralNetwork"]` - - `["Architecture"]` - - `["mpnn_type"]` - Accepted types: `CGCNN`, `DimeNet`, `EGNN`, `GAT`, `GIN`, `MACE`, `MFC`, `PAINN`, `PNAEq`, `PNAPlus`, `PNA`, `SAGE`, `SchNet` (str) - - `["num_conv_layers"]` - Examples: `1`, `2`, `3`, `4` ... (int) - - `["output_heads"]` - Task types: `node`, `graph` (int) - - `["global_attn_engine"]` - Accepted types: `GPS`, `None` - - `["global_attn_type"]` - Accepted types: `multihead` - - `["pe_dim"]` - Dimension of positional encodings (int) - - `["global_attn_heads"]` - Examples: `1`, `2`, `3`, `4` ... (int) - - `["hidden_dim"]` - Dimension of node embeddings during convolution (int) - must be a multiple of "global_attn_heads" if "global_attn_engine" is not "None" - - - - `["Variables of Interest"]` - - `["input_node_features"]` - Indices from nodal data used as inputs (int) - - `["output_index"]` - Indices from data used as targets (int) - - `["type"]` - Either `node` or `graph` (string) - - `["output_dim"]` - Dimensions of prediction tasks (list) - - - `["Training"]` - - `["num_epoch"]` - Examples: `75`, `100`, `250` (int) - - `["batch_size"]` - Examples: `16`, `32`, `64` (int) - - `["Optimizer"]["learning_rate"]` - Examples: `2e-3`, `0.005` (float) - - `["compute_grad_energy"]` - Use the gradient of energy to predict forces (bool) - - -### Citations -"HydraGNN: Distributed PyTorch implementation of multi-headed graph convolutional neural networks", Copyright ID#: 81929619 -https://doi.org/10.11578/dc.20211019.2 +Built-in examples are provided in the `examples/` directory, covering various domains: +- **Molecular datasets**: QM9, ANI-1x, QM7x, ZINC +- **Materials datasets**: Alexandria, Open Catalyst (2020/2022), Open Materials 2024, Open Molecules 2025 +- **Physics simulations**: Lennard-Jones, Ising Model, Power Grid + +One source of data to create HydraGNN surrogate predictions is DFT output on the OLCF Constellation: https://doi.ccs.ornl.gov/ + +Detailed instructions are available on the [Wiki](https://github.com/ORNL/HydraGNN/wiki/Datasets). + +### Configuration + +**For complete configuration documentation, see the [User Manual](USER_MANUAL.md).** + +HydraGNN uses a JSON configuration file to specify all aspects of model architecture, training, and data processing. Modern HydraGNN configurations use a feature-based approach for defining inputs and outputs. + +#### Key Configuration Sections + +**Dataset Configuration:** +```json +{ + "Dataset": { + "name": "my_dataset", + "format": "XYZ", // or "LSMS", "Pickle", "ADIOS2" + "node_features": { + "name": ["atomic_number", "coordinates"], + "dim": [1, 3], + "column_index": [0, 1] + }, + "graph_features": { + "name": ["energy"], + "dim": [1], + "column_index": [0] + } + } +} +``` + +**Neural Network Architecture:** +```json +{ + "NeuralNetwork": { + "Architecture": { + "mpnn_type": "EGNN", // See supported architectures below + "hidden_dim": 64, + "num_conv_layers": 4, + "output_heads": { + "graph": { + "num_headlayers": 2, + "dim_headlayers": [50, 25] + } + } + } + } +} +``` + +**Supported MPNN Architectures:** +- `CGCNN`, `DimeNet`, `EGNN`, `GAT`, `GIN`, `MACE`, `MFC`, `PAINN`, `PNA`, `PNAPlus`, `PNAEq`, `SAGE`, `SchNet` + +**Training Configuration:** +```json +{ + "NeuralNetwork": { + "Training": { + "num_epoch": 100, + "batch_size": 32, + "Optimizer": { + "type": "Adam", + "learning_rate": 0.001 + } + } + } +} +``` + +**Variables of Interest (Modern Approach):** + +HydraGNN now uses a declarative feature configuration with `node_features` and `graph_features` in the `Dataset` section. Each feature specifies: +- `name`: Descriptive name(s) +- `dim`: Dimensionality of each feature +- `column_index`: Which columns in `data.x` or `data.y` to extract (optional but recommended) +- `role`: `"input"` or `"output"` (specified in `Variables_of_interest` section) + +Example in `Variables_of_interest` section: +```json +{ + "NeuralNetwork": { + "Variables_of_interest": { + "node_features": { + "atomic_number": {"dim": 1, "role": "input"}, + "forces": {"dim": 3, "role": "output"} + }, + "graph_features": { + "energy": {"dim": 1, "role": "output"} + } + } + } +} +``` + +**Note:** Legacy configurations using `input_node_features` and `output_index` are still supported but not recommended for new projects. + +For comprehensive configuration details including: +- Node feature extraction rules and `column_index` usage +- Multi-dataset training +- DeepSpeed integration +- Global attention mechanisms +- Hyperparameter optimization + +Please refer to the [User Manual](USER_MANUAL.md). + ## Contributing -We encourage you to contribute to HydraGNN! Please check the -[guidelines](CONTRIBUTING.md) on how to do so. +We encourage you to contribute to HydraGNN! Please check the [guidelines](CONTRIBUTING.md) on how to do so. + +## Citations + +"HydraGNN: Distributed PyTorch implementation of multi-headed graph convolutional neural networks", Copyright ID#: 81929619 +https://doi.org/10.11578/dc.20211019.2 ## Documentation -- **Quick Start**: This README provides basic usage examples -- **[Comprehensive User Manual](USER_MANUAL.md)**: Detailed guide covering data pre-processing, model construction, scalable data management, and training +- **Quick Start**: This README provides basic usage and installation +- **[Comprehensive User Manual](USER_MANUAL.md)**: Detailed guide covering all features, data pre-processing, model construction, scalable data management, and training - **[Wiki](https://github.com/ORNL/HydraGNN/wiki)**: Additional technical documentation and datasets +- **Examples**: The `examples/` directory contains working configurations for various datasets and use cases diff --git a/USER_MANUAL.md b/USER_MANUAL.md index 4135533ac..4422aa579 100644 --- a/USER_MANUAL.md +++ b/USER_MANUAL.md @@ -332,19 +332,96 @@ HydraGNN provides extensive configuration options for building graph neural netw #### Variables of Interest +HydraGNN supports two configuration formats for defining features: a **new self-documenting format** (recommended) and a **legacy index-based format** (for backward compatibility). + +##### New Format (Recommended) + +The new format uses named features with explicit roles, making configurations self-documenting and less error-prone: + +```json +{ + "Variables_of_interest": { + "node_features": { + "atomic_number": { + "dim": 1, + "role": "input" + }, + "cartesian_coordinates": { + "dim": 3, + "role": "input" + }, + "forces": { + "dim": 3, + "role": "output", + "output_type": "node" + } + }, + "graph_features": { + "energy": { + "dim": 1, + "role": "output", + "output_type": "graph" + } + }, + "denormalize_output": true + } +} +``` + +**Feature Properties:** +- `dim` (required): Integer dimension of the feature +- `role` (required): One of: + - `"input"`: Feature used as model input + - `"output"`: Feature used as prediction target + - `"both"`: Feature used as both input and output +- `output_type` (optional): Either `"node"` or `"graph"` + +**Benefits:** +- ✅ Self-documenting: Feature names visible in config +- ✅ No manual index management +- ✅ Automatic validation on load +- ✅ Less prone to synchronization errors +- ✅ Easier to modify and maintain + +##### Legacy Format (Backward Compatible) + +The legacy format uses separate lists for names, dimensions, and indices: + ```json { "Variables_of_interest": { - "input_node_features": [0, 1, 2], // Input feature indices - "output_names": ["energy", "forces"], // Output property names - "output_index": [0, 1], // Output target indices - "type": ["graph", "node"], // Prediction types - "output_dim": [1, 3], // Output dimensions - "denormalize_output": true // Whether to denormalize predictions + "node_feature_names": ["atomic_number", "cartesian_coordinates", "forces"], + "node_feature_dims": [1, 3, 3], + "graph_feature_names": ["energy"], + "graph_feature_dims": [1], + "input_node_features": [0, 1], // Input feature indices + "output_names": ["forces", "energy"], // Output property names + "output_index": [2, 0], // Output target indices + "type": ["node", "graph"], // Prediction types + "output_dim": [3, 1], // Output dimensions + "denormalize_output": true } } ``` +**Note:** Both formats are fully supported. The new format is recommended for new projects, while existing configurations using the legacy format will continue to work without modification. + +##### Automatic Feature Processing + +Feature configuration is now handled automatically in `update_config()`. You no longer need to manually set feature names and dimensions in your training script: + +```python +# Old approach (no longer needed): +# config["NeuralNetwork"]["Variables_of_interest"]["graph_feature_names"] = ["energy"] +# config["NeuralNetwork"]["Variables_of_interest"]["node_feature_names"] = [...] + +# New approach (automatic): +config = hydragnn.utils.input_config_parsing.update_config( + config, train_loader, val_loader, test_loader +) +# Features are parsed and validated automatically from JSON config +``` + ### Global Attention Mechanisms #### GPS (Graph Positional and Structural Attention) @@ -399,6 +476,139 @@ from hydragnn.utils.model import print_model print_model(model) ``` +### Feature Configuration Utilities + + +### Important Note on Node Feature Dimensions + + +**If you do not specify `column_index` for each feature, the number of columns in `data.x` must exactly match the sum of the dimensions of the node features specified in your configuration JSON, and the columns must be contiguous and in the order given. If you do specify `column_index` for each feature, `data.x` can have extra columns, and only the specified columns will be used for each feature.** + +For example, if your node features are defined as: + +```json +"node_features": { + "atomic_number": {"dim": 1, "role": "input"}, + "cartesian_coordinates": {"dim": 3, "role": "input"}, + "forces": {"dim": 3, "role": "output"} +} +``` + +then your data.x must have exactly 7 columns (1 + 3 + 3 = 7). + +If your data.x has more columns than this, you must specify the correct column indices for each feature in the config. Otherwise, HydraGNN will raise a validation error and refuse to run. + +**Example: Using column_index to select specific columns** + +Suppose your `data.x` has 10 columns, but you only want to use columns 0, 5, and 6 as node features. You can specify this in your config as follows: + +```json +"node_features": { + "atomic_number": {"dim": 1, "role": "input", "column_index": 0}, + "charge_density": {"dim": 1, "role": "input", "column_index": 5}, + "magnetic_moment": {"dim": 1, "role": "input", "column_index": 6}, + "cartesian_coordinates": {"dim": 3, "role": "input", "column_index": 1} +} + +For features with `dim > 1`, `column_index` specifies the starting column, and HydraGNN will use a range of columns: `[column_index, column_index + dim)`. For example, with `cartesian_coordinates` above, HydraGNN will use columns 1, 2, and 3 from `data.x`. + +**Explicit JSON example:** + +Suppose your `data.x` has 10 columns, and you want to extract: +- column 0 for atomic_number (dim=1) +- columns 1, 2, 3 for cartesian_coordinates (dim=3) +- column 5 for charge_density (dim=1) +- column 6 for magnetic_moment (dim=1) + +Your JSON config would look like: + +```json +{ + "Variables_of_interest": { + "node_features": { + "atomic_number": {"dim": 1, "role": "input", "column_index": 0}, + "cartesian_coordinates": {"dim": 3, "role": "input", "column_index": 1}, + "charge_density": {"dim": 1, "role": "input", "column_index": 5}, + "magnetic_moment": {"dim": 1, "role": "input", "column_index": 6} + } + } +} +``` +``` + +This tells HydraGNN to extract only the specified columns from `data.x` for each feature, even if there are extra columns present. If you do not specify `column_index`, HydraGNN expects the columns to be contiguous and to match the sum of the feature dimensions exactly. + +**Consequences of Strict Column Matching:** + +- If you want to reduce the number of node features extracted (for example, by removing a feature from your config), you must re-preprocess your dataset so that `data.x` matches the new sum of feature dimensions. HydraGNN does not support skipping columns or ignoring extra features unless you explicitly specify column indices for each feature in the config. +- This strict design choice ensures that your feature configuration and dataset are always synchronized, preventing subtle bugs and training errors, but it also means that any change to the node features in your config requires regenerating your dataset to match. + +--- + +HydraGNN provides utilities for validating and working with feature configurations. + +#### Validating Feature Configuration + +```python +from hydragnn.utils.input_config_parsing import validate_feature_config + +var_config = config["NeuralNetwork"]["Variables_of_interest"] +is_valid, errors = validate_feature_config(var_config) + +if not is_valid: + print("Configuration errors:") + for error in errors: + print(f" - {error}") +``` + +#### Printing Feature Summary + +```python +from hydragnn.utils.input_config_parsing import print_feature_summary + +var_config = config["NeuralNetwork"]["Variables_of_interest"] +summary = print_feature_summary(var_config) +print(summary) +``` + +**Example Output:** +``` +============================================================ +Feature Configuration Summary +============================================================ + +Node Features: +------------------------------------------------------------ + [0] atomic_number dim= 1 role=input + [1] cartesian_coordinates dim= 3 role=input + [2] forces dim= 3 role=output + +Graph Features: +------------------------------------------------------------ + [0] energy dim= 1 + +Output Configuration: +------------------------------------------------------------ + forces index= 2 dim= 3 type=node + energy index= 0 dim= 1 type=graph +============================================================ +``` + +#### Programmatic Feature Parsing + +```python +from hydragnn.utils.input_config_parsing import parse_feature_config + +var_config = config["NeuralNetwork"]["Variables_of_interest"] +parsed = parse_feature_config(var_config) + +# Access parsed information +print(f"Input features: {parsed['input_node_features']}") +print(f"Output names: {parsed['output_names']}") +print(f"Node feature dims: {parsed['node_feature_dims']}") +print(f"Graph feature dims: {parsed['graph_feature_dims']}") +``` + --- ## Scalable Data Management @@ -1147,7 +1357,101 @@ export NCCL_DEBUG=INFO } ``` -### 5. Production Deployment +### 5. Configuration Migration Guide + +#### Migrating from Legacy to New Feature Format + +If you have existing configurations using the legacy format, you can easily migrate to the new self-documenting format. + +**Before (Legacy Format):** +```json +{ + "Variables_of_interest": { + "node_feature_names": ["atomic_number", "coordinates", "forces"], + "node_feature_dims": [1, 3, 3], + "graph_feature_names": ["energy"], + "graph_feature_dims": [1], + "input_node_features": [0, 1], + "output_names": ["forces", "energy"], + "output_index": [2, 0], + "type": ["node", "graph"], + "output_dim": [3, 1] + } +} +``` + +**After (New Format):** +```json +{ + "Variables_of_interest": { + "node_features": { + "atomic_number": { + "dim": 1, + "role": "input" + }, + "coordinates": { + "dim": 3, + "role": "input" + }, + "forces": { + "dim": 3, + "role": "output", + "output_type": "node" + } + }, + "graph_features": { + "energy": { + "dim": 1, + "role": "output", + "output_type": "graph" + } + } + } +} +``` + +**Migration Steps:** + +1. **Remove hardcoded feature assignments from train.py:** + ```python + # Delete these lines: + # config["NeuralNetwork"]["Variables_of_interest"]["graph_feature_names"] = [...] + # config["NeuralNetwork"]["Variables_of_interest"]["node_feature_names"] = [...] + # config["NeuralNetwork"]["Variables_of_interest"]["input_node_features"] = [...] + # etc. + ``` + +2. **Update JSON configuration to use new format:** + - Create `node_features` and `graph_features` dictionaries + - For each feature, specify `dim`, `role`, and `output_type` (if output) + - Feature names become dictionary keys (order preserved) + +3. **Let update_config() handle the rest:** + ```python + # This now automatically parses and validates features + config = hydragnn.utils.input_config_parsing.update_config( + config, train_loader, val_loader, test_loader + ) + ``` + +**Backward Compatibility:** +- Legacy format continues to work without modification +- Both formats can coexist in the same codebase +- No breaking changes to existing workflows + +**Validation:** +```python +from hydragnn.utils.input_config_parsing import validate_feature_config + +# Validate your configuration +is_valid, errors = validate_feature_config(config["NeuralNetwork"]["Variables_of_interest"]) +if not is_valid: + print("Configuration issues found:") + for error in errors: + print(f" - {error}") +``` + +### 6. Production Deployment #### Model Checkpointing ```json diff --git a/examples/alexandria/train.py b/examples/alexandria/train.py index 1a6f5b653..7f26fec51 100644 --- a/examples/alexandria/train.py +++ b/examples/alexandria/train.py @@ -470,10 +470,6 @@ def get(self, idx): parser.set_defaults(format="adios") args = parser.parse_args() - graph_feature_names = ["energy"] - graph_feature_dims = [1] - node_feature_names = ["atomic_number", "cartesian_coordinates", "forces"] - node_feature_dims = [1, 3, 3] dirpwd = os.path.dirname(os.path.abspath(__file__)) datadir = os.path.join(dirpwd, "dataset") ################################################################################################################## @@ -484,10 +480,13 @@ def get(self, idx): config = json.load(f) verbosity = config["Verbosity"]["level"] var_config = config["NeuralNetwork"]["Variables_of_interest"] - var_config["graph_feature_names"] = graph_feature_names - var_config["graph_feature_dims"] = graph_feature_dims - var_config["node_feature_names"] = node_feature_names - var_config["node_feature_dims"] = node_feature_dims + + # Note: Feature configuration now handled automatically in update_config() + # If using legacy format, you can still override features here if needed: + # var_config["graph_feature_names"] = ["energy"] + # var_config["graph_feature_dims"] = [1] + # var_config["node_feature_names"] = ["atomic_number", "cartesian_coordinates", "forces"] + # var_config["node_feature_dims"] = [1, 3, 3] # Transformation to create positional and structural laplacian encoders """ diff --git a/examples/ani1_x/ani1x_energy.json b/examples/ani1_x/ani1x_energy.json index 600395c4f..15dd40292 100644 --- a/examples/ani1_x/ani1x_energy.json +++ b/examples/ani1_x/ani1x_energy.json @@ -32,11 +32,26 @@ "task_weights": [1.0] }, "Variables_of_interest": { - "input_node_features": [0, 1, 2, 3], - "output_names": ["energy"], - "output_index": [0], - "output_dim": [1], - "type": ["graph"] + "node_features": { + "atomic_number": { + "dim": 1, + "role": "input" + }, + "cartesian_coordinates": { + "dim": 3, + "role": "input" + }, + "forces": { + "dim": 3, + "role": "output" + } + }, + "graph_features": { + "energy": { + "dim": 1, + "role": "output" + } + } }, "Training": { "num_epoch": 50, diff --git a/examples/ani1_x/ani1x_forces.json b/examples/ani1_x/ani1x_forces.json index 994c8f423..bc1487ee6 100644 --- a/examples/ani1_x/ani1x_forces.json +++ b/examples/ani1_x/ani1x_forces.json @@ -31,11 +31,26 @@ "task_weights": [1.0] }, "Variables_of_interest": { - "input_node_features": [0, 1, 2, 3], - "output_names": ["forces"], - "output_index": [2], - "output_dim": [3], - "type": ["node"] + "node_features": { + "atomic_number": { + "dim": 1, + "role": "input" + }, + "cartesian_coordinates": { + "dim": 3, + "role": "input" + }, + "forces": { + "dim": 3, + "role": "output" + } + }, + "graph_features": { + "energy": { + "dim": 1, + "role": "output" + } + } }, "Training": { "num_epoch": 50, diff --git a/examples/ani1_x/ani1x_mlip.json b/examples/ani1_x/ani1x_mlip.json index 19fd90239..0f60108b6 100644 --- a/examples/ani1_x/ani1x_mlip.json +++ b/examples/ani1_x/ani1x_mlip.json @@ -33,16 +33,19 @@ ] }, "Variables_of_interest": { - "input_node_features": [0], - "output_index": [ - 1 - ], - "type": [ - "node" - ], - "output_dim": [1], - "output_names": ["graph_energy"], - "denormalize_output": false + "node_features": { + "atomic_number": { + "dim": 1, + "role": "input" + } + }, + "graph_features": { + "graph_energy": { + "dim": 1, + "role": "output", + "denormalize": false + } + } }, "Training": { "Checkpoint" : true, diff --git a/examples/ani1_x/train.py b/examples/ani1_x/train.py index 0d568f4e3..bdcb6d024 100644 --- a/examples/ani1_x/train.py +++ b/examples/ani1_x/train.py @@ -309,10 +309,6 @@ def get(self, idx): parser.set_defaults(format="adios") args = parser.parse_args() - graph_feature_names = ["energy"] - graph_feature_dims = [1] - node_feature_names = ["atomic_number", "cartesian_coordinates", "forces"] - node_feature_dims = [1, 3, 3] dirpwd = os.path.dirname(os.path.abspath(__file__)) datadir = os.path.join(dirpwd, "dataset") ################################################################################################################## @@ -323,10 +319,13 @@ def get(self, idx): config = json.load(f) verbosity = config["Verbosity"]["level"] var_config = config["NeuralNetwork"]["Variables_of_interest"] - var_config["graph_feature_names"] = graph_feature_names - var_config["graph_feature_dims"] = graph_feature_dims - var_config["node_feature_names"] = node_feature_names - var_config["node_feature_dims"] = node_feature_dims + + # Note: Feature configuration now handled automatically in update_config() + # If using legacy format, you can still override features here if needed: + # var_config["graph_feature_names"] = ["energy"] + # var_config["graph_feature_dims"] = [1] + # var_config["node_feature_names"] = ["atomic_number", "cartesian_coordinates", "forces"] + # var_config["node_feature_dims"] = [1, 3, 3] # Transformation to create positional and structural laplacian encoders """ diff --git a/examples/dftb_uv_spectrum/dftb_discrete_uv_spectrum.json b/examples/dftb_uv_spectrum/dftb_discrete_uv_spectrum.json index 95739836b..602a7fedf 100644 --- a/examples/dftb_uv_spectrum/dftb_discrete_uv_spectrum.json +++ b/examples/dftb_uv_spectrum/dftb_discrete_uv_spectrum.json @@ -25,28 +25,23 @@ ] }, "Variables_of_interest": { - "input_node_features": [ - 0, - 1, - 2, - 3, - 4, - 5, - 6, - 7, - 8, - 9, - 10, - 11 - ], - "output_index": [ - 0, - 1 - ], - "type": [ - "graph", - "graph" - ], + "node_features": { + "atomic_features": { + "dim": 12, + "role": "input", + "comment": "Dynamically generated from dftb_node_types - 12 features from one-hot encoding" + } + }, + "graph_features": { + "frequencies": { + "dim": 50, + "role": "output" + }, + "intensities": { + "dim": 50, + "role": "output" + } + }, "denormalize_output": false }, "Training": { diff --git a/examples/dftb_uv_spectrum/dftb_smooth_uv_spectrum.json b/examples/dftb_uv_spectrum/dftb_smooth_uv_spectrum.json index eef76cfbc..e6b28b3eb 100644 --- a/examples/dftb_uv_spectrum/dftb_smooth_uv_spectrum.json +++ b/examples/dftb_uv_spectrum/dftb_smooth_uv_spectrum.json @@ -24,28 +24,19 @@ ] }, "Variables_of_interest": { - "input_node_features": [ - 0, - 1, - 2, - 3, - 4, - 5, - 6, - 7, - 8, - 9, - 10, - 11 - ], - "output_index": [ - 0 - ], - "type": [ - "graph" - ], - "output_dim": [37500], - "output_names": ["spectrum"], + "node_features": { + "atomic_features": { + "dim": 12, + "role": "input", + "comment": "Dynamically generated from dftb_node_types - 12 features from one-hot encoding" + } + }, + "graph_features": { + "spectrum": { + "dim": 37500, + "role": "output" + } + }, "denormalize_output": false }, "Training": { diff --git a/examples/dftb_uv_spectrum/train_discrete_uv_spectrum.py b/examples/dftb_uv_spectrum/train_discrete_uv_spectrum.py index b0e5fc98c..4fccfdcc8 100644 --- a/examples/dftb_uv_spectrum/train_discrete_uv_spectrum.py +++ b/examples/dftb_uv_spectrum/train_discrete_uv_spectrum.py @@ -162,8 +162,6 @@ def get(self, idx): parser.set_defaults(format="adios") args = parser.parse_args() - graph_feature_names = ["frequencies", "intensities"] - graph_feature_dim = [50, 50] dirpwd = os.path.dirname(os.path.abspath(__file__)) datafile = os.path.join(dirpwd, "dataset/dftb_aisd_electronic_excitation_spectrum") ################################################################################################################## @@ -174,16 +172,22 @@ def get(self, idx): config = json.load(f) verbosity = config["Verbosity"]["level"] var_config = config["NeuralNetwork"]["Variables_of_interest"] - var_config["output_names"] = [ - graph_feature_names[item] - for ihead, item in enumerate(var_config["output_index"]) - ] - var_config["graph_feature_names"] = graph_feature_names - var_config["graph_feature_dims"] = graph_feature_dim - ( - var_config["input_node_feature_names"], - var_config["input_node_feature_dims"], - ) = get_node_attribute_name(dftb_node_types) + + # Note: Feature configuration now handled automatically in update_config() + # Extract feature names/dims from config for use in post-processing + # Node features are dynamically determined from dftb_node_types + if "graph_features" in var_config: + # New format + graph_feature_names = list(var_config["graph_features"].keys()) + graph_feature_dim = [ + var_config["graph_features"][k]["dim"] for k in graph_feature_names + ] + else: + # Legacy format fallback + graph_feature_names = var_config.get( + "graph_feature_names", ["frequencies", "intensities"] + ) + graph_feature_dim = var_config.get("graph_feature_dims", [50, 50]) if args.batch_size is not None: config["NeuralNetwork"]["Training"]["batch_size"] = args.batch_size ################################################################################################################## diff --git a/examples/dftb_uv_spectrum/train_smooth_uv_spectrum.py b/examples/dftb_uv_spectrum/train_smooth_uv_spectrum.py index 333043dd5..86fef679b 100644 --- a/examples/dftb_uv_spectrum/train_smooth_uv_spectrum.py +++ b/examples/dftb_uv_spectrum/train_smooth_uv_spectrum.py @@ -160,8 +160,6 @@ def get(self, idx): parser.set_defaults(format="adios") args = parser.parse_args() - graph_feature_names = ["spectrum"] - graph_feature_dim = [37500] dirpwd = os.path.dirname(os.path.abspath(__file__)) datafile = os.path.join(dirpwd, "dataset/dftb_aisd_electronic_excitation_spectrum") ################################################################################################################## @@ -172,16 +170,20 @@ def get(self, idx): config = json.load(f) verbosity = config["Verbosity"]["level"] var_config = config["NeuralNetwork"]["Variables_of_interest"] - var_config["output_names"] = [ - graph_feature_names[item] - for ihead, item in enumerate(var_config["output_index"]) - ] - var_config["graph_feature_names"] = graph_feature_names - var_config["graph_feature_dims"] = graph_feature_dim - ( - var_config["input_node_feature_names"], - var_config["input_node_feature_dims"], - ) = get_node_attribute_name(dftb_node_types) + + # Note: Feature configuration now handled automatically in update_config() + # Extract feature names/dims from config for use in post-processing + # Node features are dynamically determined from dftb_node_types + if "graph_features" in var_config: + # New format + graph_feature_names = list(var_config["graph_features"].keys()) + graph_feature_dim = [ + var_config["graph_features"][k]["dim"] for k in graph_feature_names + ] + else: + # Legacy format fallback + graph_feature_names = var_config.get("graph_feature_names", ["spectrum"]) + graph_feature_dim = var_config.get("graph_feature_dims", [37500]) if args.batch_size is not None: config["NeuralNetwork"]["Training"]["batch_size"] = args.batch_size ################################################################################################################## diff --git a/examples/md17/md17.json b/examples/md17/md17.json index 72ca5e01b..e65837968 100644 --- a/examples/md17/md17.json +++ b/examples/md17/md17.json @@ -38,11 +38,18 @@ "task_weights": [1.0] }, "Variables_of_interest": { - "input_node_features": [0], - "output_names": ["energy"], - "output_index": [0], - "output_dim": [1], - "type": ["graph"], + "node_features": { + "atomic_number": { + "dim": 1, + "role": "input" + } + }, + "graph_features": { + "energy": { + "dim": 1, + "role": "output" + } + }, "denormalize_output": false }, "Training": { diff --git a/examples/mptrj/train.py b/examples/mptrj/train.py index 9390421b1..910deab0b 100644 --- a/examples/mptrj/train.py +++ b/examples/mptrj/train.py @@ -321,10 +321,6 @@ def get(self, idx): parser.set_defaults(format="adios") args = parser.parse_args() - graph_feature_names = ["energy"] - graph_feature_dims = [1] - node_feature_names = ["atomic_number", "cartesian_coordinates", "forces"] - node_feature_dims = [1, 3, 3] dirpwd = os.path.dirname(os.path.abspath(__file__)) datadir = os.path.join(dirpwd, "dataset") ################################################################################################################## @@ -335,10 +331,13 @@ def get(self, idx): config = json.load(f) verbosity = config["Verbosity"]["level"] var_config = config["NeuralNetwork"]["Variables_of_interest"] - var_config["graph_feature_names"] = graph_feature_names - var_config["graph_feature_dims"] = graph_feature_dims - var_config["node_feature_names"] = node_feature_names - var_config["node_feature_dims"] = node_feature_dims + + # Note: Feature configuration now handled automatically in update_config() + # If using legacy format, you can still override features here if needed: + # var_config["graph_feature_names"] = ["energy"] + # var_config["graph_feature_dims"] = [1] + # var_config["node_feature_names"] = ["atomic_number", "cartesian_coordinates", "forces"] + # var_config["node_feature_dims"] = [1, 3, 3] # Transformation to create positional and structural laplacian encoders """ diff --git a/examples/multibranch/train.py b/examples/multibranch/train.py index 754dc6369..0b5ccb1a6 100644 --- a/examples/multibranch/train.py +++ b/examples/multibranch/train.py @@ -113,10 +113,6 @@ def info(*args, logtype="info", sep=" "): parser.set_defaults(format="adios") args = parser.parse_args() - graph_feature_names = ["energy"] - graph_feature_dims = [1] - node_feature_names = ["atomic_number", "cartesian_coordinates", "forces"] - node_feature_dims = [1, 3, 3] dirpwd = os.path.dirname(os.path.abspath(__file__)) datadir = os.path.join(dirpwd, "dataset") ################################################################################################################## @@ -127,10 +123,13 @@ def info(*args, logtype="info", sep=" "): config = json.load(f) verbosity = config["Verbosity"]["level"] var_config = config["NeuralNetwork"]["Variables_of_interest"] - var_config["graph_feature_names"] = graph_feature_names - var_config["graph_feature_dims"] = graph_feature_dims - var_config["node_feature_names"] = node_feature_names - var_config["node_feature_dims"] = node_feature_dims + + # Note: Feature configuration now handled automatically in update_config() + # If using legacy format, you can still override features here if needed: + # var_config["graph_feature_names"] = ["energy"] + # var_config["graph_feature_dims"] = [1] + # var_config["node_feature_names"] = ["atomic_number", "cartesian_coordinates", "forces"] + # var_config["node_feature_dims"] = [1, 3, 3] if args.batch_size is not None: config["NeuralNetwork"]["Training"]["batch_size"] = args.batch_size diff --git a/examples/multibranch_hpo/train.py b/examples/multibranch_hpo/train.py index 1d488a6fb..c3efbf0fc 100644 --- a/examples/multibranch_hpo/train.py +++ b/examples/multibranch_hpo/train.py @@ -123,10 +123,6 @@ def info(*args, logtype="info", sep=" "): args = parser.parse_args() args.parameters = vars(args) - graph_feature_names = ["energy"] - graph_feature_dims = [1] - node_feature_names = ["atomic_number" "cartesian_coordinates", "forces"] - node_feature_dims = [1, 3, 3] dirpwd = os.path.dirname(os.path.abspath(__file__)) datadir = os.path.join(dirpwd, "dataset") ################################################################################################################## @@ -137,10 +133,14 @@ def info(*args, logtype="info", sep=" "): config = json.load(f) verbosity = config["Verbosity"]["level"] var_config = config["NeuralNetwork"]["Variables_of_interest"] - var_config["graph_feature_names"] = graph_feature_names - var_config["graph_feature_dims"] = graph_feature_dims - var_config["node_feature_names"] = node_feature_names - var_config["node_feature_dims"] = node_feature_dims + + # Note: Feature configuration now handled automatically in update_config() + # If using legacy format, you can still override features here if needed: + # var_config["graph_feature_names"] = ["energy"] + # var_config["graph_feature_dims"] = [1] + # var_config["node_feature_names"] = ["atomic_number", "cartesian_coordinates", "forces"] + # var_config["node_feature_dims"] = [1, 3, 3] + if args.parameters["mpnn_type"] == "MACE": # Currently, MACE only supports atomic number node attributes, see hydragnn/models/MACEStack.py var_config["input_node_features"] = [0] diff --git a/examples/multidataset/gfm_energy.json b/examples/multidataset/gfm_energy.json index eef9c4ec7..3504e5590 100644 --- a/examples/multidataset/gfm_energy.json +++ b/examples/multidataset/gfm_energy.json @@ -32,11 +32,30 @@ "task_weights": [1.0] }, "Variables_of_interest": { - "input_node_features": [0, 1, 2, 3], - "output_names": ["energy"], - "output_index": [0], - "output_dim": [1], - "type": ["graph"] + "node_features": { + "atomic_number": { + "dim": 1, + "role": "input", + "description": "Atomic number (element type)" + }, + "cartesian_coordinates": { + "dim": 3, + "role": "input", + "description": "3D atomic positions" + }, + "forces": { + "dim": 3, + "role": "output", + "description": "Atomic forces" + } + }, + "graph_features": { + "energy": { + "dim": 1, + "role": "output", + "description": "Total energy" + } + } }, "Training": { "num_epoch": 50, diff --git a/examples/multidataset/train.py b/examples/multidataset/train.py index 262647e38..2183075cc 100644 --- a/examples/multidataset/train.py +++ b/examples/multidataset/train.py @@ -94,10 +94,6 @@ def info(*args, logtype="info", sep=" "): parser.set_defaults(format="adios") args = parser.parse_args() - graph_feature_names = ["energy"] - graph_feature_dims = [1] - node_feature_names = ["atomic_number", "cartesian_coordinates", "forces"] - node_feature_dims = [1, 3, 3] dirpwd = os.path.dirname(os.path.abspath(__file__)) datadir = os.path.join(dirpwd, "dataset") ################################################################################################################## @@ -108,10 +104,9 @@ def info(*args, logtype="info", sep=" "): config = json.load(f) verbosity = config["Verbosity"]["level"] var_config = config["NeuralNetwork"]["Variables_of_interest"] - var_config["graph_feature_names"] = graph_feature_names - var_config["graph_feature_dims"] = graph_feature_dims - var_config["node_feature_names"] = node_feature_names - var_config["node_feature_dims"] = node_feature_dims + + # Note: Feature configuration now handled automatically in update_config() + # Features are defined in the JSON config file if args.batch_size is not None: config["NeuralNetwork"]["Training"]["batch_size"] = args.batch_size diff --git a/examples/multidataset_deepspeed/train.py b/examples/multidataset_deepspeed/train.py index 4c6d5bc12..e8eb2c4f4 100644 --- a/examples/multidataset_deepspeed/train.py +++ b/examples/multidataset_deepspeed/train.py @@ -155,10 +155,6 @@ def info(*args, logtype="info", sep=" "): args = parser.parse_args() args.parameters = vars(args) - graph_feature_names = ["energy"] - graph_feature_dims = [1] - node_feature_names = ["atomic_number", "cartesian_coordinates", "forces"] - node_feature_dims = [1, 3, 3] dirpwd = os.path.dirname(os.path.abspath(__file__)) ################################################################################################################## input_filename = os.path.join(dirpwd, args.inputfile) @@ -168,10 +164,13 @@ def info(*args, logtype="info", sep=" "): config = json.load(f) verbosity = config["Verbosity"]["level"] var_config = config["NeuralNetwork"]["Variables_of_interest"] - var_config["graph_feature_names"] = graph_feature_names - var_config["graph_feature_dims"] = graph_feature_dims - var_config["node_feature_names"] = node_feature_names - var_config["node_feature_dims"] = node_feature_dims + + # Note: Feature configuration now handled automatically in update_config() + # If using legacy format, you can still override features here if needed: + # var_config["graph_feature_names"] = ["energy"] + # var_config["graph_feature_dims"] = [1] + # var_config["node_feature_names"] = ["atomic_number", "cartesian_coordinates", "forces"] + # var_config["node_feature_dims"] = [1, 3, 3] if args.batch_size is not None: config["NeuralNetwork"]["Training"]["batch_size"] = args.batch_size diff --git a/examples/open_catalyst_2020/open_catalyst_energy.json b/examples/open_catalyst_2020/open_catalyst_energy.json index 600395c4f..15dd40292 100644 --- a/examples/open_catalyst_2020/open_catalyst_energy.json +++ b/examples/open_catalyst_2020/open_catalyst_energy.json @@ -32,11 +32,26 @@ "task_weights": [1.0] }, "Variables_of_interest": { - "input_node_features": [0, 1, 2, 3], - "output_names": ["energy"], - "output_index": [0], - "output_dim": [1], - "type": ["graph"] + "node_features": { + "atomic_number": { + "dim": 1, + "role": "input" + }, + "cartesian_coordinates": { + "dim": 3, + "role": "input" + }, + "forces": { + "dim": 3, + "role": "output" + } + }, + "graph_features": { + "energy": { + "dim": 1, + "role": "output" + } + } }, "Training": { "num_epoch": 50, diff --git a/examples/open_catalyst_2020/open_catalyst_forces.json b/examples/open_catalyst_2020/open_catalyst_forces.json index 994c8f423..bc1487ee6 100644 --- a/examples/open_catalyst_2020/open_catalyst_forces.json +++ b/examples/open_catalyst_2020/open_catalyst_forces.json @@ -31,11 +31,26 @@ "task_weights": [1.0] }, "Variables_of_interest": { - "input_node_features": [0, 1, 2, 3], - "output_names": ["forces"], - "output_index": [2], - "output_dim": [3], - "type": ["node"] + "node_features": { + "atomic_number": { + "dim": 1, + "role": "input" + }, + "cartesian_coordinates": { + "dim": 3, + "role": "input" + }, + "forces": { + "dim": 3, + "role": "output" + } + }, + "graph_features": { + "energy": { + "dim": 1, + "role": "output" + } + } }, "Training": { "num_epoch": 50, diff --git a/examples/open_catalyst_2020/train.py b/examples/open_catalyst_2020/train.py index c1895c6a9..a2f5f83cd 100644 --- a/examples/open_catalyst_2020/train.py +++ b/examples/open_catalyst_2020/train.py @@ -213,10 +213,6 @@ def get(self, idx): parser.set_defaults(format="adios") args = parser.parse_args() - graph_feature_names = ["energy"] - graph_feature_dims = [1] - node_feature_names = ["atomic_number", "cartesian_coordinates", "forces"] - node_feature_dims = [1, 3, 3] dirpwd = os.path.dirname(os.path.abspath(__file__)) datadir = os.path.join(dirpwd, "dataset") ################################################################################################################## @@ -227,10 +223,13 @@ def get(self, idx): config = json.load(f) verbosity = config["Verbosity"]["level"] var_config = config["NeuralNetwork"]["Variables_of_interest"] - var_config["graph_feature_names"] = graph_feature_names - var_config["graph_feature_dims"] = graph_feature_dims - var_config["node_feature_names"] = node_feature_names - var_config["node_feature_dims"] = node_feature_dims + + # Note: Feature configuration now handled automatically in update_config() + # If using legacy format, you can still override features here if needed: + # var_config["graph_feature_names"] = ["energy"] + # var_config["graph_feature_dims"] = [1] + # var_config["node_feature_names"] = ["atomic_number", "cartesian_coordinates", "forces"] + # var_config["node_feature_dims"] = [1, 3, 3] # Transformation to create positional and structural laplacian encoders """ diff --git a/examples/open_catalyst_2022/train.py b/examples/open_catalyst_2022/train.py index 05d756976..e030042fe 100644 --- a/examples/open_catalyst_2022/train.py +++ b/examples/open_catalyst_2022/train.py @@ -341,10 +341,6 @@ def get(self, idx): parser.set_defaults(format="adios") args = parser.parse_args() - graph_feature_names = ["energy"] - graph_feature_dims = [1] - node_feature_names = ["atomic_number", "cartesian_coordinates", "forces"] - node_feature_dims = [1, 3, 3] dirpwd = os.path.dirname(os.path.abspath(__file__)) datadir = os.path.join(dirpwd, "dataset") ################################################################################################################## @@ -355,10 +351,13 @@ def get(self, idx): config = json.load(f) verbosity = config["Verbosity"]["level"] var_config = config["NeuralNetwork"]["Variables_of_interest"] - var_config["graph_feature_names"] = graph_feature_names - var_config["graph_feature_dims"] = graph_feature_dims - var_config["node_feature_names"] = node_feature_names - var_config["node_feature_dims"] = node_feature_dims + + # Note: Feature configuration now handled automatically in update_config() + # If using legacy format, you can still override features here if needed: + # var_config["graph_feature_names"] = ["energy"] + # var_config["graph_feature_dims"] = [1] + # var_config["node_feature_names"] = ["atomic_number", "cartesian_coordinates", "forces"] + # var_config["node_feature_dims"] = [1, 3, 3] # Transformation to create positional and structural laplacian encoders """ diff --git a/examples/open_materials_2024/omat24_energy.json b/examples/open_materials_2024/omat24_energy.json index 600395c4f..a77968d15 100644 --- a/examples/open_materials_2024/omat24_energy.json +++ b/examples/open_materials_2024/omat24_energy.json @@ -32,11 +32,29 @@ "task_weights": [1.0] }, "Variables_of_interest": { - "input_node_features": [0, 1, 2, 3], - "output_names": ["energy"], - "output_index": [0], - "output_dim": [1], - "type": ["graph"] + "node_features": { + "atomic_number": { + "dim": 1, + "role": "input", + "description": "Atomic number (element type)" + }, + "cartesian_coordinates": { + "dim": 3, + "role": "input" + }, + "forces": { + "dim": 3, + "role": "both", + "description": "Atomic forces" + } + }, + "graph_features": { + "energy": { + "dim": 1, + "role": "output", + "description": "Total energy of the system" + } + } }, "Training": { "num_epoch": 50, diff --git a/examples/open_materials_2024/train.py b/examples/open_materials_2024/train.py index ab6658b9e..7be7d1b83 100644 --- a/examples/open_materials_2024/train.py +++ b/examples/open_materials_2024/train.py @@ -100,10 +100,6 @@ def info(*args, logtype="info", sep=" "): parser.set_defaults(format="adios") args = parser.parse_args() - graph_feature_names = ["energy"] - graph_feature_dims = [1] - node_feature_names = ["atomic_number", "cartesian_coordinates", "forces"] - node_feature_dims = [1, 3, 3] dirpwd = os.path.dirname(os.path.abspath(__file__)) datadir = os.path.join(dirpwd, "dataset") ################################################################################################################## @@ -114,10 +110,13 @@ def info(*args, logtype="info", sep=" "): config = json.load(f) verbosity = config["Verbosity"]["level"] var_config = config["NeuralNetwork"]["Variables_of_interest"] - var_config["graph_feature_names"] = graph_feature_names - var_config["graph_feature_dims"] = graph_feature_dims - var_config["node_feature_names"] = node_feature_names - var_config["node_feature_dims"] = node_feature_dims + + # Note: Feature configuration now handled automatically in update_config() + # If using legacy format, you can still override features here if needed: + # var_config["graph_feature_names"] = ["energy"] + # var_config["graph_feature_dims"] = [1] + # var_config["node_feature_names"] = ["atomic_number", "cartesian_coordinates", "forces"] + # var_config["node_feature_dims"] = [1, 3, 3] # Transformation to create positional and structural laplacian encoders """ diff --git a/examples/open_molecules_2025/omol25_energy.json b/examples/open_molecules_2025/omol25_energy.json index 600395c4f..15dd40292 100644 --- a/examples/open_molecules_2025/omol25_energy.json +++ b/examples/open_molecules_2025/omol25_energy.json @@ -32,11 +32,26 @@ "task_weights": [1.0] }, "Variables_of_interest": { - "input_node_features": [0, 1, 2, 3], - "output_names": ["energy"], - "output_index": [0], - "output_dim": [1], - "type": ["graph"] + "node_features": { + "atomic_number": { + "dim": 1, + "role": "input" + }, + "cartesian_coordinates": { + "dim": 3, + "role": "input" + }, + "forces": { + "dim": 3, + "role": "output" + } + }, + "graph_features": { + "energy": { + "dim": 1, + "role": "output" + } + } }, "Training": { "num_epoch": 50, diff --git a/examples/open_molecules_2025/train.py b/examples/open_molecules_2025/train.py index 21b660e30..ee4192bf1 100644 --- a/examples/open_molecules_2025/train.py +++ b/examples/open_molecules_2025/train.py @@ -121,10 +121,6 @@ def info(*args, logtype="info", sep=" "): parser.set_defaults(format="adios") args = parser.parse_args() - graph_feature_names = ["energy"] - graph_feature_dims = [1] - node_feature_names = ["atomic_number", "cartesian_coordinates", "forces"] - node_feature_dims = [1, 3, 3] dirpwd = os.path.dirname(os.path.abspath(__file__)) datadir = os.path.join(dirpwd, "dataset") @@ -136,10 +132,13 @@ def info(*args, logtype="info", sep=" "): config = json.load(f) verbosity = config["Verbosity"]["level"] var_config = config["NeuralNetwork"]["Variables_of_interest"] - var_config["graph_feature_names"] = graph_feature_names - var_config["graph_feature_dims"] = graph_feature_dims - var_config["node_feature_names"] = node_feature_names - var_config["node_feature_dims"] = node_feature_dims + + # Note: Feature configuration now handled automatically in update_config() + # If using legacy format, you can still override features here if needed: + # var_config["graph_feature_names"] = ["energy"] + # var_config["graph_feature_dims"] = [1] + # var_config["node_feature_names"] = ["atomic_number", "cartesian_coordinates", "forces"] + # var_config["node_feature_dims"] = [1, 3, 3] # Transformation to create positional and structural laplacian encoders """ diff --git a/examples/qcml/train.py b/examples/qcml/train.py index 35c8a4a00..c29a4dc43 100644 --- a/examples/qcml/train.py +++ b/examples/qcml/train.py @@ -286,10 +286,6 @@ def get(self, idx): parser.set_defaults(format="adios") args = parser.parse_args() - graph_feature_names = ["energy"] - graph_feature_dims = [1] - node_feature_names = ["atomic_number", "cartesian_coordinates", "forces"] - node_feature_dims = [1, 3, 3] dirpwd = os.path.dirname(os.path.abspath(__file__)) datadir = os.path.join(dirpwd, "dataset") ################################################################################################################## @@ -300,10 +296,13 @@ def get(self, idx): config = json.load(f) verbosity = config["Verbosity"]["level"] var_config = config["NeuralNetwork"]["Variables_of_interest"] - var_config["graph_feature_names"] = graph_feature_names - var_config["graph_feature_dims"] = graph_feature_dims - var_config["node_feature_names"] = node_feature_names - var_config["node_feature_dims"] = node_feature_dims + + # Note: Feature configuration now handled automatically in update_config() + # If using legacy format, you can still override features here if needed: + # var_config["graph_feature_names"] = ["energy"] + # var_config["graph_feature_dims"] = [1] + # var_config["node_feature_names"] = ["atomic_number", "cartesian_coordinates", "forces"] + # var_config["node_feature_dims"] = [1, 3, 3] # Transformation to create positional and structural laplacian encoders """ diff --git a/examples/qm7x/train.py b/examples/qm7x/train.py index 659532064..5107a48ae 100644 --- a/examples/qm7x/train.py +++ b/examples/qm7x/train.py @@ -362,17 +362,6 @@ def get(self, idx): parser.set_defaults(format="adios") args = parser.parse_args() - graph_feature_names = ["energy"] - graph_feature_dims = [1] - node_feature_names = [ - "atomic_number", - "coordinates", - "forces", - "hCHG", - "hVDIP", - "hRAT", - ] - node_feature_dims = [1, 3, 3, 1, 1, 1] dirpwd = os.path.dirname(os.path.abspath(__file__)) datadir = os.path.join(dirpwd, "dataset/QM7-X") ################################################################################################################## @@ -383,10 +372,9 @@ def get(self, idx): config = json.load(f) verbosity = config["Verbosity"]["level"] var_config = config["NeuralNetwork"]["Variables_of_interest"] - var_config["graph_feature_names"] = graph_feature_names - var_config["graph_feature_dims"] = graph_feature_dims - var_config["node_feature_names"] = node_feature_names - var_config["node_feature_dims"] = node_feature_dims + + # Note: Feature configuration now handled automatically in update_config() + # If using legacy format, you can still override features here if needed # Transformation to create positional and structural laplacian encoders """ diff --git a/examples/qm9/qm9.json b/examples/qm9/qm9.json index 5a2d083f2..2f17aefa5 100644 --- a/examples/qm9/qm9.json +++ b/examples/qm9/qm9.json @@ -38,11 +38,18 @@ "task_weights": [1.0] }, "Variables_of_interest": { - "input_node_features": [0], - "output_names": ["free_energy"], - "output_index": [0], - "output_dim": [1], - "type": ["graph"], + "node_features": { + "atomic_number": { + "dim": 1, + "role": "input" + } + }, + "graph_features": { + "free_energy": { + "dim": 1, + "role": "output" + } + }, "denormalize_output": false }, "Training": { diff --git a/examples/qm9/qm9.py b/examples/qm9/qm9.py index cfdc9cd45..8c5dc0ab3 100644 --- a/examples/qm9/qm9.py +++ b/examples/qm9/qm9.py @@ -69,7 +69,6 @@ def main(mpnn_type=None, global_attn_engine=None, global_attn_type=None): config["NeuralNetwork"]["Architecture"]["mpnn_type"] = mpnn_type verbosity = config["Verbosity"]["level"] - var_config = config["NeuralNetwork"]["Variables_of_interest"] # Always initialize for multi-rank training. world_size, world_rank = hydragnn.utils.distributed.setup_ddp() diff --git a/examples/transition1x/train.py b/examples/transition1x/train.py index 8a874dfcf..cbf6907b1 100644 --- a/examples/transition1x/train.py +++ b/examples/transition1x/train.py @@ -313,17 +313,6 @@ def get(self, idx): parser.set_defaults(format="adios") args = parser.parse_args() - graph_feature_names = ["energy"] - graph_feature_dims = [1] - node_feature_names = [ - "atomic_number", - "coordinates", - "forces", - "hCHG", - "hVDIP", - "hRAT", - ] - node_feature_dims = [1, 3, 3] dirpwd = os.path.dirname(os.path.abspath(__file__)) datadir = os.path.join(dirpwd, "dataset") ################################################################################################################## @@ -334,10 +323,9 @@ def get(self, idx): config = json.load(f) verbosity = config["Verbosity"]["level"] var_config = config["NeuralNetwork"]["Variables_of_interest"] - var_config["graph_feature_names"] = graph_feature_names - var_config["graph_feature_dims"] = graph_feature_dims - var_config["node_feature_names"] = node_feature_names - var_config["node_feature_dims"] = node_feature_dims + + # Note: Feature configuration now handled automatically in update_config() + # If using legacy format, you can still override features here if needed # Transformation to create positional and structural laplacian encoders """ diff --git a/examples/zinc/zinc.json b/examples/zinc/zinc.json index 6835f113f..8c569126a 100644 --- a/examples/zinc/zinc.json +++ b/examples/zinc/zinc.json @@ -39,11 +39,18 @@ "task_weights": [1.0] }, "Variables_of_interest": { - "input_node_features": [0], - "output_names": ["free_energy"], - "output_index": [0], - "output_dim": [1], - "type": ["graph"], + "node_features": { + "atomic_number": { + "dim": 1, + "role": "input" + } + }, + "graph_features": { + "free_energy": { + "dim": 1, + "role": "output" + } + }, "denormalize_output": false }, "Training": { diff --git a/examples/zinc/zinc.py b/examples/zinc/zinc.py index 3a1fc126c..94702fe4b 100644 --- a/examples/zinc/zinc.py +++ b/examples/zinc/zinc.py @@ -27,7 +27,6 @@ with open(filename, "r") as f: config = json.load(f) verbosity = config["Verbosity"]["level"] -var_config = config["NeuralNetwork"]["Variables_of_interest"] # Always initialize for multi-rank training. world_size, world_rank = hydragnn.utils.distributed.setup_ddp() diff --git a/hydragnn/preprocess/graph_samples_checks_and_updates.py b/hydragnn/preprocess/graph_samples_checks_and_updates.py index 1cd4315df..66442e8b8 100644 --- a/hydragnn/preprocess/graph_samples_checks_and_updates.py +++ b/hydragnn/preprocess/graph_samples_checks_and_updates.py @@ -556,11 +556,11 @@ def update_predicted_values( def update_atom_features(atom_features: [AtomFeatures], data: Data): """Updates atom features of a structure. An atom is represented with x,y,z coordinates and associated features. - Parameters ---------- atom_features: [AtomFeatures] List of features to update. Each feature is instance of Enum AtomFeatures. + List of feature indices to keep as inputs. data: Data A Data object representing a structure that has atoms. """ diff --git a/hydragnn/preprocess/load_data.py b/hydragnn/preprocess/load_data.py index bf58f2b59..e85e5edf7 100644 --- a/hydragnn/preprocess/load_data.py +++ b/hydragnn/preprocess/load_data.py @@ -26,6 +26,9 @@ from hydragnn.preprocess.serialized_dataset_loader import SerializedDataLoader from hydragnn.preprocess.lsms_raw_dataset_loader import LSMS_RawDataLoader from hydragnn.preprocess.cfg_raw_dataset_loader import CFG_RawDataLoader +from hydragnn.utils.input_config_parsing.feature_config import ( + update_var_config_with_features, +) from hydragnn.utils.datasets.compositional_data_splitting import ( compositional_stratified_splitting, ) @@ -204,6 +207,12 @@ def __next__(self): def dataset_loading_and_splitting(config: {}): + # Parse and populate var_config with legacy keys if using new format + # This must happen before SerializedDataLoader is instantiated + var_config = config["NeuralNetwork"]["Variables_of_interest"] + var_config = update_var_config_with_features(var_config) + config["NeuralNetwork"]["Variables_of_interest"] = var_config + ##check if serialized pickle files or folders for raw files provided if not list(config["Dataset"]["path"].values())[0].endswith(".pkl"): transform_raw_data_to_serialized(config["Dataset"]) diff --git a/hydragnn/run_training.py b/hydragnn/run_training.py index b91cf98ad..05203fea6 100644 --- a/hydragnn/run_training.py +++ b/hydragnn/run_training.py @@ -34,6 +34,9 @@ save_config, parse_deepspeed_config, ) +from hydragnn.utils.input_config_parsing.feature_config import ( + update_var_config_with_features, +) from hydragnn.utils.optimizer import select_optimizer from hydragnn.models.create import create_model_config from hydragnn.train.train_validate_test import train_validate_test @@ -62,6 +65,12 @@ def _(config_file: str, use_deepspeed=False): @run_training.register def _(config: dict, use_deepspeed=False): + # Parse and populate var_config with legacy keys if using new format + # This must happen FIRST, before any other function tries to access var_config + var_config = config["NeuralNetwork"]["Variables_of_interest"] + var_config = update_var_config_with_features(var_config) + config["NeuralNetwork"]["Variables_of_interest"] = var_config + try: os.environ["SERIALIZED_DATA_PATH"] except: diff --git a/hydragnn/utils/input_config_parsing/__init__.py b/hydragnn/utils/input_config_parsing/__init__.py index 5f2c07a19..367cd0338 100644 --- a/hydragnn/utils/input_config_parsing/__init__.py +++ b/hydragnn/utils/input_config_parsing/__init__.py @@ -7,3 +7,12 @@ save_config, parse_deepspeed_config, ) + +from .feature_config import ( + parse_feature_config, + validate_feature_config, + update_var_config_with_features, + print_feature_summary, + get_feature_schema_example, + validate_node_feature_columns, +) diff --git a/hydragnn/utils/input_config_parsing/config_utils.py b/hydragnn/utils/input_config_parsing/config_utils.py index a53327f0b..1e174e075 100644 --- a/hydragnn/utils/input_config_parsing/config_utils.py +++ b/hydragnn/utils/input_config_parsing/config_utils.py @@ -17,6 +17,11 @@ from hydragnn.utils.model.model import calculate_avg_deg from hydragnn.utils.distributed import get_comm_size_and_rank from hydragnn.utils.model import update_multibranch_heads +from hydragnn.utils.input_config_parsing.feature_config import ( + parse_feature_config, + validate_feature_config, + update_var_config_with_features, +) from copy import deepcopy import warnings import json @@ -26,6 +31,21 @@ def update_config(config, train_loader, val_loader, test_loader): """check if config input consistent and update config with model and datasets""" + # Parse and validate feature configuration (new feature config system) + var_config = config["NeuralNetwork"]["Variables_of_interest"] + + # Update var_config with parsed features (supports both new and legacy formats) + var_config = update_var_config_with_features(var_config) + config["NeuralNetwork"]["Variables_of_interest"] = var_config + + # Validate feature configuration against actual data + is_valid, errors = validate_feature_config(var_config, train_loader.dataset[0]) + if not is_valid: + warnings.warn( + f"Feature configuration validation warnings:\n" + + "\n".join(f" - {e}" for e in errors) + ) + graph_size_variable = os.getenv("HYDRAGNN_USE_VARIABLE_GRAPH_SIZE") if graph_size_variable is None: graph_size_variable = check_if_graph_size_variable( diff --git a/hydragnn/utils/input_config_parsing/feature_config.py b/hydragnn/utils/input_config_parsing/feature_config.py new file mode 100644 index 000000000..ffe5916e8 --- /dev/null +++ b/hydragnn/utils/input_config_parsing/feature_config.py @@ -0,0 +1,467 @@ +############################################################################## +# Copyright (c) 2024, Oak Ridge National Laboratory # +# All rights reserved. # +# # +# This file is part of HydraGNN and is distributed under a BSD 3-clause # +# license. For the licensing terms see the LICENSE file in the top-level # +# directory. # +# # +# SPDX-License-Identifier: BSD-3-Clause # +############################################################################## + +""" +Feature configuration parsing and validation utilities. + +This module provides utilities to parse and validate feature configurations +from the Variables_of_interest section in config files. It supports both: +1. Legacy format: Using separate lists for feature names and dimensions +2. New format: Using structured dictionaries with explicit feature roles + + +IMPORTANT: The order of node_features and graph_features in the config +must match the order of columns/features in the actual dataset files. +The code assumes that the config order corresponds exactly to the data order +when mapping feature names and indices. + +Example new format: +{ + "node_features": { + "atomic_number": {"dim": 1, "role": "input"}, + "cartesian_coordinates": {"dim": 3, "role": "input"}, + "forces": {"dim": 3, "role": "output", "output_type": "node"} + }, + "graph_features": { + "energy": {"dim": 1, "role": "output", "output_type": "graph"} + } +} + +Example legacy format: +{ + "node_feature_names": ["atomic_number", "cartesian_coordinates", "forces"], + "node_feature_dims": [1, 3, 3], + "graph_feature_names": ["energy"], + "graph_feature_dims": [1], + "input_node_features": [0, 1, 2, 3], + "output_names": ["energy"], + "output_index": [0], + "output_dim": [1], + "type": ["graph"] +} +""" + +from typing import Dict, List, Tuple, Optional +import warnings + + +def parse_feature_config(var_config: dict) -> dict: + """ + Parse Variables_of_interest config and extract feature information. + + Supports both legacy and new formats: + - Legacy: Uses graph_feature_names, node_feature_names (if present) + - New: Uses node_features, graph_features dicts + + Args: + var_config: Variables_of_interest section from config + + Returns: + Standardized dict with: + - node_feature_names: List[str] + - node_feature_dims: List[int] + - graph_feature_names: List[str] + - graph_feature_dims: List[int] + - input_node_features: List[int] (indices) + - output_names: List[str] + - output_index: List[int] + - output_dim: List[int] + - type: List[str] ('graph' or 'node') + + Raises: + ValueError: If config format is invalid + """ + result = {} + + # Check if using new format + if "node_features" in var_config or "graph_features" in var_config: + result = _parse_new_format(var_config) + # Check if using legacy format + elif "node_feature_names" in var_config or "graph_feature_names" in var_config: + result = _parse_legacy_format(var_config) + else: + raise ValueError( + "Config must contain either 'node_features'/'graph_features' (new format) " + "or 'node_feature_names'/'graph_feature_names' (legacy format)" + ) + + return result + + +def _parse_new_format(var_config: dict) -> dict: + """ + Parse new feature configuration format. + + The new format uses dictionaries with explicit feature roles: + - role: 'input' or 'output' + - dim: integer dimension + - output_type: 'graph' or 'node' (for output features) + + Args: + var_config: Variables_of_interest section + + Returns: + Standardized feature dict + """ + result = { + "node_feature_names": [], + "node_feature_dims": [], + "graph_feature_names": [], + "graph_feature_dims": [], + "input_node_features": [], + "output_names": [], + "output_index": [], + "output_dim": [], + "type": [], + } + + # Parse node features + if "node_features" in var_config: + node_idx = 0 + for feat_name, feat_config in var_config["node_features"].items(): + result["node_feature_names"].append(feat_name) + dim = feat_config["dim"] + result["node_feature_dims"].append(dim) + + # Track input features + role = feat_config.get("role", "input") + if role == "input": + start_idx = sum(result["node_feature_dims"][:node_idx]) + result["input_node_features"].extend(range(start_idx, start_idx + dim)) + + # Track output features + if role == "output": + result["output_names"].append(feat_name) + result["output_index"].append(node_idx) + result["output_dim"].append(feat_config["dim"]) + result["type"].append(feat_config.get("output_type", "node")) + + node_idx += 1 + + # Parse graph features + if "graph_features" in var_config: + graph_idx = 0 + for feat_name, feat_config in var_config["graph_features"].items(): + result["graph_feature_names"].append(feat_name) + result["graph_feature_dims"].append(feat_config["dim"]) + + # Graph features are outputs + role = feat_config.get("role", "output") + if role == "output": + result["output_names"].append(feat_name) + result["output_index"].append(graph_idx) + result["output_dim"].append(feat_config["dim"]) + result["type"].append(feat_config.get("output_type", "graph")) + + graph_idx += 1 + + return result + + +def _parse_legacy_format(var_config: dict) -> dict: + """ + Parse legacy feature configuration format. + + The legacy format uses separate lists for names, dimensions, and indices. + + Args: + var_config: Variables_of_interest section + + Returns: + Standardized feature dict + """ + result = { + "node_feature_names": var_config.get("node_feature_names", []), + "node_feature_dims": var_config.get("node_feature_dims", []), + "graph_feature_names": var_config.get("graph_feature_names", []), + "graph_feature_dims": var_config.get("graph_feature_dims", []), + "input_node_features": var_config.get("input_node_features", []), + "output_names": var_config.get("output_names", []), + "output_index": var_config.get("output_index", []), + "output_dim": var_config.get("output_dim", []), + "type": var_config.get("type", []), + } + + return result + + +def validate_feature_config( + var_config: dict, data_object=None +) -> Tuple[bool, List[str]]: + """ + Validate feature configuration for consistency. + + Checks: + 1. Feature names and dimensions lists have matching lengths + 2. Output configuration is consistent + 3. Indices are within valid ranges + 4. If data_object provided, validates against actual data dimensions + + Args: + var_config: Variables_of_interest section from config + data_object: Optional PyG Data object to validate against + + Returns: + (is_valid, list_of_errors) + - is_valid: True if no errors found + - list_of_errors: List of error messages (empty if valid) + """ + errors = [] + + # Parse config + try: + parsed = parse_feature_config(var_config) + except Exception as e: + return False, [f"Failed to parse config: {str(e)}"] + + # Check lengths match + if len(parsed["node_feature_names"]) != len(parsed["node_feature_dims"]): + errors.append( + f"node_feature_names length ({len(parsed['node_feature_names'])}) " + f"!= node_feature_dims length ({len(parsed['node_feature_dims'])})" + ) + + if len(parsed["graph_feature_names"]) != len(parsed["graph_feature_dims"]): + errors.append( + f"graph_feature_names length ({len(parsed['graph_feature_names'])}) " + f"!= graph_feature_dims length ({len(parsed['graph_feature_dims'])})" + ) + + # Check output configuration consistency + output_lengths = [ + len(parsed["output_names"]), + len(parsed["output_index"]), + len(parsed["output_dim"]), + len(parsed["type"]), + ] + if len(set(output_lengths)) > 1: + errors.append( + f"Output config lengths don't match: " + f"names={output_lengths[0]}, " + f"index={output_lengths[1]}, " + f"dim={output_lengths[2]}, " + f"type={output_lengths[3]}" + ) + + # Check that input_node_features indices are valid + max_node_idx = sum(parsed["node_feature_dims"]) - 1 + if max_node_idx >= 0: # Only check if we have node features + for idx in parsed["input_node_features"]: + if idx > max_node_idx: + errors.append( + f"input_node_features contains index {idx}, " + f"but only {max_node_idx + 1} node features defined" + ) + + # Check that output_index values are valid + for i, (output_type, output_idx) in enumerate( + zip(parsed["type"], parsed["output_index"]) + ): + if output_type == "graph": + max_idx = len(parsed["graph_feature_names"]) - 1 + if output_idx > max_idx: + errors.append( + f"output_index[{i}]={output_idx} (type='graph'), " + f"but only {len(parsed['graph_feature_names'])} graph features defined" + ) + elif output_type == "node": + max_idx = len(parsed["node_feature_names"]) - 1 + if output_idx > max_idx: + errors.append( + f"output_index[{i}]={output_idx} (type='node'), " + f"but only {len(parsed['node_feature_names'])} node features defined" + ) + + # Validate against data object if provided + if data_object is not None: + data_errors = _validate_against_data(parsed, data_object) + errors.extend(data_errors) + + return len(errors) == 0, errors + + +def _validate_against_data(parsed: dict, data_object) -> List[str]: + """ + Validate parsed config against actual data object. + + Args: + parsed: Parsed feature configuration + data_object: PyG Data object + + Returns: + List of validation error messages + """ + errors = [] + + # Check x tensor dimensions + if hasattr(data_object, "x") and data_object.x is not None: + expected_x_dim = sum(parsed["node_feature_dims"]) + actual_x_dim = data_object.x.shape[1] if len(data_object.x.shape) > 1 else 1 + if expected_x_dim != actual_x_dim: + errors.append( + f"Expected x dimension {expected_x_dim} based on node_feature_dims " + f"{parsed['node_feature_dims']}, but got {actual_x_dim} in data" + ) + + # Check that features referenced in input_node_features actually exist in data + if hasattr(data_object, "x") and parsed["input_node_features"]: + # Verify we can extract the features + try: + for feat_idx in parsed["input_node_features"]: + # Just checking we can slice - don't need the actual data + if hasattr(data_object.x, "shape"): + if feat_idx >= data_object.x.shape[1]: + errors.append( + f"Feature idx {feat_idx} in input_node_features " + f"but x only has {data_object.x.shape[1]} columns" + ) + except Exception as e: + errors.append(f"Error validating input features against data: {str(e)}") + + return errors + + +def validate_node_feature_columns(data_x, node_feature_dims, column_indices=None): + """ + Validate that the columns in data.x match the expected node features from config. + If column_indices is None, checks that data.x.shape[1] == sum(node_feature_dims). + If data.x has more columns than needed and column_indices is not set, raises ValueError. + If column_indices is set, checks that all indices and their ranges are valid for the dims. + """ + expected_total_dim = sum(node_feature_dims) + if column_indices is None: + if data_x.shape[1] != expected_total_dim: + raise ValueError( + f"Mismatch between data.x columns ({data_x.shape[1]}) and total node feature dims ({expected_total_dim}). " + "If your data.x has extra columns, specify 'column_index' for each feature in the config." + ) + else: + # If column_indices is set, check that all indices and their ranges are valid + for idx, dim in zip(column_indices, node_feature_dims): + if idx + dim > data_x.shape[1]: + raise ValueError( + f"data.x does not have enough columns for feature starting at index {idx} with dim {dim} (shape {data_x.shape[1]})." + ) + + +def update_var_config_with_features(var_config: dict) -> dict: + """ + Update var_config dict with parsed feature information. + + This ensures backward compatibility by adding legacy keys if they don't exist. + If using new format (node_features/graph_features), this will populate the + legacy keys (node_feature_names, etc.) automatically. + + Args: + var_config: Variables_of_interest section (will be modified in-place) + + Returns: + Updated var_config (same object, modified in-place) + """ + parsed = parse_feature_config(var_config) + + # Update with parsed values (ensuring legacy keys exist) + for key, value in parsed.items(): + if key not in var_config: + var_config[key] = value + + return var_config + + +def get_feature_schema_example() -> dict: + """ + Return an example of the new feature configuration format. + + Returns: + Example dict showing the new feature configuration format + """ + return { + "node_features": { + "atomic_number": { + "dim": 1, + "role": "input", + "description": "Atomic number (Z)", + }, + "cartesian_coordinates": { + "dim": 3, + "role": "input", + "description": "3D position in Cartesian coordinates", + }, + "forces": { + "dim": 3, + "role": "output", + "output_type": "node", + "description": "Atomic forces", + }, + }, + "graph_features": { + "energy": { + "dim": 1, + "role": "output", + "output_type": "graph", + "description": "Total energy", + } + }, + } + + +def print_feature_summary(var_config: dict) -> str: + """ + Generate a human-readable summary of feature configuration. + + Args: + var_config: Variables_of_interest section + + Returns: + Formatted string summarizing the features + """ + try: + parsed = parse_feature_config(var_config) + except Exception as e: + return f"Error parsing config: {str(e)}" + + lines = [] + lines.append("=" * 60) + lines.append("Feature Configuration Summary") + lines.append("=" * 60) + + # Node features + if parsed["node_feature_names"]: + lines.append("\nNode Features:") + lines.append("-" * 60) + for i, (name, dim) in enumerate( + zip(parsed["node_feature_names"], parsed["node_feature_dims"]) + ): + lines.append(f" [{i}] {name:30s} dim={dim}") + + # Graph features + if parsed["graph_feature_names"]: + lines.append("\nGraph Features:") + lines.append("-" * 60) + for i, (name, dim) in enumerate( + zip(parsed["graph_feature_names"], parsed["graph_feature_dims"]) + ): + lines.append(f" [{i}] {name:30s} dim={dim}") + + # Outputs + if parsed["output_names"]: + lines.append("\nOutput Configuration:") + lines.append("-" * 60) + for name, idx, dim, otype in zip( + parsed["output_names"], + parsed["output_index"], + parsed["output_dim"], + parsed["type"], + ): + lines.append(f" {name:30s} index={idx} dim={dim} type={otype}") + + lines.append("=" * 60) + return "\n".join(lines) diff --git a/pytest.ini b/pytest.ini index 5bb5ec0ff..d79dfd42a 100644 --- a/pytest.ini +++ b/pytest.ini @@ -2,7 +2,7 @@ [pytest] norecursedirs = dataset python_classes = Pytest -python_functions = pytest_ +python_functions = test_ pytest_ filterwarnings = ignore::DeprecationWarning ignore::FutureWarning diff --git a/test_all_final.log b/test_all_final.log new file mode 100644 index 000000000..fc616e730 --- /dev/null +++ b/test_all_final.log @@ -0,0 +1,30 @@ +============================= test session starts ============================== +platform darwin -- Python 3.13.7, pytest-8.4.2, pluggy-1.6.0 -- /Users/7ml/Documents/Codes/HydraGNN/.venv/bin/python +cachedir: .pytest_cache +rootdir: /Users/7ml/Documents/Codes/HydraGNN +configfile: pytest.ini +collecting ... collected 172 items + +tests/test_atomicdescriptors.py::pytest_atomicdescriptors FAILED [ 0%] +tests/test_config.py::pytest_config[lsms/lsms.json] PASSED [ 1%] +tests/test_datasetclass_inheritance.py::pytest_train_model_vectoroutput[PNA] SKIPPED [ 1%] +tests/test_deepspeed.py::pytest_train_model_vectoroutput_w_deepspeed[PNA] SKIPPED [ 2%] +tests/test_deepspeed.py::pytest_train_model_vectoroutput_w_deepspeed_global_attention[PNA-multihead-GPS] SKIPPED [ 2%] +tests/test_enthalpy.py::pytest_formation_enthalpy PASSED [ 3%] +tests/test_examples.py::pytest_examples_energy[qm9-SAGE-multihead-GPS] FAILED [ 4%] +tests/test_examples.py::pytest_examples_energy[qm9-GIN-multihead-GPS] FAILED [ 4%] +tests/test_examples.py::pytest_examples_energy[qm9-GAT-multihead-GPS] FAILED [ 5%] +tests/test_examples.py::pytest_examples_energy[qm9-MFC-multihead-GPS] FAILED [ 5%] +tests/test_examples.py::pytest_examples_energy[qm9-PNA-multihead-GPS] FAILED [ 6%] +tests/test_examples.py::pytest_examples_energy[qm9-PNAPlus-multihead-GPS] FAILED [ 6%] +tests/test_examples.py::pytest_examples_energy[qm9-SchNet-multihead-GPS] FAILED [ 7%] +tests/test_examples.py::pytest_examples_energy[qm9-DimeNet-multihead-GPS] FAILED [ 8%] +tests/test_examples.py::pytest_examples_energy[qm9-EGNN-multihead-GPS] FAILED [ 8%] +tests/test_examples.py::pytest_examples_energy[qm9-PNAEq-multihead-GPS] FAILED [ 9%] +tests/test_examples.py::pytest_examples_energy[qm9-PAINN-multihead-GPS] FAILED [ 9%] +tests/test_examples.py::pytest_examples_energy[md17-SAGE-multihead-GPS] FAILED [ 10%] +tests/test_examples.py::pytest_examples_energy[md17-GIN-multihead-GPS] FAILED [ 11%] +tests/test_examples.py::pytest_examples_energy[md17-GAT-multihead-GPS] FAILED [ 11%] +tests/test_examples.py::pytest_examples_energy[md17-MFC-multihead-GPS] FAILED [ 12%] +tests/test_examples.py::pytest_examples_energy[md17-PNA-multihead-GPS] FAILED [ 12%] +tests/test_examples.py::pytest_examples_energy[md17-PNAPlus-multihead-GPS] \ No newline at end of file diff --git a/test_examples_complete.log b/test_examples_complete.log new file mode 100644 index 000000000..05d85de16 --- /dev/null +++ b/test_examples_complete.log @@ -0,0 +1,1524 @@ +============================= test session starts ============================== +platform darwin -- Python 3.13.7, pytest-8.4.2, pluggy-1.6.0 -- /Users/7ml/Documents/Codes/HydraGNN/.venv/bin/python +cachedir: .pytest_cache +rootdir: /Users/7ml/Documents/Codes/HydraGNN +configfile: pytest.ini +collecting ... collected 29 items + +tests/test_examples.py::pytest_examples_energy[qm9-SAGE-multihead-GPS] FAILED [ 3%] +tests/test_examples.py::pytest_examples_energy[qm9-GIN-multihead-GPS] FAILED [ 6%] +tests/test_examples.py::pytest_examples_energy[qm9-GAT-multihead-GPS] FAILED [ 10%] +tests/test_examples.py::pytest_examples_energy[qm9-MFC-multihead-GPS] FAILED [ 13%] +tests/test_examples.py::pytest_examples_energy[qm9-PNA-multihead-GPS] FAILED [ 17%] +tests/test_examples.py::pytest_examples_energy[qm9-PNAPlus-multihead-GPS] FAILED [ 20%] +tests/test_examples.py::pytest_examples_energy[qm9-SchNet-multihead-GPS] FAILED [ 24%] +tests/test_examples.py::pytest_examples_energy[qm9-DimeNet-multihead-GPS] FAILED [ 27%] +tests/test_examples.py::pytest_examples_energy[qm9-EGNN-multihead-GPS] FAILED [ 31%] +tests/test_examples.py::pytest_examples_energy[qm9-PNAEq-multihead-GPS] FAILED [ 34%] +tests/test_examples.py::pytest_examples_energy[qm9-PAINN-multihead-GPS] FAILED [ 37%] +tests/test_examples.py::pytest_examples_energy[md17-SAGE-multihead-GPS] FAILED [ 41%] +tests/test_examples.py::pytest_examples_energy[md17-GIN-multihead-GPS] FAILED [ 44%] +tests/test_examples.py::pytest_examples_energy[md17-GAT-multihead-GPS] FAILED [ 48%] +tests/test_examples.py::pytest_examples_energy[md17-MFC-multihead-GPS] FAILED [ 51%] +tests/test_examples.py::pytest_examples_energy[md17-PNA-multihead-GPS] FAILED [ 55%] +tests/test_examples.py::pytest_examples_energy[md17-PNAPlus-multihead-GPS] FAILED [ 58%] +tests/test_examples.py::pytest_examples_energy[md17-SchNet-multihead-GPS] FAILED [ 62%] +tests/test_examples.py::pytest_examples_energy[md17-DimeNet-multihead-GPS] FAILED [ 65%] +tests/test_examples.py::pytest_examples_energy[md17-EGNN-multihead-GPS] FAILED [ 68%] +tests/test_examples.py::pytest_examples_energy[md17-PNAEq-multihead-GPS] FAILED [ 72%] +tests/test_examples.py::pytest_examples_energy[md17-PAINN-multihead-GPS] FAILED [ 75%] +tests/test_examples.py::pytest_examples_grad_forces[LennardJones-PNAPlus] FAILED [ 79%] +tests/test_examples.py::pytest_examples_grad_forces[LennardJones-SchNet] FAILED [ 82%] +tests/test_examples.py::pytest_examples_grad_forces[LennardJones-DimeNet] FAILED [ 86%] +tests/test_examples.py::pytest_examples_grad_forces[LennardJones-EGNN] FAILED [ 89%] +tests/test_examples.py::pytest_examples_grad_forces[LennardJones-PNAEq] FAILED [ 93%] +tests/test_examples.py::pytest_examples_grad_forces[LennardJones-PAINN] FAILED [ 96%] +tests/test_examples.py::pytest_examples_grad_forces[LennardJones-MACE] FAILED [100%] + +=================================== FAILURES =================================== +________________ pytest_examples_energy[qm9-SAGE-multihead-GPS] ________________ + +example = 'qm9', mpnn_type = 'SAGE', global_attn_engine = 'GPS' +global_attn_type = 'multihead' + + @pytest.mark.parametrize( + "global_attn_engine", + ["GPS"], + ) + @pytest.mark.parametrize("global_attn_type", ["multihead"]) + @pytest.mark.parametrize( + "mpnn_type", + [ + "SAGE", + "GIN", + "GAT", + "MFC", + "PNA", + "PNAPlus", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + "PAINN", + ], + ) + @pytest.mark.parametrize("example", ["qm9", "md17"]) + @pytest.mark.mpi_skip() + def pytest_examples_energy(example, mpnn_type, global_attn_engine, global_attn_type): + path = os.path.join(os.path.dirname(__file__), "..", "examples", example) + file_path = os.path.join(path, example + ".py") + # Add the --mpnn_type argument for the subprocess call + return_code = subprocess.call( + [ + "python", + file_path, + "--mpnn_type", + mpnn_type, + "--global_attn_engine", + global_attn_engine, + "--global_attn_type", + global_attn_type, + ] + ) + + # Check the file ran without error. +> assert return_code == 0 +E assert 1 == 0 + +tests/test_examples.py:59: AssertionError +----------------------------- Captured stderr call ----------------------------- +Traceback (most recent call last): + File "/Users/7ml/Documents/Codes/HydraGNN/tests/../examples/qm9/qm9.py", line 16, in + import hydragnn +ModuleNotFoundError: No module named 'hydragnn' +________________ pytest_examples_energy[qm9-GIN-multihead-GPS] _________________ + +example = 'qm9', mpnn_type = 'GIN', global_attn_engine = 'GPS' +global_attn_type = 'multihead' + + @pytest.mark.parametrize( + "global_attn_engine", + ["GPS"], + ) + @pytest.mark.parametrize("global_attn_type", ["multihead"]) + @pytest.mark.parametrize( + "mpnn_type", + [ + "SAGE", + "GIN", + "GAT", + "MFC", + "PNA", + "PNAPlus", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + "PAINN", + ], + ) + @pytest.mark.parametrize("example", ["qm9", "md17"]) + @pytest.mark.mpi_skip() + def pytest_examples_energy(example, mpnn_type, global_attn_engine, global_attn_type): + path = os.path.join(os.path.dirname(__file__), "..", "examples", example) + file_path = os.path.join(path, example + ".py") + # Add the --mpnn_type argument for the subprocess call + return_code = subprocess.call( + [ + "python", + file_path, + "--mpnn_type", + mpnn_type, + "--global_attn_engine", + global_attn_engine, + "--global_attn_type", + global_attn_type, + ] + ) + + # Check the file ran without error. +> assert return_code == 0 +E assert 1 == 0 + +tests/test_examples.py:59: AssertionError +----------------------------- Captured stderr call ----------------------------- +Traceback (most recent call last): + File "/Users/7ml/Documents/Codes/HydraGNN/tests/../examples/qm9/qm9.py", line 16, in + import hydragnn +ModuleNotFoundError: No module named 'hydragnn' +________________ pytest_examples_energy[qm9-GAT-multihead-GPS] _________________ + +example = 'qm9', mpnn_type = 'GAT', global_attn_engine = 'GPS' +global_attn_type = 'multihead' + + @pytest.mark.parametrize( + "global_attn_engine", + ["GPS"], + ) + @pytest.mark.parametrize("global_attn_type", ["multihead"]) + @pytest.mark.parametrize( + "mpnn_type", + [ + "SAGE", + "GIN", + "GAT", + "MFC", + "PNA", + "PNAPlus", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + "PAINN", + ], + ) + @pytest.mark.parametrize("example", ["qm9", "md17"]) + @pytest.mark.mpi_skip() + def pytest_examples_energy(example, mpnn_type, global_attn_engine, global_attn_type): + path = os.path.join(os.path.dirname(__file__), "..", "examples", example) + file_path = os.path.join(path, example + ".py") + # Add the --mpnn_type argument for the subprocess call + return_code = subprocess.call( + [ + "python", + file_path, + "--mpnn_type", + mpnn_type, + "--global_attn_engine", + global_attn_engine, + "--global_attn_type", + global_attn_type, + ] + ) + + # Check the file ran without error. +> assert return_code == 0 +E assert 1 == 0 + +tests/test_examples.py:59: AssertionError +----------------------------- Captured stderr call ----------------------------- +Traceback (most recent call last): + File "/Users/7ml/Documents/Codes/HydraGNN/tests/../examples/qm9/qm9.py", line 16, in + import hydragnn +ModuleNotFoundError: No module named 'hydragnn' +________________ pytest_examples_energy[qm9-MFC-multihead-GPS] _________________ + +example = 'qm9', mpnn_type = 'MFC', global_attn_engine = 'GPS' +global_attn_type = 'multihead' + + @pytest.mark.parametrize( + "global_attn_engine", + ["GPS"], + ) + @pytest.mark.parametrize("global_attn_type", ["multihead"]) + @pytest.mark.parametrize( + "mpnn_type", + [ + "SAGE", + "GIN", + "GAT", + "MFC", + "PNA", + "PNAPlus", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + "PAINN", + ], + ) + @pytest.mark.parametrize("example", ["qm9", "md17"]) + @pytest.mark.mpi_skip() + def pytest_examples_energy(example, mpnn_type, global_attn_engine, global_attn_type): + path = os.path.join(os.path.dirname(__file__), "..", "examples", example) + file_path = os.path.join(path, example + ".py") + # Add the --mpnn_type argument for the subprocess call + return_code = subprocess.call( + [ + "python", + file_path, + "--mpnn_type", + mpnn_type, + "--global_attn_engine", + global_attn_engine, + "--global_attn_type", + global_attn_type, + ] + ) + + # Check the file ran without error. +> assert return_code == 0 +E assert 1 == 0 + +tests/test_examples.py:59: AssertionError +----------------------------- Captured stderr call ----------------------------- +Traceback (most recent call last): + File "/Users/7ml/Documents/Codes/HydraGNN/tests/../examples/qm9/qm9.py", line 16, in + import hydragnn +ModuleNotFoundError: No module named 'hydragnn' +________________ pytest_examples_energy[qm9-PNA-multihead-GPS] _________________ + +example = 'qm9', mpnn_type = 'PNA', global_attn_engine = 'GPS' +global_attn_type = 'multihead' + + @pytest.mark.parametrize( + "global_attn_engine", + ["GPS"], + ) + @pytest.mark.parametrize("global_attn_type", ["multihead"]) + @pytest.mark.parametrize( + "mpnn_type", + [ + "SAGE", + "GIN", + "GAT", + "MFC", + "PNA", + "PNAPlus", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + "PAINN", + ], + ) + @pytest.mark.parametrize("example", ["qm9", "md17"]) + @pytest.mark.mpi_skip() + def pytest_examples_energy(example, mpnn_type, global_attn_engine, global_attn_type): + path = os.path.join(os.path.dirname(__file__), "..", "examples", example) + file_path = os.path.join(path, example + ".py") + # Add the --mpnn_type argument for the subprocess call + return_code = subprocess.call( + [ + "python", + file_path, + "--mpnn_type", + mpnn_type, + "--global_attn_engine", + global_attn_engine, + "--global_attn_type", + global_attn_type, + ] + ) + + # Check the file ran without error. +> assert return_code == 0 +E assert 1 == 0 + +tests/test_examples.py:59: AssertionError +----------------------------- Captured stderr call ----------------------------- +Traceback (most recent call last): + File "/Users/7ml/Documents/Codes/HydraGNN/tests/../examples/qm9/qm9.py", line 16, in + import hydragnn +ModuleNotFoundError: No module named 'hydragnn' +______________ pytest_examples_energy[qm9-PNAPlus-multihead-GPS] _______________ + +example = 'qm9', mpnn_type = 'PNAPlus', global_attn_engine = 'GPS' +global_attn_type = 'multihead' + + @pytest.mark.parametrize( + "global_attn_engine", + ["GPS"], + ) + @pytest.mark.parametrize("global_attn_type", ["multihead"]) + @pytest.mark.parametrize( + "mpnn_type", + [ + "SAGE", + "GIN", + "GAT", + "MFC", + "PNA", + "PNAPlus", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + "PAINN", + ], + ) + @pytest.mark.parametrize("example", ["qm9", "md17"]) + @pytest.mark.mpi_skip() + def pytest_examples_energy(example, mpnn_type, global_attn_engine, global_attn_type): + path = os.path.join(os.path.dirname(__file__), "..", "examples", example) + file_path = os.path.join(path, example + ".py") + # Add the --mpnn_type argument for the subprocess call + return_code = subprocess.call( + [ + "python", + file_path, + "--mpnn_type", + mpnn_type, + "--global_attn_engine", + global_attn_engine, + "--global_attn_type", + global_attn_type, + ] + ) + + # Check the file ran without error. +> assert return_code == 0 +E assert 1 == 0 + +tests/test_examples.py:59: AssertionError +----------------------------- Captured stderr call ----------------------------- +Traceback (most recent call last): + File "/Users/7ml/Documents/Codes/HydraGNN/tests/../examples/qm9/qm9.py", line 16, in + import hydragnn +ModuleNotFoundError: No module named 'hydragnn' +_______________ pytest_examples_energy[qm9-SchNet-multihead-GPS] _______________ + +example = 'qm9', mpnn_type = 'SchNet', global_attn_engine = 'GPS' +global_attn_type = 'multihead' + + @pytest.mark.parametrize( + "global_attn_engine", + ["GPS"], + ) + @pytest.mark.parametrize("global_attn_type", ["multihead"]) + @pytest.mark.parametrize( + "mpnn_type", + [ + "SAGE", + "GIN", + "GAT", + "MFC", + "PNA", + "PNAPlus", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + "PAINN", + ], + ) + @pytest.mark.parametrize("example", ["qm9", "md17"]) + @pytest.mark.mpi_skip() + def pytest_examples_energy(example, mpnn_type, global_attn_engine, global_attn_type): + path = os.path.join(os.path.dirname(__file__), "..", "examples", example) + file_path = os.path.join(path, example + ".py") + # Add the --mpnn_type argument for the subprocess call + return_code = subprocess.call( + [ + "python", + file_path, + "--mpnn_type", + mpnn_type, + "--global_attn_engine", + global_attn_engine, + "--global_attn_type", + global_attn_type, + ] + ) + + # Check the file ran without error. +> assert return_code == 0 +E assert 1 == 0 + +tests/test_examples.py:59: AssertionError +----------------------------- Captured stderr call ----------------------------- +Traceback (most recent call last): + File "/Users/7ml/Documents/Codes/HydraGNN/tests/../examples/qm9/qm9.py", line 16, in + import hydragnn +ModuleNotFoundError: No module named 'hydragnn' +______________ pytest_examples_energy[qm9-DimeNet-multihead-GPS] _______________ + +example = 'qm9', mpnn_type = 'DimeNet', global_attn_engine = 'GPS' +global_attn_type = 'multihead' + + @pytest.mark.parametrize( + "global_attn_engine", + ["GPS"], + ) + @pytest.mark.parametrize("global_attn_type", ["multihead"]) + @pytest.mark.parametrize( + "mpnn_type", + [ + "SAGE", + "GIN", + "GAT", + "MFC", + "PNA", + "PNAPlus", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + "PAINN", + ], + ) + @pytest.mark.parametrize("example", ["qm9", "md17"]) + @pytest.mark.mpi_skip() + def pytest_examples_energy(example, mpnn_type, global_attn_engine, global_attn_type): + path = os.path.join(os.path.dirname(__file__), "..", "examples", example) + file_path = os.path.join(path, example + ".py") + # Add the --mpnn_type argument for the subprocess call + return_code = subprocess.call( + [ + "python", + file_path, + "--mpnn_type", + mpnn_type, + "--global_attn_engine", + global_attn_engine, + "--global_attn_type", + global_attn_type, + ] + ) + + # Check the file ran without error. +> assert return_code == 0 +E assert 1 == 0 + +tests/test_examples.py:59: AssertionError +----------------------------- Captured stderr call ----------------------------- +Traceback (most recent call last): + File "/Users/7ml/Documents/Codes/HydraGNN/tests/../examples/qm9/qm9.py", line 16, in + import hydragnn +ModuleNotFoundError: No module named 'hydragnn' +________________ pytest_examples_energy[qm9-EGNN-multihead-GPS] ________________ + +example = 'qm9', mpnn_type = 'EGNN', global_attn_engine = 'GPS' +global_attn_type = 'multihead' + + @pytest.mark.parametrize( + "global_attn_engine", + ["GPS"], + ) + @pytest.mark.parametrize("global_attn_type", ["multihead"]) + @pytest.mark.parametrize( + "mpnn_type", + [ + "SAGE", + "GIN", + "GAT", + "MFC", + "PNA", + "PNAPlus", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + "PAINN", + ], + ) + @pytest.mark.parametrize("example", ["qm9", "md17"]) + @pytest.mark.mpi_skip() + def pytest_examples_energy(example, mpnn_type, global_attn_engine, global_attn_type): + path = os.path.join(os.path.dirname(__file__), "..", "examples", example) + file_path = os.path.join(path, example + ".py") + # Add the --mpnn_type argument for the subprocess call + return_code = subprocess.call( + [ + "python", + file_path, + "--mpnn_type", + mpnn_type, + "--global_attn_engine", + global_attn_engine, + "--global_attn_type", + global_attn_type, + ] + ) + + # Check the file ran without error. +> assert return_code == 0 +E assert 1 == 0 + +tests/test_examples.py:59: AssertionError +----------------------------- Captured stderr call ----------------------------- +Traceback (most recent call last): + File "/Users/7ml/Documents/Codes/HydraGNN/tests/../examples/qm9/qm9.py", line 16, in + import hydragnn +ModuleNotFoundError: No module named 'hydragnn' +_______________ pytest_examples_energy[qm9-PNAEq-multihead-GPS] ________________ + +example = 'qm9', mpnn_type = 'PNAEq', global_attn_engine = 'GPS' +global_attn_type = 'multihead' + + @pytest.mark.parametrize( + "global_attn_engine", + ["GPS"], + ) + @pytest.mark.parametrize("global_attn_type", ["multihead"]) + @pytest.mark.parametrize( + "mpnn_type", + [ + "SAGE", + "GIN", + "GAT", + "MFC", + "PNA", + "PNAPlus", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + "PAINN", + ], + ) + @pytest.mark.parametrize("example", ["qm9", "md17"]) + @pytest.mark.mpi_skip() + def pytest_examples_energy(example, mpnn_type, global_attn_engine, global_attn_type): + path = os.path.join(os.path.dirname(__file__), "..", "examples", example) + file_path = os.path.join(path, example + ".py") + # Add the --mpnn_type argument for the subprocess call + return_code = subprocess.call( + [ + "python", + file_path, + "--mpnn_type", + mpnn_type, + "--global_attn_engine", + global_attn_engine, + "--global_attn_type", + global_attn_type, + ] + ) + + # Check the file ran without error. +> assert return_code == 0 +E assert 1 == 0 + +tests/test_examples.py:59: AssertionError +----------------------------- Captured stderr call ----------------------------- +Traceback (most recent call last): + File "/Users/7ml/Documents/Codes/HydraGNN/tests/../examples/qm9/qm9.py", line 16, in + import hydragnn +ModuleNotFoundError: No module named 'hydragnn' +_______________ pytest_examples_energy[qm9-PAINN-multihead-GPS] ________________ + +example = 'qm9', mpnn_type = 'PAINN', global_attn_engine = 'GPS' +global_attn_type = 'multihead' + + @pytest.mark.parametrize( + "global_attn_engine", + ["GPS"], + ) + @pytest.mark.parametrize("global_attn_type", ["multihead"]) + @pytest.mark.parametrize( + "mpnn_type", + [ + "SAGE", + "GIN", + "GAT", + "MFC", + "PNA", + "PNAPlus", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + "PAINN", + ], + ) + @pytest.mark.parametrize("example", ["qm9", "md17"]) + @pytest.mark.mpi_skip() + def pytest_examples_energy(example, mpnn_type, global_attn_engine, global_attn_type): + path = os.path.join(os.path.dirname(__file__), "..", "examples", example) + file_path = os.path.join(path, example + ".py") + # Add the --mpnn_type argument for the subprocess call + return_code = subprocess.call( + [ + "python", + file_path, + "--mpnn_type", + mpnn_type, + "--global_attn_engine", + global_attn_engine, + "--global_attn_type", + global_attn_type, + ] + ) + + # Check the file ran without error. +> assert return_code == 0 +E assert 1 == 0 + +tests/test_examples.py:59: AssertionError +----------------------------- Captured stderr call ----------------------------- +Traceback (most recent call last): + File "/Users/7ml/Documents/Codes/HydraGNN/tests/../examples/qm9/qm9.py", line 16, in + import hydragnn +ModuleNotFoundError: No module named 'hydragnn' +_______________ pytest_examples_energy[md17-SAGE-multihead-GPS] ________________ + +example = 'md17', mpnn_type = 'SAGE', global_attn_engine = 'GPS' +global_attn_type = 'multihead' + + @pytest.mark.parametrize( + "global_attn_engine", + ["GPS"], + ) + @pytest.mark.parametrize("global_attn_type", ["multihead"]) + @pytest.mark.parametrize( + "mpnn_type", + [ + "SAGE", + "GIN", + "GAT", + "MFC", + "PNA", + "PNAPlus", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + "PAINN", + ], + ) + @pytest.mark.parametrize("example", ["qm9", "md17"]) + @pytest.mark.mpi_skip() + def pytest_examples_energy(example, mpnn_type, global_attn_engine, global_attn_type): + path = os.path.join(os.path.dirname(__file__), "..", "examples", example) + file_path = os.path.join(path, example + ".py") + # Add the --mpnn_type argument for the subprocess call + return_code = subprocess.call( + [ + "python", + file_path, + "--mpnn_type", + mpnn_type, + "--global_attn_engine", + global_attn_engine, + "--global_attn_type", + global_attn_type, + ] + ) + + # Check the file ran without error. +> assert return_code == 0 +E assert 1 == 0 + +tests/test_examples.py:59: AssertionError +----------------------------- Captured stderr call ----------------------------- +Traceback (most recent call last): + File "/Users/7ml/Documents/Codes/HydraGNN/tests/../examples/md17/md17.py", line 17, in + import hydragnn +ModuleNotFoundError: No module named 'hydragnn' +________________ pytest_examples_energy[md17-GIN-multihead-GPS] ________________ + +example = 'md17', mpnn_type = 'GIN', global_attn_engine = 'GPS' +global_attn_type = 'multihead' + + @pytest.mark.parametrize( + "global_attn_engine", + ["GPS"], + ) + @pytest.mark.parametrize("global_attn_type", ["multihead"]) + @pytest.mark.parametrize( + "mpnn_type", + [ + "SAGE", + "GIN", + "GAT", + "MFC", + "PNA", + "PNAPlus", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + "PAINN", + ], + ) + @pytest.mark.parametrize("example", ["qm9", "md17"]) + @pytest.mark.mpi_skip() + def pytest_examples_energy(example, mpnn_type, global_attn_engine, global_attn_type): + path = os.path.join(os.path.dirname(__file__), "..", "examples", example) + file_path = os.path.join(path, example + ".py") + # Add the --mpnn_type argument for the subprocess call + return_code = subprocess.call( + [ + "python", + file_path, + "--mpnn_type", + mpnn_type, + "--global_attn_engine", + global_attn_engine, + "--global_attn_type", + global_attn_type, + ] + ) + + # Check the file ran without error. +> assert return_code == 0 +E assert 1 == 0 + +tests/test_examples.py:59: AssertionError +----------------------------- Captured stderr call ----------------------------- +Traceback (most recent call last): + File "/Users/7ml/Documents/Codes/HydraGNN/tests/../examples/md17/md17.py", line 17, in + import hydragnn +ModuleNotFoundError: No module named 'hydragnn' +________________ pytest_examples_energy[md17-GAT-multihead-GPS] ________________ + +example = 'md17', mpnn_type = 'GAT', global_attn_engine = 'GPS' +global_attn_type = 'multihead' + + @pytest.mark.parametrize( + "global_attn_engine", + ["GPS"], + ) + @pytest.mark.parametrize("global_attn_type", ["multihead"]) + @pytest.mark.parametrize( + "mpnn_type", + [ + "SAGE", + "GIN", + "GAT", + "MFC", + "PNA", + "PNAPlus", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + "PAINN", + ], + ) + @pytest.mark.parametrize("example", ["qm9", "md17"]) + @pytest.mark.mpi_skip() + def pytest_examples_energy(example, mpnn_type, global_attn_engine, global_attn_type): + path = os.path.join(os.path.dirname(__file__), "..", "examples", example) + file_path = os.path.join(path, example + ".py") + # Add the --mpnn_type argument for the subprocess call + return_code = subprocess.call( + [ + "python", + file_path, + "--mpnn_type", + mpnn_type, + "--global_attn_engine", + global_attn_engine, + "--global_attn_type", + global_attn_type, + ] + ) + + # Check the file ran without error. +> assert return_code == 0 +E assert 1 == 0 + +tests/test_examples.py:59: AssertionError +----------------------------- Captured stderr call ----------------------------- +Traceback (most recent call last): + File "/Users/7ml/Documents/Codes/HydraGNN/tests/../examples/md17/md17.py", line 17, in + import hydragnn +ModuleNotFoundError: No module named 'hydragnn' +________________ pytest_examples_energy[md17-MFC-multihead-GPS] ________________ + +example = 'md17', mpnn_type = 'MFC', global_attn_engine = 'GPS' +global_attn_type = 'multihead' + + @pytest.mark.parametrize( + "global_attn_engine", + ["GPS"], + ) + @pytest.mark.parametrize("global_attn_type", ["multihead"]) + @pytest.mark.parametrize( + "mpnn_type", + [ + "SAGE", + "GIN", + "GAT", + "MFC", + "PNA", + "PNAPlus", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + "PAINN", + ], + ) + @pytest.mark.parametrize("example", ["qm9", "md17"]) + @pytest.mark.mpi_skip() + def pytest_examples_energy(example, mpnn_type, global_attn_engine, global_attn_type): + path = os.path.join(os.path.dirname(__file__), "..", "examples", example) + file_path = os.path.join(path, example + ".py") + # Add the --mpnn_type argument for the subprocess call + return_code = subprocess.call( + [ + "python", + file_path, + "--mpnn_type", + mpnn_type, + "--global_attn_engine", + global_attn_engine, + "--global_attn_type", + global_attn_type, + ] + ) + + # Check the file ran without error. +> assert return_code == 0 +E assert 1 == 0 + +tests/test_examples.py:59: AssertionError +----------------------------- Captured stderr call ----------------------------- +Traceback (most recent call last): + File "/Users/7ml/Documents/Codes/HydraGNN/tests/../examples/md17/md17.py", line 17, in + import hydragnn +ModuleNotFoundError: No module named 'hydragnn' +________________ pytest_examples_energy[md17-PNA-multihead-GPS] ________________ + +example = 'md17', mpnn_type = 'PNA', global_attn_engine = 'GPS' +global_attn_type = 'multihead' + + @pytest.mark.parametrize( + "global_attn_engine", + ["GPS"], + ) + @pytest.mark.parametrize("global_attn_type", ["multihead"]) + @pytest.mark.parametrize( + "mpnn_type", + [ + "SAGE", + "GIN", + "GAT", + "MFC", + "PNA", + "PNAPlus", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + "PAINN", + ], + ) + @pytest.mark.parametrize("example", ["qm9", "md17"]) + @pytest.mark.mpi_skip() + def pytest_examples_energy(example, mpnn_type, global_attn_engine, global_attn_type): + path = os.path.join(os.path.dirname(__file__), "..", "examples", example) + file_path = os.path.join(path, example + ".py") + # Add the --mpnn_type argument for the subprocess call + return_code = subprocess.call( + [ + "python", + file_path, + "--mpnn_type", + mpnn_type, + "--global_attn_engine", + global_attn_engine, + "--global_attn_type", + global_attn_type, + ] + ) + + # Check the file ran without error. +> assert return_code == 0 +E assert 1 == 0 + +tests/test_examples.py:59: AssertionError +----------------------------- Captured stderr call ----------------------------- +Traceback (most recent call last): + File "/Users/7ml/Documents/Codes/HydraGNN/tests/../examples/md17/md17.py", line 17, in + import hydragnn +ModuleNotFoundError: No module named 'hydragnn' +______________ pytest_examples_energy[md17-PNAPlus-multihead-GPS] ______________ + +example = 'md17', mpnn_type = 'PNAPlus', global_attn_engine = 'GPS' +global_attn_type = 'multihead' + + @pytest.mark.parametrize( + "global_attn_engine", + ["GPS"], + ) + @pytest.mark.parametrize("global_attn_type", ["multihead"]) + @pytest.mark.parametrize( + "mpnn_type", + [ + "SAGE", + "GIN", + "GAT", + "MFC", + "PNA", + "PNAPlus", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + "PAINN", + ], + ) + @pytest.mark.parametrize("example", ["qm9", "md17"]) + @pytest.mark.mpi_skip() + def pytest_examples_energy(example, mpnn_type, global_attn_engine, global_attn_type): + path = os.path.join(os.path.dirname(__file__), "..", "examples", example) + file_path = os.path.join(path, example + ".py") + # Add the --mpnn_type argument for the subprocess call + return_code = subprocess.call( + [ + "python", + file_path, + "--mpnn_type", + mpnn_type, + "--global_attn_engine", + global_attn_engine, + "--global_attn_type", + global_attn_type, + ] + ) + + # Check the file ran without error. +> assert return_code == 0 +E assert 1 == 0 + +tests/test_examples.py:59: AssertionError +----------------------------- Captured stderr call ----------------------------- +Traceback (most recent call last): + File "/Users/7ml/Documents/Codes/HydraGNN/tests/../examples/md17/md17.py", line 17, in + import hydragnn +ModuleNotFoundError: No module named 'hydragnn' +______________ pytest_examples_energy[md17-SchNet-multihead-GPS] _______________ + +example = 'md17', mpnn_type = 'SchNet', global_attn_engine = 'GPS' +global_attn_type = 'multihead' + + @pytest.mark.parametrize( + "global_attn_engine", + ["GPS"], + ) + @pytest.mark.parametrize("global_attn_type", ["multihead"]) + @pytest.mark.parametrize( + "mpnn_type", + [ + "SAGE", + "GIN", + "GAT", + "MFC", + "PNA", + "PNAPlus", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + "PAINN", + ], + ) + @pytest.mark.parametrize("example", ["qm9", "md17"]) + @pytest.mark.mpi_skip() + def pytest_examples_energy(example, mpnn_type, global_attn_engine, global_attn_type): + path = os.path.join(os.path.dirname(__file__), "..", "examples", example) + file_path = os.path.join(path, example + ".py") + # Add the --mpnn_type argument for the subprocess call + return_code = subprocess.call( + [ + "python", + file_path, + "--mpnn_type", + mpnn_type, + "--global_attn_engine", + global_attn_engine, + "--global_attn_type", + global_attn_type, + ] + ) + + # Check the file ran without error. +> assert return_code == 0 +E assert 1 == 0 + +tests/test_examples.py:59: AssertionError +----------------------------- Captured stderr call ----------------------------- +Traceback (most recent call last): + File "/Users/7ml/Documents/Codes/HydraGNN/tests/../examples/md17/md17.py", line 17, in + import hydragnn +ModuleNotFoundError: No module named 'hydragnn' +______________ pytest_examples_energy[md17-DimeNet-multihead-GPS] ______________ + +example = 'md17', mpnn_type = 'DimeNet', global_attn_engine = 'GPS' +global_attn_type = 'multihead' + + @pytest.mark.parametrize( + "global_attn_engine", + ["GPS"], + ) + @pytest.mark.parametrize("global_attn_type", ["multihead"]) + @pytest.mark.parametrize( + "mpnn_type", + [ + "SAGE", + "GIN", + "GAT", + "MFC", + "PNA", + "PNAPlus", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + "PAINN", + ], + ) + @pytest.mark.parametrize("example", ["qm9", "md17"]) + @pytest.mark.mpi_skip() + def pytest_examples_energy(example, mpnn_type, global_attn_engine, global_attn_type): + path = os.path.join(os.path.dirname(__file__), "..", "examples", example) + file_path = os.path.join(path, example + ".py") + # Add the --mpnn_type argument for the subprocess call + return_code = subprocess.call( + [ + "python", + file_path, + "--mpnn_type", + mpnn_type, + "--global_attn_engine", + global_attn_engine, + "--global_attn_type", + global_attn_type, + ] + ) + + # Check the file ran without error. +> assert return_code == 0 +E assert 1 == 0 + +tests/test_examples.py:59: AssertionError +----------------------------- Captured stderr call ----------------------------- +Traceback (most recent call last): + File "/Users/7ml/Documents/Codes/HydraGNN/tests/../examples/md17/md17.py", line 17, in + import hydragnn +ModuleNotFoundError: No module named 'hydragnn' +_______________ pytest_examples_energy[md17-EGNN-multihead-GPS] ________________ + +example = 'md17', mpnn_type = 'EGNN', global_attn_engine = 'GPS' +global_attn_type = 'multihead' + + @pytest.mark.parametrize( + "global_attn_engine", + ["GPS"], + ) + @pytest.mark.parametrize("global_attn_type", ["multihead"]) + @pytest.mark.parametrize( + "mpnn_type", + [ + "SAGE", + "GIN", + "GAT", + "MFC", + "PNA", + "PNAPlus", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + "PAINN", + ], + ) + @pytest.mark.parametrize("example", ["qm9", "md17"]) + @pytest.mark.mpi_skip() + def pytest_examples_energy(example, mpnn_type, global_attn_engine, global_attn_type): + path = os.path.join(os.path.dirname(__file__), "..", "examples", example) + file_path = os.path.join(path, example + ".py") + # Add the --mpnn_type argument for the subprocess call + return_code = subprocess.call( + [ + "python", + file_path, + "--mpnn_type", + mpnn_type, + "--global_attn_engine", + global_attn_engine, + "--global_attn_type", + global_attn_type, + ] + ) + + # Check the file ran without error. +> assert return_code == 0 +E assert 1 == 0 + +tests/test_examples.py:59: AssertionError +----------------------------- Captured stderr call ----------------------------- +Traceback (most recent call last): + File "/Users/7ml/Documents/Codes/HydraGNN/tests/../examples/md17/md17.py", line 17, in + import hydragnn +ModuleNotFoundError: No module named 'hydragnn' +_______________ pytest_examples_energy[md17-PNAEq-multihead-GPS] _______________ + +example = 'md17', mpnn_type = 'PNAEq', global_attn_engine = 'GPS' +global_attn_type = 'multihead' + + @pytest.mark.parametrize( + "global_attn_engine", + ["GPS"], + ) + @pytest.mark.parametrize("global_attn_type", ["multihead"]) + @pytest.mark.parametrize( + "mpnn_type", + [ + "SAGE", + "GIN", + "GAT", + "MFC", + "PNA", + "PNAPlus", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + "PAINN", + ], + ) + @pytest.mark.parametrize("example", ["qm9", "md17"]) + @pytest.mark.mpi_skip() + def pytest_examples_energy(example, mpnn_type, global_attn_engine, global_attn_type): + path = os.path.join(os.path.dirname(__file__), "..", "examples", example) + file_path = os.path.join(path, example + ".py") + # Add the --mpnn_type argument for the subprocess call + return_code = subprocess.call( + [ + "python", + file_path, + "--mpnn_type", + mpnn_type, + "--global_attn_engine", + global_attn_engine, + "--global_attn_type", + global_attn_type, + ] + ) + + # Check the file ran without error. +> assert return_code == 0 +E assert 1 == 0 + +tests/test_examples.py:59: AssertionError +----------------------------- Captured stderr call ----------------------------- +Traceback (most recent call last): + File "/Users/7ml/Documents/Codes/HydraGNN/tests/../examples/md17/md17.py", line 17, in + import hydragnn +ModuleNotFoundError: No module named 'hydragnn' +_______________ pytest_examples_energy[md17-PAINN-multihead-GPS] _______________ + +example = 'md17', mpnn_type = 'PAINN', global_attn_engine = 'GPS' +global_attn_type = 'multihead' + + @pytest.mark.parametrize( + "global_attn_engine", + ["GPS"], + ) + @pytest.mark.parametrize("global_attn_type", ["multihead"]) + @pytest.mark.parametrize( + "mpnn_type", + [ + "SAGE", + "GIN", + "GAT", + "MFC", + "PNA", + "PNAPlus", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + "PAINN", + ], + ) + @pytest.mark.parametrize("example", ["qm9", "md17"]) + @pytest.mark.mpi_skip() + def pytest_examples_energy(example, mpnn_type, global_attn_engine, global_attn_type): + path = os.path.join(os.path.dirname(__file__), "..", "examples", example) + file_path = os.path.join(path, example + ".py") + # Add the --mpnn_type argument for the subprocess call + return_code = subprocess.call( + [ + "python", + file_path, + "--mpnn_type", + mpnn_type, + "--global_attn_engine", + global_attn_engine, + "--global_attn_type", + global_attn_type, + ] + ) + + # Check the file ran without error. +> assert return_code == 0 +E assert 1 == 0 + +tests/test_examples.py:59: AssertionError +----------------------------- Captured stderr call ----------------------------- +Traceback (most recent call last): + File "/Users/7ml/Documents/Codes/HydraGNN/tests/../examples/md17/md17.py", line 17, in + import hydragnn +ModuleNotFoundError: No module named 'hydragnn' +______________ pytest_examples_grad_forces[LennardJones-PNAPlus] _______________ + +example = 'LennardJones', mpnn_type = 'PNAPlus' + + @pytest.mark.parametrize( + "mpnn_type", + [ + "PNAPlus", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + "PAINN", + "MACE", + ], + ) + @pytest.mark.parametrize("example", ["LennardJones"]) + @pytest.mark.mpi_skip() + def pytest_examples_grad_forces(example, mpnn_type): + path = os.path.join(os.path.dirname(__file__), "..", "examples", example) + file_path = os.path.join(path, example + ".py") + + # Add the --mpnn_type argument for the subprocess call + return_code = subprocess.call(["python", file_path, "--mpnn_type", mpnn_type]) + + # Check the file ran without error. +> assert return_code == 0 +E assert 1 == 0 + +tests/test_examples.py:87: AssertionError +----------------------------- Captured stderr call ----------------------------- +Traceback (most recent call last): + File "/Users/7ml/Documents/Codes/HydraGNN/tests/../examples/LennardJones/LennardJones.py", line 33, in + import hydragnn +ModuleNotFoundError: No module named 'hydragnn' +_______________ pytest_examples_grad_forces[LennardJones-SchNet] _______________ + +example = 'LennardJones', mpnn_type = 'SchNet' + + @pytest.mark.parametrize( + "mpnn_type", + [ + "PNAPlus", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + "PAINN", + "MACE", + ], + ) + @pytest.mark.parametrize("example", ["LennardJones"]) + @pytest.mark.mpi_skip() + def pytest_examples_grad_forces(example, mpnn_type): + path = os.path.join(os.path.dirname(__file__), "..", "examples", example) + file_path = os.path.join(path, example + ".py") + + # Add the --mpnn_type argument for the subprocess call + return_code = subprocess.call(["python", file_path, "--mpnn_type", mpnn_type]) + + # Check the file ran without error. +> assert return_code == 0 +E assert 1 == 0 + +tests/test_examples.py:87: AssertionError +----------------------------- Captured stderr call ----------------------------- +Traceback (most recent call last): + File "/Users/7ml/Documents/Codes/HydraGNN/tests/../examples/LennardJones/LennardJones.py", line 33, in + import hydragnn +ModuleNotFoundError: No module named 'hydragnn' +______________ pytest_examples_grad_forces[LennardJones-DimeNet] _______________ + +example = 'LennardJones', mpnn_type = 'DimeNet' + + @pytest.mark.parametrize( + "mpnn_type", + [ + "PNAPlus", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + "PAINN", + "MACE", + ], + ) + @pytest.mark.parametrize("example", ["LennardJones"]) + @pytest.mark.mpi_skip() + def pytest_examples_grad_forces(example, mpnn_type): + path = os.path.join(os.path.dirname(__file__), "..", "examples", example) + file_path = os.path.join(path, example + ".py") + + # Add the --mpnn_type argument for the subprocess call + return_code = subprocess.call(["python", file_path, "--mpnn_type", mpnn_type]) + + # Check the file ran without error. +> assert return_code == 0 +E assert 1 == 0 + +tests/test_examples.py:87: AssertionError +----------------------------- Captured stderr call ----------------------------- +Traceback (most recent call last): + File "/Users/7ml/Documents/Codes/HydraGNN/tests/../examples/LennardJones/LennardJones.py", line 33, in + import hydragnn +ModuleNotFoundError: No module named 'hydragnn' +________________ pytest_examples_grad_forces[LennardJones-EGNN] ________________ + +example = 'LennardJones', mpnn_type = 'EGNN' + + @pytest.mark.parametrize( + "mpnn_type", + [ + "PNAPlus", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + "PAINN", + "MACE", + ], + ) + @pytest.mark.parametrize("example", ["LennardJones"]) + @pytest.mark.mpi_skip() + def pytest_examples_grad_forces(example, mpnn_type): + path = os.path.join(os.path.dirname(__file__), "..", "examples", example) + file_path = os.path.join(path, example + ".py") + + # Add the --mpnn_type argument for the subprocess call + return_code = subprocess.call(["python", file_path, "--mpnn_type", mpnn_type]) + + # Check the file ran without error. +> assert return_code == 0 +E assert 1 == 0 + +tests/test_examples.py:87: AssertionError +----------------------------- Captured stderr call ----------------------------- +Traceback (most recent call last): + File "/Users/7ml/Documents/Codes/HydraGNN/tests/../examples/LennardJones/LennardJones.py", line 33, in + import hydragnn +ModuleNotFoundError: No module named 'hydragnn' +_______________ pytest_examples_grad_forces[LennardJones-PNAEq] ________________ + +example = 'LennardJones', mpnn_type = 'PNAEq' + + @pytest.mark.parametrize( + "mpnn_type", + [ + "PNAPlus", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + "PAINN", + "MACE", + ], + ) + @pytest.mark.parametrize("example", ["LennardJones"]) + @pytest.mark.mpi_skip() + def pytest_examples_grad_forces(example, mpnn_type): + path = os.path.join(os.path.dirname(__file__), "..", "examples", example) + file_path = os.path.join(path, example + ".py") + + # Add the --mpnn_type argument for the subprocess call + return_code = subprocess.call(["python", file_path, "--mpnn_type", mpnn_type]) + + # Check the file ran without error. +> assert return_code == 0 +E assert 1 == 0 + +tests/test_examples.py:87: AssertionError +----------------------------- Captured stderr call ----------------------------- +Traceback (most recent call last): + File "/Users/7ml/Documents/Codes/HydraGNN/tests/../examples/LennardJones/LennardJones.py", line 33, in + import hydragnn +ModuleNotFoundError: No module named 'hydragnn' +_______________ pytest_examples_grad_forces[LennardJones-PAINN] ________________ + +example = 'LennardJones', mpnn_type = 'PAINN' + + @pytest.mark.parametrize( + "mpnn_type", + [ + "PNAPlus", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + "PAINN", + "MACE", + ], + ) + @pytest.mark.parametrize("example", ["LennardJones"]) + @pytest.mark.mpi_skip() + def pytest_examples_grad_forces(example, mpnn_type): + path = os.path.join(os.path.dirname(__file__), "..", "examples", example) + file_path = os.path.join(path, example + ".py") + + # Add the --mpnn_type argument for the subprocess call + return_code = subprocess.call(["python", file_path, "--mpnn_type", mpnn_type]) + + # Check the file ran without error. +> assert return_code == 0 +E assert 1 == 0 + +tests/test_examples.py:87: AssertionError +----------------------------- Captured stderr call ----------------------------- +Traceback (most recent call last): + File "/Users/7ml/Documents/Codes/HydraGNN/tests/../examples/LennardJones/LennardJones.py", line 33, in + import hydragnn +ModuleNotFoundError: No module named 'hydragnn' +________________ pytest_examples_grad_forces[LennardJones-MACE] ________________ + +example = 'LennardJones', mpnn_type = 'MACE' + + @pytest.mark.parametrize( + "mpnn_type", + [ + "PNAPlus", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + "PAINN", + "MACE", + ], + ) + @pytest.mark.parametrize("example", ["LennardJones"]) + @pytest.mark.mpi_skip() + def pytest_examples_grad_forces(example, mpnn_type): + path = os.path.join(os.path.dirname(__file__), "..", "examples", example) + file_path = os.path.join(path, example + ".py") + + # Add the --mpnn_type argument for the subprocess call + return_code = subprocess.call(["python", file_path, "--mpnn_type", mpnn_type]) + + # Check the file ran without error. +> assert return_code == 0 +E assert 1 == 0 + +tests/test_examples.py:87: AssertionError +----------------------------- Captured stderr call ----------------------------- +Traceback (most recent call last): + File "/Users/7ml/Documents/Codes/HydraGNN/tests/../examples/LennardJones/LennardJones.py", line 33, in + import hydragnn +ModuleNotFoundError: No module named 'hydragnn' +=========================== short test summary info ============================ +FAILED tests/test_examples.py::pytest_examples_energy[qm9-SAGE-multihead-GPS] +FAILED tests/test_examples.py::pytest_examples_energy[qm9-GIN-multihead-GPS] +FAILED tests/test_examples.py::pytest_examples_energy[qm9-GAT-multihead-GPS] +FAILED tests/test_examples.py::pytest_examples_energy[qm9-MFC-multihead-GPS] +FAILED tests/test_examples.py::pytest_examples_energy[qm9-PNA-multihead-GPS] +FAILED tests/test_examples.py::pytest_examples_energy[qm9-PNAPlus-multihead-GPS] +FAILED tests/test_examples.py::pytest_examples_energy[qm9-SchNet-multihead-GPS] +FAILED tests/test_examples.py::pytest_examples_energy[qm9-DimeNet-multihead-GPS] +FAILED tests/test_examples.py::pytest_examples_energy[qm9-EGNN-multihead-GPS] +FAILED tests/test_examples.py::pytest_examples_energy[qm9-PNAEq-multihead-GPS] +FAILED tests/test_examples.py::pytest_examples_energy[qm9-PAINN-multihead-GPS] +FAILED tests/test_examples.py::pytest_examples_energy[md17-SAGE-multihead-GPS] +FAILED tests/test_examples.py::pytest_examples_energy[md17-GIN-multihead-GPS] +FAILED tests/test_examples.py::pytest_examples_energy[md17-GAT-multihead-GPS] +FAILED tests/test_examples.py::pytest_examples_energy[md17-MFC-multihead-GPS] +FAILED tests/test_examples.py::pytest_examples_energy[md17-PNA-multihead-GPS] +FAILED tests/test_examples.py::pytest_examples_energy[md17-PNAPlus-multihead-GPS] +FAILED tests/test_examples.py::pytest_examples_energy[md17-SchNet-multihead-GPS] +FAILED tests/test_examples.py::pytest_examples_energy[md17-DimeNet-multihead-GPS] +FAILED tests/test_examples.py::pytest_examples_energy[md17-EGNN-multihead-GPS] +FAILED tests/test_examples.py::pytest_examples_energy[md17-PNAEq-multihead-GPS] +FAILED tests/test_examples.py::pytest_examples_energy[md17-PAINN-multihead-GPS] +FAILED tests/test_examples.py::pytest_examples_grad_forces[LennardJones-PNAPlus] +FAILED tests/test_examples.py::pytest_examples_grad_forces[LennardJones-SchNet] +FAILED tests/test_examples.py::pytest_examples_grad_forces[LennardJones-DimeNet] +FAILED tests/test_examples.py::pytest_examples_grad_forces[LennardJones-EGNN] +FAILED tests/test_examples.py::pytest_examples_grad_forces[LennardJones-PNAEq] +FAILED tests/test_examples.py::pytest_examples_grad_forces[LennardJones-PAINN] +FAILED tests/test_examples.py::pytest_examples_grad_forces[LennardJones-MACE] +============================= 29 failed in 48.23s ============================== diff --git a/test_graphs_complete.log b/test_graphs_complete.log new file mode 100644 index 000000000..3be03417b --- /dev/null +++ b/test_graphs_complete.log @@ -0,0 +1,539 @@ +============================= test session starts ============================== +platform darwin -- Python 3.13.7, pytest-8.4.2, pluggy-1.6.0 -- /Users/7ml/Documents/Codes/HydraGNN/.venv/bin/python +cachedir: .pytest_cache +rootdir: /Users/7ml/Documents/Codes/HydraGNN +configfile: pytest.ini +collecting ... collected 68 items + +tests/test_graphs.py::pytest_train_model[ci.json-SAGE] PASSED [ 1%] +tests/test_graphs.py::pytest_train_model[ci.json-GIN] PASSED [ 2%] +tests/test_graphs.py::pytest_train_model[ci.json-GAT] PASSED [ 4%] +tests/test_graphs.py::pytest_train_model[ci.json-MFC] PASSED [ 5%] +tests/test_graphs.py::pytest_train_model[ci.json-PNA] PASSED [ 7%] +tests/test_graphs.py::pytest_train_model[ci.json-PNAPlus] PASSED [ 8%] +tests/test_graphs.py::pytest_train_model[ci.json-CGCNN] PASSED [ 10%] +tests/test_graphs.py::pytest_train_model[ci.json-SchNet] PASSED [ 11%] +tests/test_graphs.py::pytest_train_model[ci.json-DimeNet] PASSED [ 13%] +tests/test_graphs.py::pytest_train_model[ci.json-EGNN] PASSED [ 14%] +tests/test_graphs.py::pytest_train_model[ci.json-PNAEq] PASSED [ 16%] +tests/test_graphs.py::pytest_train_model[ci.json-PAINN] PASSED [ 17%] +tests/test_graphs.py::pytest_train_model[ci.json-MACE] PASSED [ 19%] +tests/test_graphs.py::pytest_train_model[ci_multihead.json-SAGE] PASSED [ 20%] +tests/test_graphs.py::pytest_train_model[ci_multihead.json-GIN] PASSED [ 22%] +tests/test_graphs.py::pytest_train_model[ci_multihead.json-GAT] PASSED [ 23%] +tests/test_graphs.py::pytest_train_model[ci_multihead.json-MFC] PASSED [ 25%] +tests/test_graphs.py::pytest_train_model[ci_multihead.json-PNA] PASSED [ 26%] +tests/test_graphs.py::pytest_train_model[ci_multihead.json-PNAPlus] PASSED [ 27%] +tests/test_graphs.py::pytest_train_model[ci_multihead.json-CGCNN] PASSED [ 29%] +tests/test_graphs.py::pytest_train_model[ci_multihead.json-SchNet] PASSED [ 30%] +tests/test_graphs.py::pytest_train_model[ci_multihead.json-DimeNet] PASSED [ 32%] +tests/test_graphs.py::pytest_train_model[ci_multihead.json-EGNN] PASSED [ 33%] +tests/test_graphs.py::pytest_train_model[ci_multihead.json-PNAEq] PASSED [ 35%] +tests/test_graphs.py::pytest_train_model[ci_multihead.json-PAINN] PASSED [ 36%] +tests/test_graphs.py::pytest_train_model[ci_multihead.json-MACE] PASSED [ 38%] +tests/test_graphs.py::pytest_train_model_lengths[GAT] PASSED [ 39%] +tests/test_graphs.py::pytest_train_model_lengths[PNA] PASSED [ 41%] +tests/test_graphs.py::pytest_train_model_lengths[PNAPlus] PASSED [ 42%] +tests/test_graphs.py::pytest_train_model_lengths[CGCNN] PASSED [ 44%] +tests/test_graphs.py::pytest_train_model_lengths[SchNet] PASSED [ 45%] +tests/test_graphs.py::pytest_train_model_lengths[DimeNet] PASSED [ 47%] +tests/test_graphs.py::pytest_train_model_lengths[EGNN] PASSED [ 48%] +tests/test_graphs.py::pytest_train_model_lengths[PNAEq] PASSED [ 50%] +tests/test_graphs.py::pytest_train_model_lengths[PAINN] PASSED [ 51%] +tests/test_graphs.py::pytest_train_model_lengths_global_attention[GAT-multihead-GPS] PASSED [ 52%] +tests/test_graphs.py::pytest_train_model_lengths_global_attention[PNA-multihead-GPS] PASSED [ 54%] +tests/test_graphs.py::pytest_train_model_lengths_global_attention[PNAPlus-multihead-GPS] PASSED [ 55%] +tests/test_graphs.py::pytest_train_model_lengths_global_attention[CGCNN-multihead-GPS] PASSED [ 57%] +tests/test_graphs.py::pytest_train_model_lengths_global_attention[SchNet-multihead-GPS] PASSED [ 58%] +tests/test_graphs.py::pytest_train_model_lengths_global_attention[DimeNet-multihead-GPS] PASSED [ 60%] +tests/test_graphs.py::pytest_train_model_lengths_global_attention[EGNN-multihead-GPS] PASSED [ 61%] +tests/test_graphs.py::pytest_train_model_lengths_global_attention[PNAEq-multihead-GPS] PASSED [ 63%] +tests/test_graphs.py::pytest_train_model_lengths_global_attention[PAINN-multihead-GPS] PASSED [ 64%] +tests/test_graphs.py::pytest_train_mace_model_lengths[MACE] PASSED [ 66%] +tests/test_graphs.py::pytest_train_equivariant_model[EGNN] PASSED [ 67%] +tests/test_graphs.py::pytest_train_equivariant_model[SchNet] PASSED [ 69%] +tests/test_graphs.py::pytest_train_equivariant_model[PNAEq] PASSED [ 70%] +tests/test_graphs.py::pytest_train_equivariant_model[PAINN] PASSED [ 72%] +tests/test_graphs.py::pytest_train_equivariant_model[MACE] PASSED [ 73%] +tests/test_graphs.py::pytest_train_model_vectoroutput[GAT] PASSED [ 75%] +tests/test_graphs.py::pytest_train_model_vectoroutput[PNA] FAILED [ 76%] +tests/test_graphs.py::pytest_train_model_vectoroutput[PNAPlus] PASSED [ 77%] +tests/test_graphs.py::pytest_train_model_vectoroutput[SchNet] PASSED [ 79%] +tests/test_graphs.py::pytest_train_model_vectoroutput[DimeNet] PASSED [ 80%] +tests/test_graphs.py::pytest_train_model_vectoroutput[EGNN] PASSED [ 82%] +tests/test_graphs.py::pytest_train_model_vectoroutput[PNAEq] PASSED [ 83%] +tests/test_graphs.py::pytest_train_model_conv_head[SAGE] PASSED [ 85%] +tests/test_graphs.py::pytest_train_model_conv_head[GIN] PASSED [ 86%] +tests/test_graphs.py::pytest_train_model_conv_head[GAT] PASSED [ 88%] +tests/test_graphs.py::pytest_train_model_conv_head[MFC] PASSED [ 89%] +tests/test_graphs.py::pytest_train_model_conv_head[PNA] PASSED [ 91%] +tests/test_graphs.py::pytest_train_model_conv_head[PNAPlus] PASSED [ 92%] +tests/test_graphs.py::pytest_train_model_conv_head[SchNet] PASSED [ 94%] +tests/test_graphs.py::pytest_train_model_conv_head[DimeNet] PASSED [ 95%] +tests/test_graphs.py::pytest_train_model_conv_head[EGNN] PASSED [ 97%] +tests/test_graphs.py::pytest_train_model_conv_head[PNAEq] PASSED [ 98%] +tests/test_graphs.py::pytest_train_model_conv_head[PAINN] PASSED [100%] + +=================================== FAILURES =================================== +_____________________ pytest_train_model_vectoroutput[PNA] _____________________ + +mpnn_type = 'PNA', overwrite_data = False + + @pytest.mark.parametrize( + "mpnn_type", + [ + "GAT", + "PNA", + "PNAPlus", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + ], + ) + def pytest_train_model_vectoroutput(mpnn_type, overwrite_data=False): +> unittest_train_model( + mpnn_type, None, None, "ci_vectoroutput.json", True, overwrite_data + ) + +tests/test_graphs.py:284: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +mpnn_type = 'PNA', global_attn_engine = None, global_attn_type = None +ci_input = 'ci_vectoroutput.json', use_lengths = True, overwrite_data = False +use_deepspeed = False, overwrite_config = None + + def unittest_train_model( + mpnn_type, + global_attn_engine, + global_attn_type, + ci_input, + use_lengths, + overwrite_data=False, + use_deepspeed=False, + overwrite_config=None, + ): + world_size, rank = hydragnn.utils.distributed.get_comm_size_and_rank() + + os.environ["SERIALIZED_DATA_PATH"] = os.getcwd() + + # Read in config settings and override model type. + config_file = os.path.join(os.getcwd(), "tests/inputs", ci_input) + with open(config_file, "r") as f: + config = json.load(f) + config["NeuralNetwork"]["Architecture"]["global_attn_engine"] = global_attn_engine + config["NeuralNetwork"]["Architecture"]["global_attn_type"] = global_attn_type + config["NeuralNetwork"]["Architecture"]["mpnn_type"] = mpnn_type + + # Overwrite config settings if provided + if overwrite_config: + config = merge_config(config, overwrite_config) + + """ + to test this locally, set ci.json as + "Dataset": { + ... + "path": { + "train": "serialized_dataset/unit_test_singlehead_train.pkl", + "test": "serialized_dataset/unit_test_singlehead_test.pkl", + "validate": "serialized_dataset/unit_test_singlehead_validate.pkl"} + ... + """ + # use pkl files if exist by default + for dataset_name in config["Dataset"]["path"].keys(): + if dataset_name == "total": + pkl_file = ( + os.environ["SERIALIZED_DATA_PATH"] + + "/serialized_dataset/" + + config["Dataset"]["name"] + + ".pkl" + ) + else: + pkl_file = ( + os.environ["SERIALIZED_DATA_PATH"] + + "/serialized_dataset/" + + config["Dataset"]["name"] + + "_" + + dataset_name + + ".pkl" + ) + if os.path.exists(pkl_file): + config["Dataset"]["path"][dataset_name] = pkl_file + + # In the unit test runs, it is found MFC favors graph-level features over node-level features, compared with other models; + # hence here we decrease the loss weight coefficient for graph-level head in MFC. + if mpnn_type == "MFC" and ci_input == "ci_multihead.json": + config["NeuralNetwork"]["Architecture"]["task_weights"][0] = 2 + + # Only run with edge lengths for models that support them. + if use_lengths: + config["NeuralNetwork"]["Architecture"]["edge_features"] = ["lengths"] + + if rank == 0: + num_samples_tot = 500 + # check if serialized pickle files or folders for raw files provided + pkl_input = False + if list(config["Dataset"]["path"].values())[0].endswith(".pkl"): + pkl_input = True + # only generate new datasets, if not pkl + if not pkl_input: + for dataset_name, data_path in config["Dataset"]["path"].items(): + if overwrite_data: + shutil.rmtree(data_path) + if not os.path.exists(data_path): + os.makedirs(data_path) + if dataset_name == "total": + num_samples = num_samples_tot + elif dataset_name == "train": + num_samples = int( + num_samples_tot + * config["NeuralNetwork"]["Training"]["perc_train"] + ) + elif dataset_name == "test": + num_samples = int( + num_samples_tot + * (1 - config["NeuralNetwork"]["Training"]["perc_train"]) + * 0.5 + ) + elif dataset_name == "validate": + num_samples = int( + num_samples_tot + * (1 - config["NeuralNetwork"]["Training"]["perc_train"]) + * 0.5 + ) + if not os.listdir(data_path): + tests.deterministic_graph_data( + data_path, number_configurations=num_samples + ) + MPI.COMM_WORLD.Barrier() + + # Since the config file uses PNA already, test the file overload here. + # All the other models need to use the locally modified dictionary. + if mpnn_type == "PNA" and not use_lengths: + hydragnn.run_training(config_file, use_deepspeed) + else: + hydragnn.run_training(config, use_deepspeed) + + ( + error, + error_mse_task, + true_values, + predicted_values, + ) = hydragnn.run_prediction(config, use_deepspeed) + + # Set RMSE and sample MAE error thresholds + thresholds = { + "SAGE": [0.20, 0.20], + "PNA": [0.20, 0.20], + "PNAPlus": [0.20, 0.20], + "MFC": [0.20, 0.30], + "GIN": [0.25, 0.20], + "GAT": [0.60, 0.70], + "CGCNN": [0.50, 0.40], + "SchNet": [0.20, 0.20], + "DimeNet": [0.50, 0.50], + "EGNN": [0.20, 0.20], + "PNAEq": [0.60, 0.60], + "PAINN": [0.60, 0.60], + "MACE": [0.60, 0.70], + } + if use_lengths and ("vector" not in ci_input): + thresholds["CGCNN"] = [0.175, 0.175] + thresholds["PNA"] = [0.10, 0.10] + thresholds["PNAPlus"] = [0.10, 0.10] + if use_lengths and "vector" in ci_input: + thresholds["PNA"] = [0.2, 0.15] + thresholds["PNAPlus"] = [0.2, 0.15] + if ci_input == "ci_conv_head.json": + thresholds["GIN"] = [0.26, 0.51] + thresholds["SchNet"] = [0.30, 0.30] + + verbosity = 2 + + for ihead in range(len(true_values)): + error_head_mse = error_mse_task[ihead] + error_str = ( + str("{:.6f}".format(error_head_mse)) + " < " + str(thresholds[mpnn_type][0]) + ) + hydragnn.utils.print.print_distributed(verbosity, "head: " + error_str) + assert ( + error_head_mse < thresholds[mpnn_type][0] + ), "Head RMSE checking failed for " + str(ihead) + + head_true = true_values[ihead] + head_pred = predicted_values[ihead] + # Check individual samples + mae = torch.nn.L1Loss() + sample_mean_abs_error = mae(head_true, head_pred) + error_str = ( + "{:.6f}".format(sample_mean_abs_error) + + " < " + + str(thresholds[mpnn_type][1]) + ) +> assert ( + sample_mean_abs_error < thresholds[mpnn_type][1] + ), f"MAE sample checking failed! MAE: {sample_mean_abs_error:.6f} >= threshold: {thresholds[mpnn_type][1]} for model: {mpnn_type}" +E AssertionError: MAE sample checking failed! MAE: 0.152753 >= threshold: 0.15 for model: PNA +E assert tensor(0.1528) < 0.15 + +tests/test_graphs.py:192: AssertionError +----------------------------- Captured stdout call ----------------------------- +dist.is_initialized(),sync_batch_norm,device_name: True False cpu +Using FSDP: False Sharding: ShardingStrategy.FULL_SHARD +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +dist.is_initialized(),sync_batch_norm,device_name: True False cpu +Using FSDP: False Sharding: ShardingStrategy.FULL_SHARD +get_autocast_and_scaler use_bf16: False +----------------------------- Captured stderr call ----------------------------- + Degree max: 0%| | 0/354 [00:00 unittest_train_model(mpnn_type, None, None, ci_input, False, overwrite_data) + +tests/test_graphs.py:223: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_graphs.py:134: in unittest_train_model + hydragnn.run_training(config, use_deepspeed) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:97: in _ + model = get_distributed_model( +hydragnn/utils/distributed/distributed.py:379: in get_distributed_model + model = DDP(model, **ddp_kwargs) + ^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/nn/parallel/distributed.py:802: in __init__ + self._log_and_throw( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +self = DistributedDataParallel( + (module): SAGEStack( + (graph_convs): ModuleList( + (0): Sequential( + (0) - SA... (1): ReLU() + (2): Linear(in_features=10, out_features=10, bias=True) + (3): ReLU() + ) + ) + ) +) +err_type = +err_msg = "Modules with uninitialized parameters can't be used with `DistributedDataParallel`. Run a dummy forward pass to correctly initialize the modules" + + def _log_and_throw(self, err_type, err_msg): + if self.logger is not None: + self.logger.set_error_and_log(f"{str(err_type)}: {err_msg}") +> raise err_type(err_msg) +E RuntimeError: Modules with uninitialized parameters can't be used with `DistributedDataParallel`. Run a dummy forward pass to correctly initialize the modules + +.venv/lib/python3.13/site-packages/torch/nn/parallel/distributed.py:1143: RuntimeError +----------------------------- Captured stdout call ----------------------------- +dist.is_initialized(),sync_batch_norm,device_name: True False cpu +Using FSDP: False Sharding: ShardingStrategy.FULL_SHARD +__________________ pytest_train_model[ci_multihead.json-GIN] ___________________ + +mpnn_type = 'GIN', ci_input = 'ci_multihead.json', overwrite_data = False + + @pytest.mark.parametrize( + "mpnn_type", + [ + "SAGE", + "GIN", + "GAT", + "MFC", + "PNA", + "PNAPlus", + "CGCNN", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + "PAINN", + "MACE", + ], + ) + @pytest.mark.parametrize("ci_input", ["ci.json", "ci_multihead.json"]) + def pytest_train_model(mpnn_type, ci_input, overwrite_data=False): +> unittest_train_model(mpnn_type, None, None, ci_input, False, overwrite_data) + +tests/test_graphs.py:223: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +mpnn_type = 'GIN', global_attn_engine = None, global_attn_type = None +ci_input = 'ci_multihead.json', use_lengths = False, overwrite_data = False +use_deepspeed = False, overwrite_config = None + + def unittest_train_model( + mpnn_type, + global_attn_engine, + global_attn_type, + ci_input, + use_lengths, + overwrite_data=False, + use_deepspeed=False, + overwrite_config=None, + ): + world_size, rank = hydragnn.utils.distributed.get_comm_size_and_rank() + + os.environ["SERIALIZED_DATA_PATH"] = os.getcwd() + + # Read in config settings and override model type. + config_file = os.path.join(os.getcwd(), "tests/inputs", ci_input) + with open(config_file, "r") as f: + config = json.load(f) + config["NeuralNetwork"]["Architecture"]["global_attn_engine"] = global_attn_engine + config["NeuralNetwork"]["Architecture"]["global_attn_type"] = global_attn_type + config["NeuralNetwork"]["Architecture"]["mpnn_type"] = mpnn_type + + # Overwrite config settings if provided + if overwrite_config: + config = merge_config(config, overwrite_config) + + """ + to test this locally, set ci.json as + "Dataset": { + ... + "path": { + "train": "serialized_dataset/unit_test_singlehead_train.pkl", + "test": "serialized_dataset/unit_test_singlehead_test.pkl", + "validate": "serialized_dataset/unit_test_singlehead_validate.pkl"} + ... + """ + # use pkl files if exist by default + for dataset_name in config["Dataset"]["path"].keys(): + if dataset_name == "total": + pkl_file = ( + os.environ["SERIALIZED_DATA_PATH"] + + "/serialized_dataset/" + + config["Dataset"]["name"] + + ".pkl" + ) + else: + pkl_file = ( + os.environ["SERIALIZED_DATA_PATH"] + + "/serialized_dataset/" + + config["Dataset"]["name"] + + "_" + + dataset_name + + ".pkl" + ) + if os.path.exists(pkl_file): + config["Dataset"]["path"][dataset_name] = pkl_file + + # In the unit test runs, it is found MFC favors graph-level features over node-level features, compared with other models; + # hence here we decrease the loss weight coefficient for graph-level head in MFC. + if mpnn_type == "MFC" and ci_input == "ci_multihead.json": + config["NeuralNetwork"]["Architecture"]["task_weights"][0] = 2 + + # Only run with edge lengths for models that support them. + if use_lengths: + config["NeuralNetwork"]["Architecture"]["edge_features"] = ["lengths"] + + if rank == 0: + num_samples_tot = 500 + # check if serialized pickle files or folders for raw files provided + pkl_input = False + if list(config["Dataset"]["path"].values())[0].endswith(".pkl"): + pkl_input = True + # only generate new datasets, if not pkl + if not pkl_input: + for dataset_name, data_path in config["Dataset"]["path"].items(): + if overwrite_data: + shutil.rmtree(data_path) + if not os.path.exists(data_path): + os.makedirs(data_path) + if dataset_name == "total": + num_samples = num_samples_tot + elif dataset_name == "train": + num_samples = int( + num_samples_tot + * config["NeuralNetwork"]["Training"]["perc_train"] + ) + elif dataset_name == "test": + num_samples = int( + num_samples_tot + * (1 - config["NeuralNetwork"]["Training"]["perc_train"]) + * 0.5 + ) + elif dataset_name == "validate": + num_samples = int( + num_samples_tot + * (1 - config["NeuralNetwork"]["Training"]["perc_train"]) + * 0.5 + ) + if not os.listdir(data_path): + tests.deterministic_graph_data( + data_path, number_configurations=num_samples + ) + MPI.COMM_WORLD.Barrier() + + # Since the config file uses PNA already, test the file overload here. + # All the other models need to use the locally modified dictionary. + if mpnn_type == "PNA" and not use_lengths: + hydragnn.run_training(config_file, use_deepspeed) + else: + hydragnn.run_training(config, use_deepspeed) + + ( + error, + error_mse_task, + true_values, + predicted_values, + ) = hydragnn.run_prediction(config, use_deepspeed) + + # Set RMSE and sample MAE error thresholds + thresholds = { + "SAGE": [0.20, 0.20], + "PNA": [0.20, 0.20], + "PNAPlus": [0.20, 0.20], + "MFC": [0.20, 0.30], + "GIN": [0.25, 0.20], + "GAT": [0.60, 0.70], + "CGCNN": [0.50, 0.40], + "SchNet": [0.20, 0.20], + "DimeNet": [0.50, 0.50], + "EGNN": [0.20, 0.20], + "PNAEq": [0.60, 0.60], + "PAINN": [0.60, 0.60], + "MACE": [0.60, 0.70], + } + if use_lengths and ("vector" not in ci_input): + thresholds["CGCNN"] = [0.175, 0.175] + thresholds["PNA"] = [0.10, 0.10] + thresholds["PNAPlus"] = [0.10, 0.10] + if use_lengths and "vector" in ci_input: + thresholds["PNA"] = [0.2, 0.15] + thresholds["PNAPlus"] = [0.2, 0.15] + if ci_input == "ci_conv_head.json": + thresholds["GIN"] = [0.26, 0.51] + thresholds["SchNet"] = [0.30, 0.30] + + verbosity = 2 + + for ihead in range(len(true_values)): + error_head_mse = error_mse_task[ihead] + error_str = ( + str("{:.6f}".format(error_head_mse)) + " < " + str(thresholds[mpnn_type][0]) + ) + hydragnn.utils.print.print_distributed(verbosity, "head: " + error_str) + assert ( + error_head_mse < thresholds[mpnn_type][0] + ), "Head RMSE checking failed for " + str(ihead) + + head_true = true_values[ihead] + head_pred = predicted_values[ihead] + # Check individual samples + mae = torch.nn.L1Loss() + sample_mean_abs_error = mae(head_true, head_pred) + error_str = ( + "{:.6f}".format(sample_mean_abs_error) + + " < " + + str(thresholds[mpnn_type][1]) + ) +> assert ( + sample_mean_abs_error < thresholds[mpnn_type][1] + ), f"MAE sample checking failed! MAE: {sample_mean_abs_error:.6f} >= threshold: {thresholds[mpnn_type][1]} for model: {mpnn_type}" +E AssertionError: MAE sample checking failed! MAE: 0.389315 >= threshold: 0.2 for model: GIN +E assert tensor(0.3893) < 0.2 + +tests/test_graphs.py:192: AssertionError +----------------------------- Captured stdout call ----------------------------- +dist.is_initialized(),sync_batch_norm,device_name: True False cpu +Using FSDP: False Sharding: ShardingStrategy.FULL_SHARD +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +dist.is_initialized(),sync_batch_norm,device_name: True False cpu +Using FSDP: False Sharding: ShardingStrategy.FULL_SHARD +get_autocast_and_scaler use_bf16: False +----------------------------- Captured stderr call ----------------------------- +0: Load existing model: ./logs/GIN-r-2.0-ncl-2-hd-8-ne-100-lr-0.01-bs-16-data-unit_test-node_ft--task_weights-20.0-1.0-1.0-1.0-/GIN-r-2.0-ncl-2-hd-8-ne-100-lr-0.01-bs-16-data-unit_test-node_ft--task_weights-20.0-1.0-1.0-1.0-.pk +0: head: 0.194092 < 0.25 +__________________ pytest_train_model[ci_multihead.json-GAT] ___________________ + +mpnn_type = 'GAT', ci_input = 'ci_multihead.json', overwrite_data = False + + @pytest.mark.parametrize( + "mpnn_type", + [ + "SAGE", + "GIN", + "GAT", + "MFC", + "PNA", + "PNAPlus", + "CGCNN", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + "PAINN", + "MACE", + ], + ) + @pytest.mark.parametrize("ci_input", ["ci.json", "ci_multihead.json"]) + def pytest_train_model(mpnn_type, ci_input, overwrite_data=False): +> unittest_train_model(mpnn_type, None, None, ci_input, False, overwrite_data) + +tests/test_graphs.py:223: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_graphs.py:134: in unittest_train_model + hydragnn.run_training(config, use_deepspeed) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:97: in _ + model = get_distributed_model( +hydragnn/utils/distributed/distributed.py:379: in get_distributed_model + model = DDP(model, **ddp_kwargs) + ^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/nn/parallel/distributed.py:802: in __init__ + self._log_and_throw( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +self = DistributedDataParallel( + (module): GATStack( + (graph_convs): ModuleList( + (0): Sequential( + (0) - GAT... (1): ReLU() + (2): Linear(in_features=10, out_features=10, bias=True) + (3): ReLU() + ) + ) + ) +) +err_type = +err_msg = "Modules with uninitialized parameters can't be used with `DistributedDataParallel`. Run a dummy forward pass to correctly initialize the modules" + + def _log_and_throw(self, err_type, err_msg): + if self.logger is not None: + self.logger.set_error_and_log(f"{str(err_type)}: {err_msg}") +> raise err_type(err_msg) +E RuntimeError: Modules with uninitialized parameters can't be used with `DistributedDataParallel`. Run a dummy forward pass to correctly initialize the modules + +.venv/lib/python3.13/site-packages/torch/nn/parallel/distributed.py:1143: RuntimeError +----------------------------- Captured stdout call ----------------------------- +dist.is_initialized(),sync_batch_norm,device_name: True False cpu +Using FSDP: False Sharding: ShardingStrategy.FULL_SHARD +__________________ pytest_train_model[ci_multihead.json-MFC] ___________________ + +mpnn_type = 'MFC', ci_input = 'ci_multihead.json', overwrite_data = False + + @pytest.mark.parametrize( + "mpnn_type", + [ + "SAGE", + "GIN", + "GAT", + "MFC", + "PNA", + "PNAPlus", + "CGCNN", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + "PAINN", + "MACE", + ], + ) + @pytest.mark.parametrize("ci_input", ["ci.json", "ci_multihead.json"]) + def pytest_train_model(mpnn_type, ci_input, overwrite_data=False): +> unittest_train_model(mpnn_type, None, None, ci_input, False, overwrite_data) + +tests/test_graphs.py:223: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_graphs.py:134: in unittest_train_model + hydragnn.run_training(config, use_deepspeed) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:97: in _ + model = get_distributed_model( +hydragnn/utils/distributed/distributed.py:379: in get_distributed_model + model = DDP(model, **ddp_kwargs) + ^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/nn/parallel/distributed.py:802: in __init__ + self._log_and_throw( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +self = DistributedDataParallel( + (module): MFCStack( + (graph_convs): ModuleList( + (0): Sequential( + (0) - MFC... (1): ReLU() + (2): Linear(in_features=10, out_features=10, bias=True) + (3): ReLU() + ) + ) + ) +) +err_type = +err_msg = "Modules with uninitialized parameters can't be used with `DistributedDataParallel`. Run a dummy forward pass to correctly initialize the modules" + + def _log_and_throw(self, err_type, err_msg): + if self.logger is not None: + self.logger.set_error_and_log(f"{str(err_type)}: {err_msg}") +> raise err_type(err_msg) +E RuntimeError: Modules with uninitialized parameters can't be used with `DistributedDataParallel`. Run a dummy forward pass to correctly initialize the modules + +.venv/lib/python3.13/site-packages/torch/nn/parallel/distributed.py:1143: RuntimeError +----------------------------- Captured stdout call ----------------------------- +dist.is_initialized(),sync_batch_norm,device_name: True False cpu +Using FSDP: False Sharding: ShardingStrategy.FULL_SHARD +__________________ pytest_train_model[ci_multihead.json-PNA] ___________________ + +mpnn_type = 'PNA', ci_input = 'ci_multihead.json', overwrite_data = False + + @pytest.mark.parametrize( + "mpnn_type", + [ + "SAGE", + "GIN", + "GAT", + "MFC", + "PNA", + "PNAPlus", + "CGCNN", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + "PAINN", + "MACE", + ], + ) + @pytest.mark.parametrize("ci_input", ["ci.json", "ci_multihead.json"]) + def pytest_train_model(mpnn_type, ci_input, overwrite_data=False): +> unittest_train_model(mpnn_type, None, None, ci_input, False, overwrite_data) + +tests/test_graphs.py:223: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_graphs.py:132: in unittest_train_model + hydragnn.run_training(config_file, use_deepspeed) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:62: in _ + run_training(config) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:97: in _ + model = get_distributed_model( +hydragnn/utils/distributed/distributed.py:379: in get_distributed_model + model = DDP(model, **ddp_kwargs) + ^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/nn/parallel/distributed.py:802: in __init__ + self._log_and_throw( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +self = DistributedDataParallel( + (module): PNAStack( + (graph_convs): ModuleList( + (0): Sequential( + (0) - PNA... (1): ReLU() + (2): Linear(in_features=10, out_features=10, bias=True) + (3): ReLU() + ) + ) + ) +) +err_type = +err_msg = "Modules with uninitialized parameters can't be used with `DistributedDataParallel`. Run a dummy forward pass to correctly initialize the modules" + + def _log_and_throw(self, err_type, err_msg): + if self.logger is not None: + self.logger.set_error_and_log(f"{str(err_type)}: {err_msg}") +> raise err_type(err_msg) +E RuntimeError: Modules with uninitialized parameters can't be used with `DistributedDataParallel`. Run a dummy forward pass to correctly initialize the modules + +.venv/lib/python3.13/site-packages/torch/nn/parallel/distributed.py:1143: RuntimeError +----------------------------- Captured stdout call ----------------------------- +dist.is_initialized(),sync_batch_norm,device_name: True False cpu +Using FSDP: False Sharding: ShardingStrategy.FULL_SHARD +----------------------------- Captured stderr call ----------------------------- + Degree max: 0%| | 0/354 [00:00 unittest_train_model(mpnn_type, None, None, ci_input, False, overwrite_data) + +tests/test_graphs.py:223: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_graphs.py:134: in unittest_train_model + hydragnn.run_training(config, use_deepspeed) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:97: in _ + model = get_distributed_model( +hydragnn/utils/distributed/distributed.py:379: in get_distributed_model + model = DDP(model, **ddp_kwargs) + ^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/nn/parallel/distributed.py:802: in __init__ + self._log_and_throw( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +self = DistributedDataParallel( + (module): PNAPlusStack( + (graph_convs): ModuleList( + (0): Sequential( + (0) -...=10, bias=True) + (3): ReLU() + ) + ) + (rbf): BesselBasisLayer( + (envelope): Envelope() + ) + ) +) +err_type = +err_msg = "Modules with uninitialized parameters can't be used with `DistributedDataParallel`. Run a dummy forward pass to correctly initialize the modules" + + def _log_and_throw(self, err_type, err_msg): + if self.logger is not None: + self.logger.set_error_and_log(f"{str(err_type)}: {err_msg}") +> raise err_type(err_msg) +E RuntimeError: Modules with uninitialized parameters can't be used with `DistributedDataParallel`. Run a dummy forward pass to correctly initialize the modules + +.venv/lib/python3.13/site-packages/torch/nn/parallel/distributed.py:1143: RuntimeError +----------------------------- Captured stdout call ----------------------------- +dist.is_initialized(),sync_batch_norm,device_name: True False cpu +Using FSDP: False Sharding: ShardingStrategy.FULL_SHARD +----------------------------- Captured stderr call ----------------------------- + Degree max: 0%| | 0/354 [00:00 unittest_train_model(mpnn_type, None, None, ci_input, False, overwrite_data) + +tests/test_graphs.py:223: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_graphs.py:134: in unittest_train_model + hydragnn.run_training(config, use_deepspeed) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:176: in _ + train_validate_test( +hydragnn/train/train_validate_test.py:151: in train_validate_test + _, _, true_values, predicted_values = test(test_loader, model, verbosity) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/utils/_contextlib.py:120: in decorate_context + return func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +hydragnn/train/train_validate_test.py:728: in test + pred = model(data) + ^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/nn/modules/module.py:1773: in _wrapped_call_impl + return self._call_impl(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/nn/modules/module.py:1784: in _call_impl + return forward_call(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/nn/parallel/distributed.py:1648: in forward + else self._run_ddp_forward(*inputs, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/nn/parallel/distributed.py:1474: in _run_ddp_forward + return self.module(*inputs, **kwargs) # type: ignore[index] + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/nn/modules/module.py:1773: in _wrapped_call_impl + return self._call_impl(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/nn/modules/module.py:1784: in _call_impl + return forward_call(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/models/Base.py:466: in forward + inv_node_feat = self.activation_function(feat_layer(inv_node_feat)) + ^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/nn/modules/module.py:1773: in _wrapped_call_impl + return self._call_impl(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/nn/modules/module.py:1784: in _call_impl + return forward_call(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch_geometric/nn/norm/batch_norm.py:88: in forward + return self.module(x) + ^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/nn/modules/module.py:1773: in _wrapped_call_impl + return self._call_impl(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/nn/modules/module.py:1784: in _call_impl + return forward_call(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/nn/modules/batchnorm.py:193: in forward + return F.batch_norm( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +input = tensor([], size=(76, 0)), running_mean = tensor([]) +running_var = tensor([]) +weight = Parameter containing: +tensor([], requires_grad=True) +bias = Parameter containing: +tensor([], requires_grad=True), training = False +momentum = 0.1, eps = 1e-05 + + def batch_norm( + input: Tensor, + running_mean: Optional[Tensor], + running_var: Optional[Tensor], + weight: Optional[Tensor] = None, + bias: Optional[Tensor] = None, + training: bool = False, + momentum: float = 0.1, + eps: float = 1e-5, + ) -> Tensor: + r"""Apply Batch Normalization for each channel across a batch of data. + + See :class:`~torch.nn.BatchNorm1d`, :class:`~torch.nn.BatchNorm2d`, + :class:`~torch.nn.BatchNorm3d` for details. + """ + if has_torch_function_variadic(input, running_mean, running_var, weight, bias): + return handle_torch_function( + batch_norm, + (input, running_mean, running_var, weight, bias), + input, + running_mean, + running_var, + weight=weight, + bias=bias, + training=training, + momentum=momentum, + eps=eps, + ) + if training: + _verify_batch_size(input.size()) + +> return torch.batch_norm( + input, + weight, + bias, + running_mean, + running_var, + training, + momentum, + eps, + torch.backends.cudnn.enabled, + ) +E IndexError: select(): index 0 out of range for tensor of size [0] at dimension 0 + +.venv/lib/python3.13/site-packages/torch/nn/functional.py:2817: IndexError +----------------------------- Captured stdout call ----------------------------- +dist.is_initialized(),sync_batch_norm,device_name: True False cpu +Using FSDP: False Sharding: ShardingStrategy.FULL_SHARD +get_autocast_and_scaler use_bf16: False +_________________ pytest_train_model[ci_multihead.json-SchNet] _________________ + +mpnn_type = 'SchNet', ci_input = 'ci_multihead.json', overwrite_data = False + + @pytest.mark.parametrize( + "mpnn_type", + [ + "SAGE", + "GIN", + "GAT", + "MFC", + "PNA", + "PNAPlus", + "CGCNN", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + "PAINN", + "MACE", + ], + ) + @pytest.mark.parametrize("ci_input", ["ci.json", "ci_multihead.json"]) + def pytest_train_model(mpnn_type, ci_input, overwrite_data=False): +> unittest_train_model(mpnn_type, None, None, ci_input, False, overwrite_data) + +tests/test_graphs.py:223: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +mpnn_type = 'SchNet', global_attn_engine = None, global_attn_type = None +ci_input = 'ci_multihead.json', use_lengths = False, overwrite_data = False +use_deepspeed = False, overwrite_config = None + + def unittest_train_model( + mpnn_type, + global_attn_engine, + global_attn_type, + ci_input, + use_lengths, + overwrite_data=False, + use_deepspeed=False, + overwrite_config=None, + ): + world_size, rank = hydragnn.utils.distributed.get_comm_size_and_rank() + + os.environ["SERIALIZED_DATA_PATH"] = os.getcwd() + + # Read in config settings and override model type. + config_file = os.path.join(os.getcwd(), "tests/inputs", ci_input) + with open(config_file, "r") as f: + config = json.load(f) + config["NeuralNetwork"]["Architecture"]["global_attn_engine"] = global_attn_engine + config["NeuralNetwork"]["Architecture"]["global_attn_type"] = global_attn_type + config["NeuralNetwork"]["Architecture"]["mpnn_type"] = mpnn_type + + # Overwrite config settings if provided + if overwrite_config: + config = merge_config(config, overwrite_config) + + """ + to test this locally, set ci.json as + "Dataset": { + ... + "path": { + "train": "serialized_dataset/unit_test_singlehead_train.pkl", + "test": "serialized_dataset/unit_test_singlehead_test.pkl", + "validate": "serialized_dataset/unit_test_singlehead_validate.pkl"} + ... + """ + # use pkl files if exist by default + for dataset_name in config["Dataset"]["path"].keys(): + if dataset_name == "total": + pkl_file = ( + os.environ["SERIALIZED_DATA_PATH"] + + "/serialized_dataset/" + + config["Dataset"]["name"] + + ".pkl" + ) + else: + pkl_file = ( + os.environ["SERIALIZED_DATA_PATH"] + + "/serialized_dataset/" + + config["Dataset"]["name"] + + "_" + + dataset_name + + ".pkl" + ) + if os.path.exists(pkl_file): + config["Dataset"]["path"][dataset_name] = pkl_file + + # In the unit test runs, it is found MFC favors graph-level features over node-level features, compared with other models; + # hence here we decrease the loss weight coefficient for graph-level head in MFC. + if mpnn_type == "MFC" and ci_input == "ci_multihead.json": + config["NeuralNetwork"]["Architecture"]["task_weights"][0] = 2 + + # Only run with edge lengths for models that support them. + if use_lengths: + config["NeuralNetwork"]["Architecture"]["edge_features"] = ["lengths"] + + if rank == 0: + num_samples_tot = 500 + # check if serialized pickle files or folders for raw files provided + pkl_input = False + if list(config["Dataset"]["path"].values())[0].endswith(".pkl"): + pkl_input = True + # only generate new datasets, if not pkl + if not pkl_input: + for dataset_name, data_path in config["Dataset"]["path"].items(): + if overwrite_data: + shutil.rmtree(data_path) + if not os.path.exists(data_path): + os.makedirs(data_path) + if dataset_name == "total": + num_samples = num_samples_tot + elif dataset_name == "train": + num_samples = int( + num_samples_tot + * config["NeuralNetwork"]["Training"]["perc_train"] + ) + elif dataset_name == "test": + num_samples = int( + num_samples_tot + * (1 - config["NeuralNetwork"]["Training"]["perc_train"]) + * 0.5 + ) + elif dataset_name == "validate": + num_samples = int( + num_samples_tot + * (1 - config["NeuralNetwork"]["Training"]["perc_train"]) + * 0.5 + ) + if not os.listdir(data_path): + tests.deterministic_graph_data( + data_path, number_configurations=num_samples + ) + MPI.COMM_WORLD.Barrier() + + # Since the config file uses PNA already, test the file overload here. + # All the other models need to use the locally modified dictionary. + if mpnn_type == "PNA" and not use_lengths: + hydragnn.run_training(config_file, use_deepspeed) + else: + hydragnn.run_training(config, use_deepspeed) + + ( + error, + error_mse_task, + true_values, + predicted_values, + ) = hydragnn.run_prediction(config, use_deepspeed) + + # Set RMSE and sample MAE error thresholds + thresholds = { + "SAGE": [0.20, 0.20], + "PNA": [0.20, 0.20], + "PNAPlus": [0.20, 0.20], + "MFC": [0.20, 0.30], + "GIN": [0.25, 0.20], + "GAT": [0.60, 0.70], + "CGCNN": [0.50, 0.40], + "SchNet": [0.20, 0.20], + "DimeNet": [0.50, 0.50], + "EGNN": [0.20, 0.20], + "PNAEq": [0.60, 0.60], + "PAINN": [0.60, 0.60], + "MACE": [0.60, 0.70], + } + if use_lengths and ("vector" not in ci_input): + thresholds["CGCNN"] = [0.175, 0.175] + thresholds["PNA"] = [0.10, 0.10] + thresholds["PNAPlus"] = [0.10, 0.10] + if use_lengths and "vector" in ci_input: + thresholds["PNA"] = [0.2, 0.15] + thresholds["PNAPlus"] = [0.2, 0.15] + if ci_input == "ci_conv_head.json": + thresholds["GIN"] = [0.26, 0.51] + thresholds["SchNet"] = [0.30, 0.30] + + verbosity = 2 + + for ihead in range(len(true_values)): + error_head_mse = error_mse_task[ihead] + error_str = ( + str("{:.6f}".format(error_head_mse)) + " < " + str(thresholds[mpnn_type][0]) + ) + hydragnn.utils.print.print_distributed(verbosity, "head: " + error_str) + assert ( + error_head_mse < thresholds[mpnn_type][0] + ), "Head RMSE checking failed for " + str(ihead) + + head_true = true_values[ihead] + head_pred = predicted_values[ihead] + # Check individual samples + mae = torch.nn.L1Loss() + sample_mean_abs_error = mae(head_true, head_pred) + error_str = ( + "{:.6f}".format(sample_mean_abs_error) + + " < " + + str(thresholds[mpnn_type][1]) + ) +> assert ( + sample_mean_abs_error < thresholds[mpnn_type][1] + ), f"MAE sample checking failed! MAE: {sample_mean_abs_error:.6f} >= threshold: {thresholds[mpnn_type][1]} for model: {mpnn_type}" +E AssertionError: MAE sample checking failed! MAE: 0.339667 >= threshold: 0.2 for model: SchNet +E assert tensor(0.3397) < 0.2 + +tests/test_graphs.py:192: AssertionError +----------------------------- Captured stdout call ----------------------------- +dist.is_initialized(),sync_batch_norm,device_name: True False cpu +Using FSDP: False Sharding: ShardingStrategy.FULL_SHARD +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +dist.is_initialized(),sync_batch_norm,device_name: True False cpu +Using FSDP: False Sharding: ShardingStrategy.FULL_SHARD +get_autocast_and_scaler use_bf16: False +----------------------------- Captured stderr call ----------------------------- +0: Load existing model: ./logs/SchNet-r-2.0-ncl-2-hd-8-ne-100-lr-0.01-bs-16-data-unit_test-node_ft--task_weights-20.0-1.0-1.0-1.0-/SchNet-r-2.0-ncl-2-hd-8-ne-100-lr-0.01-bs-16-data-unit_test-node_ft--task_weights-20.0-1.0-1.0-1.0-.pk +0: head: 0.169193 < 0.2 +________________ pytest_train_model[ci_multihead.json-DimeNet] _________________ + +mpnn_type = 'DimeNet', ci_input = 'ci_multihead.json', overwrite_data = False + + @pytest.mark.parametrize( + "mpnn_type", + [ + "SAGE", + "GIN", + "GAT", + "MFC", + "PNA", + "PNAPlus", + "CGCNN", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + "PAINN", + "MACE", + ], + ) + @pytest.mark.parametrize("ci_input", ["ci.json", "ci_multihead.json"]) + def pytest_train_model(mpnn_type, ci_input, overwrite_data=False): +> unittest_train_model(mpnn_type, None, None, ci_input, False, overwrite_data) + +tests/test_graphs.py:223: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_graphs.py:134: in unittest_train_model + hydragnn.run_training(config, use_deepspeed) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:89: in _ + model = create_model_config( +hydragnn/models/create.py:45: in create_model_config + return create_model( +hydragnn/models/create.py:369: in create_model + model = DIMEStack( +hydragnn/models/DIMEStack.py:68: in __init__ + super().__init__(input_args, conv_args, *args, **kwargs) +hydragnn/models/Base.py:174: in __init__ + self._init_conv() +hydragnn/models/DIMEStack.py:80: in _init_conv + self.get_conv( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +self = DIMEStack( + (graph_convs): ModuleList() + (feature_layers): ModuleList() + (heads_NN): ModuleList() + (convs_node_hid...eDict() + (convs_node_output): ModuleDict() + (batch_norms_node_output): ModuleDict() + (activation_function): ReLU() +) +input_dim = 0, output_dim = 8, edge_dim = None + + def get_conv(self, input_dim, output_dim, edge_dim=None): + hidden_dim = output_dim if input_dim == 1 else input_dim + assert ( +> hidden_dim > 1 + ), "DimeNet requires more than one hidden dimension between input_dim and output_dim." +E AssertionError: DimeNet requires more than one hidden dimension between input_dim and output_dim. + +hydragnn/models/DIMEStack.py:99: AssertionError +__________________ pytest_train_model[ci_multihead.json-EGNN] __________________ + +mpnn_type = 'EGNN', ci_input = 'ci_multihead.json', overwrite_data = False + + @pytest.mark.parametrize( + "mpnn_type", + [ + "SAGE", + "GIN", + "GAT", + "MFC", + "PNA", + "PNAPlus", + "CGCNN", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + "PAINN", + "MACE", + ], + ) + @pytest.mark.parametrize("ci_input", ["ci.json", "ci_multihead.json"]) + def pytest_train_model(mpnn_type, ci_input, overwrite_data=False): +> unittest_train_model(mpnn_type, None, None, ci_input, False, overwrite_data) + +tests/test_graphs.py:223: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +mpnn_type = 'EGNN', global_attn_engine = None, global_attn_type = None +ci_input = 'ci_multihead.json', use_lengths = False, overwrite_data = False +use_deepspeed = False, overwrite_config = None + + def unittest_train_model( + mpnn_type, + global_attn_engine, + global_attn_type, + ci_input, + use_lengths, + overwrite_data=False, + use_deepspeed=False, + overwrite_config=None, + ): + world_size, rank = hydragnn.utils.distributed.get_comm_size_and_rank() + + os.environ["SERIALIZED_DATA_PATH"] = os.getcwd() + + # Read in config settings and override model type. + config_file = os.path.join(os.getcwd(), "tests/inputs", ci_input) + with open(config_file, "r") as f: + config = json.load(f) + config["NeuralNetwork"]["Architecture"]["global_attn_engine"] = global_attn_engine + config["NeuralNetwork"]["Architecture"]["global_attn_type"] = global_attn_type + config["NeuralNetwork"]["Architecture"]["mpnn_type"] = mpnn_type + + # Overwrite config settings if provided + if overwrite_config: + config = merge_config(config, overwrite_config) + + """ + to test this locally, set ci.json as + "Dataset": { + ... + "path": { + "train": "serialized_dataset/unit_test_singlehead_train.pkl", + "test": "serialized_dataset/unit_test_singlehead_test.pkl", + "validate": "serialized_dataset/unit_test_singlehead_validate.pkl"} + ... + """ + # use pkl files if exist by default + for dataset_name in config["Dataset"]["path"].keys(): + if dataset_name == "total": + pkl_file = ( + os.environ["SERIALIZED_DATA_PATH"] + + "/serialized_dataset/" + + config["Dataset"]["name"] + + ".pkl" + ) + else: + pkl_file = ( + os.environ["SERIALIZED_DATA_PATH"] + + "/serialized_dataset/" + + config["Dataset"]["name"] + + "_" + + dataset_name + + ".pkl" + ) + if os.path.exists(pkl_file): + config["Dataset"]["path"][dataset_name] = pkl_file + + # In the unit test runs, it is found MFC favors graph-level features over node-level features, compared with other models; + # hence here we decrease the loss weight coefficient for graph-level head in MFC. + if mpnn_type == "MFC" and ci_input == "ci_multihead.json": + config["NeuralNetwork"]["Architecture"]["task_weights"][0] = 2 + + # Only run with edge lengths for models that support them. + if use_lengths: + config["NeuralNetwork"]["Architecture"]["edge_features"] = ["lengths"] + + if rank == 0: + num_samples_tot = 500 + # check if serialized pickle files or folders for raw files provided + pkl_input = False + if list(config["Dataset"]["path"].values())[0].endswith(".pkl"): + pkl_input = True + # only generate new datasets, if not pkl + if not pkl_input: + for dataset_name, data_path in config["Dataset"]["path"].items(): + if overwrite_data: + shutil.rmtree(data_path) + if not os.path.exists(data_path): + os.makedirs(data_path) + if dataset_name == "total": + num_samples = num_samples_tot + elif dataset_name == "train": + num_samples = int( + num_samples_tot + * config["NeuralNetwork"]["Training"]["perc_train"] + ) + elif dataset_name == "test": + num_samples = int( + num_samples_tot + * (1 - config["NeuralNetwork"]["Training"]["perc_train"]) + * 0.5 + ) + elif dataset_name == "validate": + num_samples = int( + num_samples_tot + * (1 - config["NeuralNetwork"]["Training"]["perc_train"]) + * 0.5 + ) + if not os.listdir(data_path): + tests.deterministic_graph_data( + data_path, number_configurations=num_samples + ) + MPI.COMM_WORLD.Barrier() + + # Since the config file uses PNA already, test the file overload here. + # All the other models need to use the locally modified dictionary. + if mpnn_type == "PNA" and not use_lengths: + hydragnn.run_training(config_file, use_deepspeed) + else: + hydragnn.run_training(config, use_deepspeed) + + ( + error, + error_mse_task, + true_values, + predicted_values, + ) = hydragnn.run_prediction(config, use_deepspeed) + + # Set RMSE and sample MAE error thresholds + thresholds = { + "SAGE": [0.20, 0.20], + "PNA": [0.20, 0.20], + "PNAPlus": [0.20, 0.20], + "MFC": [0.20, 0.30], + "GIN": [0.25, 0.20], + "GAT": [0.60, 0.70], + "CGCNN": [0.50, 0.40], + "SchNet": [0.20, 0.20], + "DimeNet": [0.50, 0.50], + "EGNN": [0.20, 0.20], + "PNAEq": [0.60, 0.60], + "PAINN": [0.60, 0.60], + "MACE": [0.60, 0.70], + } + if use_lengths and ("vector" not in ci_input): + thresholds["CGCNN"] = [0.175, 0.175] + thresholds["PNA"] = [0.10, 0.10] + thresholds["PNAPlus"] = [0.10, 0.10] + if use_lengths and "vector" in ci_input: + thresholds["PNA"] = [0.2, 0.15] + thresholds["PNAPlus"] = [0.2, 0.15] + if ci_input == "ci_conv_head.json": + thresholds["GIN"] = [0.26, 0.51] + thresholds["SchNet"] = [0.30, 0.30] + + verbosity = 2 + + for ihead in range(len(true_values)): + error_head_mse = error_mse_task[ihead] + error_str = ( + str("{:.6f}".format(error_head_mse)) + " < " + str(thresholds[mpnn_type][0]) + ) + hydragnn.utils.print.print_distributed(verbosity, "head: " + error_str) + assert ( + error_head_mse < thresholds[mpnn_type][0] + ), "Head RMSE checking failed for " + str(ihead) + + head_true = true_values[ihead] + head_pred = predicted_values[ihead] + # Check individual samples + mae = torch.nn.L1Loss() + sample_mean_abs_error = mae(head_true, head_pred) + error_str = ( + "{:.6f}".format(sample_mean_abs_error) + + " < " + + str(thresholds[mpnn_type][1]) + ) +> assert ( + sample_mean_abs_error < thresholds[mpnn_type][1] + ), f"MAE sample checking failed! MAE: {sample_mean_abs_error:.6f} >= threshold: {thresholds[mpnn_type][1]} for model: {mpnn_type}" +E AssertionError: MAE sample checking failed! MAE: 0.339692 >= threshold: 0.2 for model: EGNN +E assert tensor(0.3397) < 0.2 + +tests/test_graphs.py:192: AssertionError +----------------------------- Captured stdout call ----------------------------- +dist.is_initialized(),sync_batch_norm,device_name: True False cpu +Using FSDP: False Sharding: ShardingStrategy.FULL_SHARD +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +dist.is_initialized(),sync_batch_norm,device_name: True False cpu +Using FSDP: False Sharding: ShardingStrategy.FULL_SHARD +get_autocast_and_scaler use_bf16: False +----------------------------- Captured stderr call ----------------------------- +0: Load existing model: ./logs/EGNN-r-2.0-ncl-2-hd-8-ne-100-lr-0.01-bs-16-data-unit_test-node_ft--task_weights-20.0-1.0-1.0-1.0-/EGNN-r-2.0-ncl-2-hd-8-ne-100-lr-0.01-bs-16-data-unit_test-node_ft--task_weights-20.0-1.0-1.0-1.0-.pk +0: head: 0.169195 < 0.2 +_________________ pytest_train_model[ci_multihead.json-PNAEq] __________________ + +mpnn_type = 'PNAEq', ci_input = 'ci_multihead.json', overwrite_data = False + + @pytest.mark.parametrize( + "mpnn_type", + [ + "SAGE", + "GIN", + "GAT", + "MFC", + "PNA", + "PNAPlus", + "CGCNN", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + "PAINN", + "MACE", + ], + ) + @pytest.mark.parametrize("ci_input", ["ci.json", "ci_multihead.json"]) + def pytest_train_model(mpnn_type, ci_input, overwrite_data=False): +> unittest_train_model(mpnn_type, None, None, ci_input, False, overwrite_data) + +tests/test_graphs.py:223: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_graphs.py:134: in unittest_train_model + hydragnn.run_training(config, use_deepspeed) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:89: in _ + model = create_model_config( +hydragnn/models/create.py:45: in create_model_config + return create_model( +hydragnn/models/create.py:455: in create_model + model = PNAEqStack( +hydragnn/models/PNAEqStack.py:72: in __init__ + super().__init__(input_args, conv_args, *args, **kwargs) +hydragnn/models/Base.py:174: in __init__ + self._init_conv() +hydragnn/models/PNAEqStack.py:80: in _init_conv + self.get_conv( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +self = PNAEqStack( + (graph_convs): ModuleList() + (feature_layers): ModuleList() + (heads_NN): ModuleList() + (convs_node_hi...eDict() + (convs_node_output): ModuleDict() + (batch_norms_node_output): ModuleDict() + (activation_function): ReLU() +) +input_dim = 0, output_dim = 8, last_layer = False, edge_dim = None + + def get_conv(self, input_dim, output_dim, last_layer=False, edge_dim=None): + hidden_dim = output_dim if input_dim == 1 else input_dim + assert ( +> hidden_dim > 1 + ), "PNAEq requires more than one hidden dimension between input_dim and output_dim." +E AssertionError: PNAEq requires more than one hidden dimension between input_dim and output_dim. + +hydragnn/models/PNAEqStack.py:106: AssertionError +----------------------------- Captured stderr call ----------------------------- + Degree max: 0%| | 0/354 [00:00 unittest_train_model(mpnn_type, None, None, ci_input, False, overwrite_data) + +tests/test_graphs.py:223: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_graphs.py:134: in unittest_train_model + hydragnn.run_training(config, use_deepspeed) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:89: in _ + model = create_model_config( +hydragnn/models/create.py:45: in create_model_config + return create_model( +hydragnn/models/create.py:428: in create_model + model = PAINNStack( +hydragnn/models/PAINNStack.py:47: in __init__ + super().__init__(input_args, conv_args, *args, **kwargs) +hydragnn/models/Base.py:174: in __init__ + self._init_conv() +hydragnn/models/PAINNStack.py:53: in _init_conv + self.get_conv( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +self = PAINNStack( + (graph_convs): ModuleList() + (feature_layers): ModuleList() + (heads_NN): ModuleList() + (convs_node_hi...eDict() + (convs_node_output): ModuleDict() + (batch_norms_node_output): ModuleDict() + (activation_function): ReLU() +) +input_dim = 0, output_dim = 8, last_layer = False, edge_dim = None + + def get_conv(self, input_dim, output_dim, last_layer=False, edge_dim=None): + hidden_dim = output_dim if input_dim == 1 else input_dim + assert ( +> hidden_dim > 1 + ), "PainnNet requires more than one hidden dimension between input_dim and output_dim." +E AssertionError: PainnNet requires more than one hidden dimension between input_dim and output_dim. + +hydragnn/models/PAINNStack.py:79: AssertionError +__________________ pytest_train_model[ci_multihead.json-MACE] __________________ + +mpnn_type = 'MACE', ci_input = 'ci_multihead.json', overwrite_data = False + + @pytest.mark.parametrize( + "mpnn_type", + [ + "SAGE", + "GIN", + "GAT", + "MFC", + "PNA", + "PNAPlus", + "CGCNN", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + "PAINN", + "MACE", + ], + ) + @pytest.mark.parametrize("ci_input", ["ci.json", "ci_multihead.json"]) + def pytest_train_model(mpnn_type, ci_input, overwrite_data=False): +> unittest_train_model(mpnn_type, None, None, ci_input, False, overwrite_data) + +tests/test_graphs.py:223: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_graphs.py:134: in unittest_train_model + hydragnn.run_training(config, use_deepspeed) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:176: in _ + train_validate_test( +hydragnn/train/train_validate_test.py:151: in train_validate_test + _, _, true_values, predicted_values = test(test_loader, model, verbosity) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/utils/_contextlib.py:120: in decorate_context + return func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +hydragnn/train/train_validate_test.py:728: in test + pred = model(data) + ^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/nn/modules/module.py:1773: in _wrapped_call_impl + return self._call_impl(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/nn/modules/module.py:1784: in _call_impl + return forward_call(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/nn/parallel/distributed.py:1648: in forward + else self._run_ddp_forward(*inputs, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/nn/parallel/distributed.py:1474: in _run_ddp_forward + return self.module(*inputs, **kwargs) # type: ignore[index] + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/nn/modules/module.py:1773: in _wrapped_call_impl + return self._call_impl(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/nn/modules/module.py:1784: in _call_impl + return forward_call(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/models/MACEStack.py:366: in forward + inv_node_feat, equiv_node_feat, conv_args = self._embedding(data) + ^^^^^^^^^^^^^^^^^^^^^ +hydragnn/models/MACEStack.py:428: in _embedding + data.node_attributes = process_node_attributes(data["x"], self.num_elements) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +node_attributes = tensor([], size=(76, 0)), num_elements = 118 + + def process_node_attributes(node_attributes, num_elements): + # Check that node attributes are atomic numbers and process accordingly + node_attributes = node_attributes.squeeze() # Squeeze all unnecessary dimensions + assert ( +> node_attributes.dim() == 1 + ), "MACE only supports raw atomic numbers as node_attributes. Your data.x \ + isn't a 1D tensor after squeezing, are you using vector features?" +E AssertionError: MACE only supports raw atomic numbers as node_attributes. Your data.x isn't a 1D tensor after squeezing, are you using vector features? + +hydragnn/models/MACEStack.py:487: AssertionError +----------------------------- Captured stdout call ----------------------------- +dist.is_initialized(),sync_batch_norm,device_name: True False cpu +Using FSDP: False Sharding: ShardingStrategy.FULL_SHARD +get_autocast_and_scaler use_bf16: False +----------------------------- Captured stderr call ----------------------------- + Calculate avg degree: 0%| | 0/354 [00:00 unittest_train_model( + mpnn_type, None, None, "ci_vectoroutput.json", True, overwrite_data + ) + +tests/test_graphs.py:284: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +mpnn_type = 'PNA', global_attn_engine = None, global_attn_type = None +ci_input = 'ci_vectoroutput.json', use_lengths = True, overwrite_data = False +use_deepspeed = False, overwrite_config = None + + def unittest_train_model( + mpnn_type, + global_attn_engine, + global_attn_type, + ci_input, + use_lengths, + overwrite_data=False, + use_deepspeed=False, + overwrite_config=None, + ): + world_size, rank = hydragnn.utils.distributed.get_comm_size_and_rank() + + os.environ["SERIALIZED_DATA_PATH"] = os.getcwd() + + # Read in config settings and override model type. + config_file = os.path.join(os.getcwd(), "tests/inputs", ci_input) + with open(config_file, "r") as f: + config = json.load(f) + config["NeuralNetwork"]["Architecture"]["global_attn_engine"] = global_attn_engine + config["NeuralNetwork"]["Architecture"]["global_attn_type"] = global_attn_type + config["NeuralNetwork"]["Architecture"]["mpnn_type"] = mpnn_type + + # Overwrite config settings if provided + if overwrite_config: + config = merge_config(config, overwrite_config) + + """ + to test this locally, set ci.json as + "Dataset": { + ... + "path": { + "train": "serialized_dataset/unit_test_singlehead_train.pkl", + "test": "serialized_dataset/unit_test_singlehead_test.pkl", + "validate": "serialized_dataset/unit_test_singlehead_validate.pkl"} + ... + """ + # use pkl files if exist by default + for dataset_name in config["Dataset"]["path"].keys(): + if dataset_name == "total": + pkl_file = ( + os.environ["SERIALIZED_DATA_PATH"] + + "/serialized_dataset/" + + config["Dataset"]["name"] + + ".pkl" + ) + else: + pkl_file = ( + os.environ["SERIALIZED_DATA_PATH"] + + "/serialized_dataset/" + + config["Dataset"]["name"] + + "_" + + dataset_name + + ".pkl" + ) + if os.path.exists(pkl_file): + config["Dataset"]["path"][dataset_name] = pkl_file + + # In the unit test runs, it is found MFC favors graph-level features over node-level features, compared with other models; + # hence here we decrease the loss weight coefficient for graph-level head in MFC. + if mpnn_type == "MFC" and ci_input == "ci_multihead.json": + config["NeuralNetwork"]["Architecture"]["task_weights"][0] = 2 + + # Only run with edge lengths for models that support them. + if use_lengths: + config["NeuralNetwork"]["Architecture"]["edge_features"] = ["lengths"] + + if rank == 0: + num_samples_tot = 500 + # check if serialized pickle files or folders for raw files provided + pkl_input = False + if list(config["Dataset"]["path"].values())[0].endswith(".pkl"): + pkl_input = True + # only generate new datasets, if not pkl + if not pkl_input: + for dataset_name, data_path in config["Dataset"]["path"].items(): + if overwrite_data: + shutil.rmtree(data_path) + if not os.path.exists(data_path): + os.makedirs(data_path) + if dataset_name == "total": + num_samples = num_samples_tot + elif dataset_name == "train": + num_samples = int( + num_samples_tot + * config["NeuralNetwork"]["Training"]["perc_train"] + ) + elif dataset_name == "test": + num_samples = int( + num_samples_tot + * (1 - config["NeuralNetwork"]["Training"]["perc_train"]) + * 0.5 + ) + elif dataset_name == "validate": + num_samples = int( + num_samples_tot + * (1 - config["NeuralNetwork"]["Training"]["perc_train"]) + * 0.5 + ) + if not os.listdir(data_path): + tests.deterministic_graph_data( + data_path, number_configurations=num_samples + ) + MPI.COMM_WORLD.Barrier() + + # Since the config file uses PNA already, test the file overload here. + # All the other models need to use the locally modified dictionary. + if mpnn_type == "PNA" and not use_lengths: + hydragnn.run_training(config_file, use_deepspeed) + else: + hydragnn.run_training(config, use_deepspeed) + + ( + error, + error_mse_task, + true_values, + predicted_values, + ) = hydragnn.run_prediction(config, use_deepspeed) + + # Set RMSE and sample MAE error thresholds + thresholds = { + "SAGE": [0.20, 0.20], + "PNA": [0.20, 0.20], + "PNAPlus": [0.20, 0.20], + "MFC": [0.20, 0.30], + "GIN": [0.25, 0.20], + "GAT": [0.60, 0.70], + "CGCNN": [0.50, 0.40], + "SchNet": [0.20, 0.20], + "DimeNet": [0.50, 0.50], + "EGNN": [0.20, 0.20], + "PNAEq": [0.60, 0.60], + "PAINN": [0.60, 0.60], + "MACE": [0.60, 0.70], + } + if use_lengths and ("vector" not in ci_input): + thresholds["CGCNN"] = [0.175, 0.175] + thresholds["PNA"] = [0.10, 0.10] + thresholds["PNAPlus"] = [0.10, 0.10] + if use_lengths and "vector" in ci_input: + thresholds["PNA"] = [0.2, 0.15] + thresholds["PNAPlus"] = [0.2, 0.15] + if ci_input == "ci_conv_head.json": + thresholds["GIN"] = [0.26, 0.51] + thresholds["SchNet"] = [0.30, 0.30] + + verbosity = 2 + + for ihead in range(len(true_values)): + error_head_mse = error_mse_task[ihead] + error_str = ( + str("{:.6f}".format(error_head_mse)) + " < " + str(thresholds[mpnn_type][0]) + ) + hydragnn.utils.print.print_distributed(verbosity, "head: " + error_str) + assert ( + error_head_mse < thresholds[mpnn_type][0] + ), "Head RMSE checking failed for " + str(ihead) + + head_true = true_values[ihead] + head_pred = predicted_values[ihead] + # Check individual samples + mae = torch.nn.L1Loss() + sample_mean_abs_error = mae(head_true, head_pred) + error_str = ( + "{:.6f}".format(sample_mean_abs_error) + + " < " + + str(thresholds[mpnn_type][1]) + ) +> assert ( + sample_mean_abs_error < thresholds[mpnn_type][1] + ), f"MAE sample checking failed! MAE: {sample_mean_abs_error:.6f} >= threshold: {thresholds[mpnn_type][1]} for model: {mpnn_type}" +E AssertionError: MAE sample checking failed! MAE: 0.152753 >= threshold: 0.15 for model: PNA +E assert tensor(0.1528) < 0.15 + +tests/test_graphs.py:192: AssertionError +----------------------------- Captured stdout call ----------------------------- +dist.is_initialized(),sync_batch_norm,device_name: True False cpu +Using FSDP: False Sharding: ShardingStrategy.FULL_SHARD +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +dist.is_initialized(),sync_batch_norm,device_name: True False cpu +Using FSDP: False Sharding: ShardingStrategy.FULL_SHARD +get_autocast_and_scaler use_bf16: False +----------------------------- Captured stderr call ----------------------------- + Degree max: 0%| | 0/354 [00:00 unittest_train_model( + mpnn_type, None, None, "ci_conv_head.json", False, overwrite_data + ) + +tests/test_graphs.py:306: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_graphs.py:134: in unittest_train_model + hydragnn.run_training(config, use_deepspeed) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:97: in _ + model = get_distributed_model( +hydragnn/utils/distributed/distributed.py:379: in get_distributed_model + model = DDP(model, **ddp_kwargs) + ^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/nn/parallel/distributed.py:802: in __init__ + self._log_and_throw( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +self = DistributedDataParallel( + (module): SAGEStack( + (graph_convs): ModuleList( + (0): Sequential( + (0) - SA...=True, track_running_stats=True) + ) + ) + (activation_function): ReLU() + (graph_shared): ModuleDict() + ) +) +err_type = +err_msg = "Modules with uninitialized parameters can't be used with `DistributedDataParallel`. Run a dummy forward pass to correctly initialize the modules" + + def _log_and_throw(self, err_type, err_msg): + if self.logger is not None: + self.logger.set_error_and_log(f"{str(err_type)}: {err_msg}") +> raise err_type(err_msg) +E RuntimeError: Modules with uninitialized parameters can't be used with `DistributedDataParallel`. Run a dummy forward pass to correctly initialize the modules + +.venv/lib/python3.13/site-packages/torch/nn/parallel/distributed.py:1143: RuntimeError +----------------------------- Captured stdout call ----------------------------- +dist.is_initialized(),sync_batch_norm,device_name: True False cpu +Using FSDP: False Sharding: ShardingStrategy.FULL_SHARD +______________________ pytest_train_model_conv_head[GAT] _______________________ + +mpnn_type = 'GAT', overwrite_data = False + + @pytest.mark.parametrize( + "mpnn_type", + [ + "SAGE", + "GIN", + "GAT", + "MFC", + "PNA", + "PNAPlus", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + "PAINN", + ], + ) + def pytest_train_model_conv_head(mpnn_type, overwrite_data=False): +> unittest_train_model( + mpnn_type, None, None, "ci_conv_head.json", False, overwrite_data + ) + +tests/test_graphs.py:306: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_graphs.py:134: in unittest_train_model + hydragnn.run_training(config, use_deepspeed) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:97: in _ + model = get_distributed_model( +hydragnn/utils/distributed/distributed.py:379: in get_distributed_model + model = DDP(model, **ddp_kwargs) + ^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/nn/parallel/distributed.py:802: in __init__ + self._log_and_throw( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +self = DistributedDataParallel( + (module): GATStack( + (graph_convs): ModuleList( + (0): Sequential( + (0) - GAT...=True) + ) + ) + (activation_function): ReLU() + (out_lin): Identity() + (graph_shared): ModuleDict() + ) +) +err_type = +err_msg = "Modules with uninitialized parameters can't be used with `DistributedDataParallel`. Run a dummy forward pass to correctly initialize the modules" + + def _log_and_throw(self, err_type, err_msg): + if self.logger is not None: + self.logger.set_error_and_log(f"{str(err_type)}: {err_msg}") +> raise err_type(err_msg) +E RuntimeError: Modules with uninitialized parameters can't be used with `DistributedDataParallel`. Run a dummy forward pass to correctly initialize the modules + +.venv/lib/python3.13/site-packages/torch/nn/parallel/distributed.py:1143: RuntimeError +----------------------------- Captured stdout call ----------------------------- +dist.is_initialized(),sync_batch_norm,device_name: True False cpu +Using FSDP: False Sharding: ShardingStrategy.FULL_SHARD +______________________ pytest_train_model_conv_head[MFC] _______________________ + +mpnn_type = 'MFC', overwrite_data = False + + @pytest.mark.parametrize( + "mpnn_type", + [ + "SAGE", + "GIN", + "GAT", + "MFC", + "PNA", + "PNAPlus", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + "PAINN", + ], + ) + def pytest_train_model_conv_head(mpnn_type, overwrite_data=False): +> unittest_train_model( + mpnn_type, None, None, "ci_conv_head.json", False, overwrite_data + ) + +tests/test_graphs.py:306: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_graphs.py:134: in unittest_train_model + hydragnn.run_training(config, use_deepspeed) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:97: in _ + model = get_distributed_model( +hydragnn/utils/distributed/distributed.py:379: in get_distributed_model + model = DDP(model, **ddp_kwargs) + ^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/nn/parallel/distributed.py:802: in __init__ + self._log_and_throw( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +self = DistributedDataParallel( + (module): MFCStack( + (graph_convs): ModuleList( + (0): Sequential( + (0) - MFC...=True, track_running_stats=True) + ) + ) + (activation_function): ReLU() + (graph_shared): ModuleDict() + ) +) +err_type = +err_msg = "Modules with uninitialized parameters can't be used with `DistributedDataParallel`. Run a dummy forward pass to correctly initialize the modules" + + def _log_and_throw(self, err_type, err_msg): + if self.logger is not None: + self.logger.set_error_and_log(f"{str(err_type)}: {err_msg}") +> raise err_type(err_msg) +E RuntimeError: Modules with uninitialized parameters can't be used with `DistributedDataParallel`. Run a dummy forward pass to correctly initialize the modules + +.venv/lib/python3.13/site-packages/torch/nn/parallel/distributed.py:1143: RuntimeError +----------------------------- Captured stdout call ----------------------------- +dist.is_initialized(),sync_batch_norm,device_name: True False cpu +Using FSDP: False Sharding: ShardingStrategy.FULL_SHARD +______________________ pytest_train_model_conv_head[PNA] _______________________ + +mpnn_type = 'PNA', overwrite_data = False + + @pytest.mark.parametrize( + "mpnn_type", + [ + "SAGE", + "GIN", + "GAT", + "MFC", + "PNA", + "PNAPlus", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + "PAINN", + ], + ) + def pytest_train_model_conv_head(mpnn_type, overwrite_data=False): +> unittest_train_model( + mpnn_type, None, None, "ci_conv_head.json", False, overwrite_data + ) + +tests/test_graphs.py:306: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_graphs.py:132: in unittest_train_model + hydragnn.run_training(config_file, use_deepspeed) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:62: in _ + run_training(config) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:97: in _ + model = get_distributed_model( +hydragnn/utils/distributed/distributed.py:379: in get_distributed_model + model = DDP(model, **ddp_kwargs) + ^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/nn/parallel/distributed.py:802: in __init__ + self._log_and_throw( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +self = DistributedDataParallel( + (module): PNAStack( + (graph_convs): ModuleList( + (0): Sequential( + (0) - PNA...=True, track_running_stats=True) + ) + ) + (activation_function): ReLU() + (graph_shared): ModuleDict() + ) +) +err_type = +err_msg = "Modules with uninitialized parameters can't be used with `DistributedDataParallel`. Run a dummy forward pass to correctly initialize the modules" + + def _log_and_throw(self, err_type, err_msg): + if self.logger is not None: + self.logger.set_error_and_log(f"{str(err_type)}: {err_msg}") +> raise err_type(err_msg) +E RuntimeError: Modules with uninitialized parameters can't be used with `DistributedDataParallel`. Run a dummy forward pass to correctly initialize the modules + +.venv/lib/python3.13/site-packages/torch/nn/parallel/distributed.py:1143: RuntimeError +----------------------------- Captured stdout call ----------------------------- +dist.is_initialized(),sync_batch_norm,device_name: True False cpu +Using FSDP: False Sharding: ShardingStrategy.FULL_SHARD +----------------------------- Captured stderr call ----------------------------- + Degree max: 0%| | 0/350 [00:00 unittest_train_model( + mpnn_type, None, None, "ci_conv_head.json", False, overwrite_data + ) + +tests/test_graphs.py:306: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_graphs.py:134: in unittest_train_model + hydragnn.run_training(config, use_deepspeed) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:97: in _ + model = get_distributed_model( +hydragnn/utils/distributed/distributed.py:379: in get_distributed_model + model = DDP(model, **ddp_kwargs) + ^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/nn/parallel/distributed.py:802: in __init__ + self._log_and_throw( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +self = DistributedDataParallel( + (module): PNAPlusStack( + (graph_convs): ModuleList( + (0): Sequential( + (0) -...unction): ReLU() + (graph_shared): ModuleDict() + (rbf): BesselBasisLayer( + (envelope): Envelope() + ) + ) +) +err_type = +err_msg = "Modules with uninitialized parameters can't be used with `DistributedDataParallel`. Run a dummy forward pass to correctly initialize the modules" + + def _log_and_throw(self, err_type, err_msg): + if self.logger is not None: + self.logger.set_error_and_log(f"{str(err_type)}: {err_msg}") +> raise err_type(err_msg) +E RuntimeError: Modules with uninitialized parameters can't be used with `DistributedDataParallel`. Run a dummy forward pass to correctly initialize the modules + +.venv/lib/python3.13/site-packages/torch/nn/parallel/distributed.py:1143: RuntimeError +----------------------------- Captured stdout call ----------------------------- +dist.is_initialized(),sync_batch_norm,device_name: True False cpu +Using FSDP: False Sharding: ShardingStrategy.FULL_SHARD +----------------------------- Captured stderr call ----------------------------- + Degree max: 0%| | 0/350 [00:00 unittest_train_model( + mpnn_type, None, None, "ci_conv_head.json", False, overwrite_data + ) + +tests/test_graphs.py:306: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +mpnn_type = 'SchNet', global_attn_engine = None, global_attn_type = None +ci_input = 'ci_conv_head.json', use_lengths = False, overwrite_data = False +use_deepspeed = False, overwrite_config = None + + def unittest_train_model( + mpnn_type, + global_attn_engine, + global_attn_type, + ci_input, + use_lengths, + overwrite_data=False, + use_deepspeed=False, + overwrite_config=None, + ): + world_size, rank = hydragnn.utils.distributed.get_comm_size_and_rank() + + os.environ["SERIALIZED_DATA_PATH"] = os.getcwd() + + # Read in config settings and override model type. + config_file = os.path.join(os.getcwd(), "tests/inputs", ci_input) + with open(config_file, "r") as f: + config = json.load(f) + config["NeuralNetwork"]["Architecture"]["global_attn_engine"] = global_attn_engine + config["NeuralNetwork"]["Architecture"]["global_attn_type"] = global_attn_type + config["NeuralNetwork"]["Architecture"]["mpnn_type"] = mpnn_type + + # Overwrite config settings if provided + if overwrite_config: + config = merge_config(config, overwrite_config) + + """ + to test this locally, set ci.json as + "Dataset": { + ... + "path": { + "train": "serialized_dataset/unit_test_singlehead_train.pkl", + "test": "serialized_dataset/unit_test_singlehead_test.pkl", + "validate": "serialized_dataset/unit_test_singlehead_validate.pkl"} + ... + """ + # use pkl files if exist by default + for dataset_name in config["Dataset"]["path"].keys(): + if dataset_name == "total": + pkl_file = ( + os.environ["SERIALIZED_DATA_PATH"] + + "/serialized_dataset/" + + config["Dataset"]["name"] + + ".pkl" + ) + else: + pkl_file = ( + os.environ["SERIALIZED_DATA_PATH"] + + "/serialized_dataset/" + + config["Dataset"]["name"] + + "_" + + dataset_name + + ".pkl" + ) + if os.path.exists(pkl_file): + config["Dataset"]["path"][dataset_name] = pkl_file + + # In the unit test runs, it is found MFC favors graph-level features over node-level features, compared with other models; + # hence here we decrease the loss weight coefficient for graph-level head in MFC. + if mpnn_type == "MFC" and ci_input == "ci_multihead.json": + config["NeuralNetwork"]["Architecture"]["task_weights"][0] = 2 + + # Only run with edge lengths for models that support them. + if use_lengths: + config["NeuralNetwork"]["Architecture"]["edge_features"] = ["lengths"] + + if rank == 0: + num_samples_tot = 500 + # check if serialized pickle files or folders for raw files provided + pkl_input = False + if list(config["Dataset"]["path"].values())[0].endswith(".pkl"): + pkl_input = True + # only generate new datasets, if not pkl + if not pkl_input: + for dataset_name, data_path in config["Dataset"]["path"].items(): + if overwrite_data: + shutil.rmtree(data_path) + if not os.path.exists(data_path): + os.makedirs(data_path) + if dataset_name == "total": + num_samples = num_samples_tot + elif dataset_name == "train": + num_samples = int( + num_samples_tot + * config["NeuralNetwork"]["Training"]["perc_train"] + ) + elif dataset_name == "test": + num_samples = int( + num_samples_tot + * (1 - config["NeuralNetwork"]["Training"]["perc_train"]) + * 0.5 + ) + elif dataset_name == "validate": + num_samples = int( + num_samples_tot + * (1 - config["NeuralNetwork"]["Training"]["perc_train"]) + * 0.5 + ) + if not os.listdir(data_path): + tests.deterministic_graph_data( + data_path, number_configurations=num_samples + ) + MPI.COMM_WORLD.Barrier() + + # Since the config file uses PNA already, test the file overload here. + # All the other models need to use the locally modified dictionary. + if mpnn_type == "PNA" and not use_lengths: + hydragnn.run_training(config_file, use_deepspeed) + else: + hydragnn.run_training(config, use_deepspeed) + + ( + error, + error_mse_task, + true_values, + predicted_values, + ) = hydragnn.run_prediction(config, use_deepspeed) + + # Set RMSE and sample MAE error thresholds + thresholds = { + "SAGE": [0.20, 0.20], + "PNA": [0.20, 0.20], + "PNAPlus": [0.20, 0.20], + "MFC": [0.20, 0.30], + "GIN": [0.25, 0.20], + "GAT": [0.60, 0.70], + "CGCNN": [0.50, 0.40], + "SchNet": [0.20, 0.20], + "DimeNet": [0.50, 0.50], + "EGNN": [0.20, 0.20], + "PNAEq": [0.60, 0.60], + "PAINN": [0.60, 0.60], + "MACE": [0.60, 0.70], + } + if use_lengths and ("vector" not in ci_input): + thresholds["CGCNN"] = [0.175, 0.175] + thresholds["PNA"] = [0.10, 0.10] + thresholds["PNAPlus"] = [0.10, 0.10] + if use_lengths and "vector" in ci_input: + thresholds["PNA"] = [0.2, 0.15] + thresholds["PNAPlus"] = [0.2, 0.15] + if ci_input == "ci_conv_head.json": + thresholds["GIN"] = [0.26, 0.51] + thresholds["SchNet"] = [0.30, 0.30] + + verbosity = 2 + + for ihead in range(len(true_values)): + error_head_mse = error_mse_task[ihead] + error_str = ( + str("{:.6f}".format(error_head_mse)) + " < " + str(thresholds[mpnn_type][0]) + ) + hydragnn.utils.print.print_distributed(verbosity, "head: " + error_str) +> assert ( + error_head_mse < thresholds[mpnn_type][0] + ), "Head RMSE checking failed for " + str(ihead) +E AssertionError: Head RMSE checking failed for 0 +E assert tensor(0.4173) < 0.3 + +tests/test_graphs.py:178: AssertionError +----------------------------- Captured stdout call ----------------------------- +dist.is_initialized(),sync_batch_norm,device_name: True False cpu +Using FSDP: False Sharding: ShardingStrategy.FULL_SHARD +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +dist.is_initialized(),sync_batch_norm,device_name: True False cpu +Using FSDP: False Sharding: ShardingStrategy.FULL_SHARD +get_autocast_and_scaler use_bf16: False +----------------------------- Captured stderr call ----------------------------- +0: Load existing model: ./logs/SchNet-r-2.0-ncl-2-hd-20-ne-100-lr-0.02-bs-32-data-unit_test-node_ft--task_weights-1.0-/SchNet-r-2.0-ncl-2-hd-20-ne-100-lr-0.02-bs-32-data-unit_test-node_ft--task_weights-1.0-.pk +0: head: 0.417273 < 0.3 +____________________ pytest_train_model_conv_head[DimeNet] _____________________ + +mpnn_type = 'DimeNet', overwrite_data = False + + @pytest.mark.parametrize( + "mpnn_type", + [ + "SAGE", + "GIN", + "GAT", + "MFC", + "PNA", + "PNAPlus", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + "PAINN", + ], + ) + def pytest_train_model_conv_head(mpnn_type, overwrite_data=False): +> unittest_train_model( + mpnn_type, None, None, "ci_conv_head.json", False, overwrite_data + ) + +tests/test_graphs.py:306: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_graphs.py:134: in unittest_train_model + hydragnn.run_training(config, use_deepspeed) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:89: in _ + model = create_model_config( +hydragnn/models/create.py:45: in create_model_config + return create_model( +hydragnn/models/create.py:369: in create_model + model = DIMEStack( +hydragnn/models/DIMEStack.py:68: in __init__ + super().__init__(input_args, conv_args, *args, **kwargs) +hydragnn/models/Base.py:174: in __init__ + self._init_conv() +hydragnn/models/DIMEStack.py:80: in _init_conv + self.get_conv( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +self = DIMEStack( + (graph_convs): ModuleList() + (feature_layers): ModuleList() + (heads_NN): ModuleList() + (convs_node_hid...eDict() + (convs_node_output): ModuleDict() + (batch_norms_node_output): ModuleDict() + (activation_function): ReLU() +) +input_dim = 0, output_dim = 20, edge_dim = None + + def get_conv(self, input_dim, output_dim, edge_dim=None): + hidden_dim = output_dim if input_dim == 1 else input_dim + assert ( +> hidden_dim > 1 + ), "DimeNet requires more than one hidden dimension between input_dim and output_dim." +E AssertionError: DimeNet requires more than one hidden dimension between input_dim and output_dim. + +hydragnn/models/DIMEStack.py:99: AssertionError +______________________ pytest_train_model_conv_head[EGNN] ______________________ + +mpnn_type = 'EGNN', overwrite_data = False + + @pytest.mark.parametrize( + "mpnn_type", + [ + "SAGE", + "GIN", + "GAT", + "MFC", + "PNA", + "PNAPlus", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + "PAINN", + ], + ) + def pytest_train_model_conv_head(mpnn_type, overwrite_data=False): +> unittest_train_model( + mpnn_type, None, None, "ci_conv_head.json", False, overwrite_data + ) + +tests/test_graphs.py:306: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +mpnn_type = 'EGNN', global_attn_engine = None, global_attn_type = None +ci_input = 'ci_conv_head.json', use_lengths = False, overwrite_data = False +use_deepspeed = False, overwrite_config = None + + def unittest_train_model( + mpnn_type, + global_attn_engine, + global_attn_type, + ci_input, + use_lengths, + overwrite_data=False, + use_deepspeed=False, + overwrite_config=None, + ): + world_size, rank = hydragnn.utils.distributed.get_comm_size_and_rank() + + os.environ["SERIALIZED_DATA_PATH"] = os.getcwd() + + # Read in config settings and override model type. + config_file = os.path.join(os.getcwd(), "tests/inputs", ci_input) + with open(config_file, "r") as f: + config = json.load(f) + config["NeuralNetwork"]["Architecture"]["global_attn_engine"] = global_attn_engine + config["NeuralNetwork"]["Architecture"]["global_attn_type"] = global_attn_type + config["NeuralNetwork"]["Architecture"]["mpnn_type"] = mpnn_type + + # Overwrite config settings if provided + if overwrite_config: + config = merge_config(config, overwrite_config) + + """ + to test this locally, set ci.json as + "Dataset": { + ... + "path": { + "train": "serialized_dataset/unit_test_singlehead_train.pkl", + "test": "serialized_dataset/unit_test_singlehead_test.pkl", + "validate": "serialized_dataset/unit_test_singlehead_validate.pkl"} + ... + """ + # use pkl files if exist by default + for dataset_name in config["Dataset"]["path"].keys(): + if dataset_name == "total": + pkl_file = ( + os.environ["SERIALIZED_DATA_PATH"] + + "/serialized_dataset/" + + config["Dataset"]["name"] + + ".pkl" + ) + else: + pkl_file = ( + os.environ["SERIALIZED_DATA_PATH"] + + "/serialized_dataset/" + + config["Dataset"]["name"] + + "_" + + dataset_name + + ".pkl" + ) + if os.path.exists(pkl_file): + config["Dataset"]["path"][dataset_name] = pkl_file + + # In the unit test runs, it is found MFC favors graph-level features over node-level features, compared with other models; + # hence here we decrease the loss weight coefficient for graph-level head in MFC. + if mpnn_type == "MFC" and ci_input == "ci_multihead.json": + config["NeuralNetwork"]["Architecture"]["task_weights"][0] = 2 + + # Only run with edge lengths for models that support them. + if use_lengths: + config["NeuralNetwork"]["Architecture"]["edge_features"] = ["lengths"] + + if rank == 0: + num_samples_tot = 500 + # check if serialized pickle files or folders for raw files provided + pkl_input = False + if list(config["Dataset"]["path"].values())[0].endswith(".pkl"): + pkl_input = True + # only generate new datasets, if not pkl + if not pkl_input: + for dataset_name, data_path in config["Dataset"]["path"].items(): + if overwrite_data: + shutil.rmtree(data_path) + if not os.path.exists(data_path): + os.makedirs(data_path) + if dataset_name == "total": + num_samples = num_samples_tot + elif dataset_name == "train": + num_samples = int( + num_samples_tot + * config["NeuralNetwork"]["Training"]["perc_train"] + ) + elif dataset_name == "test": + num_samples = int( + num_samples_tot + * (1 - config["NeuralNetwork"]["Training"]["perc_train"]) + * 0.5 + ) + elif dataset_name == "validate": + num_samples = int( + num_samples_tot + * (1 - config["NeuralNetwork"]["Training"]["perc_train"]) + * 0.5 + ) + if not os.listdir(data_path): + tests.deterministic_graph_data( + data_path, number_configurations=num_samples + ) + MPI.COMM_WORLD.Barrier() + + # Since the config file uses PNA already, test the file overload here. + # All the other models need to use the locally modified dictionary. + if mpnn_type == "PNA" and not use_lengths: + hydragnn.run_training(config_file, use_deepspeed) + else: + hydragnn.run_training(config, use_deepspeed) + + ( + error, + error_mse_task, + true_values, + predicted_values, + ) = hydragnn.run_prediction(config, use_deepspeed) + + # Set RMSE and sample MAE error thresholds + thresholds = { + "SAGE": [0.20, 0.20], + "PNA": [0.20, 0.20], + "PNAPlus": [0.20, 0.20], + "MFC": [0.20, 0.30], + "GIN": [0.25, 0.20], + "GAT": [0.60, 0.70], + "CGCNN": [0.50, 0.40], + "SchNet": [0.20, 0.20], + "DimeNet": [0.50, 0.50], + "EGNN": [0.20, 0.20], + "PNAEq": [0.60, 0.60], + "PAINN": [0.60, 0.60], + "MACE": [0.60, 0.70], + } + if use_lengths and ("vector" not in ci_input): + thresholds["CGCNN"] = [0.175, 0.175] + thresholds["PNA"] = [0.10, 0.10] + thresholds["PNAPlus"] = [0.10, 0.10] + if use_lengths and "vector" in ci_input: + thresholds["PNA"] = [0.2, 0.15] + thresholds["PNAPlus"] = [0.2, 0.15] + if ci_input == "ci_conv_head.json": + thresholds["GIN"] = [0.26, 0.51] + thresholds["SchNet"] = [0.30, 0.30] + + verbosity = 2 + + for ihead in range(len(true_values)): + error_head_mse = error_mse_task[ihead] + error_str = ( + str("{:.6f}".format(error_head_mse)) + " < " + str(thresholds[mpnn_type][0]) + ) + hydragnn.utils.print.print_distributed(verbosity, "head: " + error_str) +> assert ( + error_head_mse < thresholds[mpnn_type][0] + ), "Head RMSE checking failed for " + str(ihead) +E AssertionError: Head RMSE checking failed for 0 +E assert tensor(0.2124) < 0.2 + +tests/test_graphs.py:178: AssertionError +----------------------------- Captured stdout call ----------------------------- +dist.is_initialized(),sync_batch_norm,device_name: True False cpu +Using FSDP: False Sharding: ShardingStrategy.FULL_SHARD +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +dist.is_initialized(),sync_batch_norm,device_name: True False cpu +Using FSDP: False Sharding: ShardingStrategy.FULL_SHARD +get_autocast_and_scaler use_bf16: False +----------------------------- Captured stderr call ----------------------------- +0: Load existing model: ./logs/EGNN-r-2.0-ncl-2-hd-20-ne-100-lr-0.02-bs-32-data-unit_test-node_ft--task_weights-1.0-/EGNN-r-2.0-ncl-2-hd-20-ne-100-lr-0.02-bs-32-data-unit_test-node_ft--task_weights-1.0-.pk +0: head: 0.212449 < 0.2 +_____________________ pytest_train_model_conv_head[PNAEq] ______________________ + +mpnn_type = 'PNAEq', overwrite_data = False + + @pytest.mark.parametrize( + "mpnn_type", + [ + "SAGE", + "GIN", + "GAT", + "MFC", + "PNA", + "PNAPlus", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + "PAINN", + ], + ) + def pytest_train_model_conv_head(mpnn_type, overwrite_data=False): +> unittest_train_model( + mpnn_type, None, None, "ci_conv_head.json", False, overwrite_data + ) + +tests/test_graphs.py:306: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_graphs.py:134: in unittest_train_model + hydragnn.run_training(config, use_deepspeed) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:89: in _ + model = create_model_config( +hydragnn/models/create.py:45: in create_model_config + return create_model( +hydragnn/models/create.py:455: in create_model + model = PNAEqStack( +hydragnn/models/PNAEqStack.py:72: in __init__ + super().__init__(input_args, conv_args, *args, **kwargs) +hydragnn/models/Base.py:174: in __init__ + self._init_conv() +hydragnn/models/PNAEqStack.py:80: in _init_conv + self.get_conv( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +self = PNAEqStack( + (graph_convs): ModuleList() + (feature_layers): ModuleList() + (heads_NN): ModuleList() + (convs_node_hi...eDict() + (convs_node_output): ModuleDict() + (batch_norms_node_output): ModuleDict() + (activation_function): ReLU() +) +input_dim = 0, output_dim = 20, last_layer = False, edge_dim = None + + def get_conv(self, input_dim, output_dim, last_layer=False, edge_dim=None): + hidden_dim = output_dim if input_dim == 1 else input_dim + assert ( +> hidden_dim > 1 + ), "PNAEq requires more than one hidden dimension between input_dim and output_dim." +E AssertionError: PNAEq requires more than one hidden dimension between input_dim and output_dim. + +hydragnn/models/PNAEqStack.py:106: AssertionError +----------------------------- Captured stderr call ----------------------------- + Degree max: 0%| | 0/350 [00:00 unittest_train_model( + mpnn_type, None, None, "ci_conv_head.json", False, overwrite_data + ) + +tests/test_graphs.py:306: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_graphs.py:134: in unittest_train_model + hydragnn.run_training(config, use_deepspeed) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:89: in _ + model = create_model_config( +hydragnn/models/create.py:45: in create_model_config + return create_model( +hydragnn/models/create.py:428: in create_model + model = PAINNStack( +hydragnn/models/PAINNStack.py:47: in __init__ + super().__init__(input_args, conv_args, *args, **kwargs) +hydragnn/models/Base.py:174: in __init__ + self._init_conv() +hydragnn/models/PAINNStack.py:53: in _init_conv + self.get_conv( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +self = PAINNStack( + (graph_convs): ModuleList() + (feature_layers): ModuleList() + (heads_NN): ModuleList() + (convs_node_hi...eDict() + (convs_node_output): ModuleDict() + (batch_norms_node_output): ModuleDict() + (activation_function): ReLU() +) +input_dim = 0, output_dim = 20, last_layer = False, edge_dim = None + + def get_conv(self, input_dim, output_dim, last_layer=False, edge_dim=None): + hidden_dim = output_dim if input_dim == 1 else input_dim + assert ( +> hidden_dim > 1 + ), "PainnNet requires more than one hidden dimension between input_dim and output_dim." +E AssertionError: PainnNet requires more than one hidden dimension between input_dim and output_dim. + +hydragnn/models/PAINNStack.py:79: AssertionError +=========================== short test summary info ============================ +FAILED tests/test_graphs.py::pytest_train_model[ci_multihead.json-SAGE] - Run... +FAILED tests/test_graphs.py::pytest_train_model[ci_multihead.json-GIN] - Asse... +FAILED tests/test_graphs.py::pytest_train_model[ci_multihead.json-GAT] - Runt... +FAILED tests/test_graphs.py::pytest_train_model[ci_multihead.json-MFC] - Runt... +FAILED tests/test_graphs.py::pytest_train_model[ci_multihead.json-PNA] - Runt... +FAILED tests/test_graphs.py::pytest_train_model[ci_multihead.json-PNAPlus] - ... +FAILED tests/test_graphs.py::pytest_train_model[ci_multihead.json-CGCNN] - In... +FAILED tests/test_graphs.py::pytest_train_model[ci_multihead.json-SchNet] - A... +FAILED tests/test_graphs.py::pytest_train_model[ci_multihead.json-DimeNet] - ... +FAILED tests/test_graphs.py::pytest_train_model[ci_multihead.json-EGNN] - Ass... +FAILED tests/test_graphs.py::pytest_train_model[ci_multihead.json-PNAEq] - As... +FAILED tests/test_graphs.py::pytest_train_model[ci_multihead.json-PAINN] - As... +FAILED tests/test_graphs.py::pytest_train_model[ci_multihead.json-MACE] - Ass... +FAILED tests/test_graphs.py::pytest_train_model_vectoroutput[PNA] - Assertion... +FAILED tests/test_graphs.py::pytest_train_model_conv_head[SAGE] - RuntimeErro... +FAILED tests/test_graphs.py::pytest_train_model_conv_head[GAT] - RuntimeError... +FAILED tests/test_graphs.py::pytest_train_model_conv_head[MFC] - RuntimeError... +FAILED tests/test_graphs.py::pytest_train_model_conv_head[PNA] - RuntimeError... +FAILED tests/test_graphs.py::pytest_train_model_conv_head[PNAPlus] - RuntimeE... +FAILED tests/test_graphs.py::pytest_train_model_conv_head[SchNet] - Assertion... +FAILED tests/test_graphs.py::pytest_train_model_conv_head[DimeNet] - Assertio... +FAILED tests/test_graphs.py::pytest_train_model_conv_head[EGNN] - AssertionEr... +FAILED tests/test_graphs.py::pytest_train_model_conv_head[PNAEq] - AssertionE... +FAILED tests/test_graphs.py::pytest_train_model_conv_head[PAINN] - AssertionE... +================== 24 failed, 44 passed in 371.28s (0:06:11) =================== diff --git a/test_output_all_final.log b/test_output_all_final.log new file mode 100644 index 000000000..77a703530 --- /dev/null +++ b/test_output_all_final.log @@ -0,0 +1,24 @@ +============================= test session starts ============================== +platform darwin -- Python 3.13.7, pytest-8.4.2, pluggy-1.6.0 -- /Users/7ml/Documents/Codes/HydraGNN/.venv/bin/python +cachedir: .pytest_cache +rootdir: /Users/7ml/Documents/Codes/HydraGNN +configfile: pytest.ini +collecting ... collected 68 items + +tests/test_graphs.py::pytest_train_model[ci.json-SAGE] PASSED [ 1%] +tests/test_graphs.py::pytest_train_model[ci.json-GIN] PASSED [ 2%] +tests/test_graphs.py::pytest_train_model[ci.json-GAT] PASSED [ 4%] +tests/test_graphs.py::pytest_train_model[ci.json-MFC] PASSED [ 5%] +tests/test_graphs.py::pytest_train_model[ci.json-PNA] PASSED [ 7%] +tests/test_graphs.py::pytest_train_model[ci.json-PNAPlus] PASSED [ 8%] +tests/test_graphs.py::pytest_train_model[ci.json-CGCNN] PASSED [ 10%] +tests/test_graphs.py::pytest_train_model[ci.json-SchNet] PASSED [ 11%] +tests/test_graphs.py::pytest_train_model[ci.json-DimeNet] PASSED [ 13%] +tests/test_graphs.py::pytest_train_model[ci.json-EGNN] PASSED [ 14%] +tests/test_graphs.py::pytest_train_model[ci.json-PNAEq] PASSED [ 16%] +tests/test_graphs.py::pytest_train_model[ci.json-PAINN] PASSED [ 17%] +tests/test_graphs.py::pytest_train_model[ci.json-MACE] PASSED [ 19%] +tests/test_graphs.py::pytest_train_model[ci_multihead.json-SAGE] PASSED [ 20%] +tests/test_graphs.py::pytest_train_model[ci_multihead.json-GIN] PASSED [ 22%] +tests/test_graphs.py::pytest_train_model[ci_multihead.json-GAT] PASSED [ 23%] +tests/test_graphs.py::pytest_train_model[ci_multihead.json-MFC] \ No newline at end of file diff --git a/test_output_all_tests.log b/test_output_all_tests.log new file mode 100644 index 000000000..7d7b452b8 --- /dev/null +++ b/test_output_all_tests.log @@ -0,0 +1,12922 @@ +============================= test session starts ============================== +platform darwin -- Python 3.13.7, pytest-8.4.2, pluggy-1.6.0 -- /Users/7ml/Documents/Codes/HydraGNN/.venv/bin/python +cachedir: .pytest_cache +rootdir: /Users/7ml/Documents/Codes/HydraGNN +configfile: pytest.ini +collecting ... collected 172 items + +tests/test_atomicdescriptors.py::pytest_atomicdescriptors FAILED [ 0%] +tests/test_config.py::pytest_config[lsms/lsms.json] PASSED [ 1%] +tests/test_datasetclass_inheritance.py::pytest_train_model_vectoroutput[PNA] SKIPPED [ 1%] +tests/test_deepspeed.py::pytest_train_model_vectoroutput_w_deepspeed[PNA] SKIPPED [ 2%] +tests/test_deepspeed.py::pytest_train_model_vectoroutput_w_deepspeed_global_attention[PNA-multihead-GPS] SKIPPED [ 2%] +tests/test_enthalpy.py::pytest_formation_enthalpy PASSED [ 3%] +tests/test_examples.py::pytest_examples_energy[qm9-SAGE-multihead-GPS] FAILED [ 4%] +tests/test_examples.py::pytest_examples_energy[qm9-GIN-multihead-GPS] FAILED [ 4%] +tests/test_examples.py::pytest_examples_energy[qm9-GAT-multihead-GPS] FAILED [ 5%] +tests/test_examples.py::pytest_examples_energy[qm9-MFC-multihead-GPS] FAILED [ 5%] +tests/test_examples.py::pytest_examples_energy[qm9-PNA-multihead-GPS] FAILED [ 6%] +tests/test_examples.py::pytest_examples_energy[qm9-PNAPlus-multihead-GPS] FAILED [ 6%] +tests/test_examples.py::pytest_examples_energy[qm9-SchNet-multihead-GPS] FAILED [ 7%] +tests/test_examples.py::pytest_examples_energy[qm9-DimeNet-multihead-GPS] FAILED [ 8%] +tests/test_examples.py::pytest_examples_energy[qm9-EGNN-multihead-GPS] FAILED [ 8%] +tests/test_examples.py::pytest_examples_energy[qm9-PNAEq-multihead-GPS] FAILED [ 9%] +tests/test_examples.py::pytest_examples_energy[qm9-PAINN-multihead-GPS] FAILED [ 9%] +tests/test_examples.py::pytest_examples_energy[md17-SAGE-multihead-GPS] FAILED [ 10%] +tests/test_examples.py::pytest_examples_energy[md17-GIN-multihead-GPS] FAILED [ 11%] +tests/test_examples.py::pytest_examples_energy[md17-GAT-multihead-GPS] FAILED [ 11%] +tests/test_examples.py::pytest_examples_energy[md17-MFC-multihead-GPS] FAILED [ 12%] +tests/test_examples.py::pytest_examples_energy[md17-PNA-multihead-GPS] FAILED [ 12%] +tests/test_examples.py::pytest_examples_energy[md17-PNAPlus-multihead-GPS] FAILED [ 13%] +tests/test_examples.py::pytest_examples_energy[md17-SchNet-multihead-GPS] FAILED [ 13%] +tests/test_examples.py::pytest_examples_energy[md17-DimeNet-multihead-GPS] FAILED [ 14%] +tests/test_examples.py::pytest_examples_energy[md17-EGNN-multihead-GPS] FAILED [ 15%] +tests/test_examples.py::pytest_examples_energy[md17-PNAEq-multihead-GPS] FAILED [ 15%] +tests/test_examples.py::pytest_examples_energy[md17-PAINN-multihead-GPS] FAILED [ 16%] +tests/test_examples.py::pytest_examples_grad_forces[LennardJones-PNAPlus] FAILED [ 16%] +tests/test_examples.py::pytest_examples_grad_forces[LennardJones-SchNet] FAILED [ 17%] +tests/test_examples.py::pytest_examples_grad_forces[LennardJones-DimeNet] FAILED [ 18%] +tests/test_examples.py::pytest_examples_grad_forces[LennardJones-EGNN] FAILED [ 18%] +tests/test_examples.py::pytest_examples_grad_forces[LennardJones-PNAEq] FAILED [ 19%] +tests/test_examples.py::pytest_examples_grad_forces[LennardJones-PAINN] FAILED [ 19%] +tests/test_examples.py::pytest_examples_grad_forces[LennardJones-MACE] FAILED [ 20%] +tests/test_feature_config.py::test_parse_new_format PASSED [ 20%] +tests/test_feature_config.py::test_parse_legacy_format PASSED [ 21%] +tests/test_feature_config.py::test_validate_config_valid PASSED [ 22%] +tests/test_feature_config.py::test_validate_config_mismatched_lengths PASSED [ 22%] +tests/test_feature_config.py::test_validate_config_invalid_indices PASSED [ 23%] +tests/test_feature_config.py::test_validate_against_data PASSED [ 23%] +tests/test_feature_config.py::test_validate_against_data_mismatch PASSED [ 24%] +tests/test_feature_config.py::test_update_var_config PASSED [ 25%] +tests/test_feature_config.py::test_get_feature_schema_example PASSED [ 25%] +tests/test_feature_config.py::test_print_feature_summary PASSED [ 26%] +tests/test_feature_config.py::test_empty_config PASSED [ 26%] +tests/test_feature_config.py::test_missing_config_raises PASSED [ 27%] +tests/test_feature_config_validation.py::test_validate_node_feature_columns_simple PASSED [ 27%] +tests/test_feature_config_validation.py::test_validate_node_feature_columns_with_column_indices PASSED [ 28%] +tests/test_feature_config_validation.py::test_validate_node_feature_columns_edge_cases PASSED [ 29%] +tests/test_forces_equivariant.py::pytest_examples[SchNet-LennardJones] FAILED [ 29%] +tests/test_forces_equivariant.py::pytest_examples[EGNN-LennardJones] FAILED [ 30%] +tests/test_forces_equivariant.py::pytest_examples[DimeNet-LennardJones] FAILED [ 30%] +tests/test_forces_equivariant.py::pytest_examples[PAINN-LennardJones] FAILED [ 31%] +tests/test_forces_equivariant.py::pytest_examples[PNAPlus-LennardJones] FAILED [ 31%] +tests/test_forces_equivariant.py::pytest_examples[MACE-LennardJones] FAILED [ 32%] +tests/test_graphs.py::pytest_train_model[ci.json-SAGE] FAILED [ 33%] +tests/test_graphs.py::pytest_train_model[ci.json-GIN] FAILED [ 33%] +tests/test_graphs.py::pytest_train_model[ci.json-GAT] FAILED [ 34%] +tests/test_graphs.py::pytest_train_model[ci.json-MFC] FAILED [ 34%] +tests/test_graphs.py::pytest_train_model[ci.json-PNA] FAILED [ 35%] +tests/test_graphs.py::pytest_train_model[ci.json-PNAPlus] FAILED [ 36%] +tests/test_graphs.py::pytest_train_model[ci.json-CGCNN] FAILED [ 36%] +tests/test_graphs.py::pytest_train_model[ci.json-SchNet] FAILED [ 37%] +tests/test_graphs.py::pytest_train_model[ci.json-DimeNet] FAILED [ 37%] +tests/test_graphs.py::pytest_train_model[ci.json-EGNN] FAILED [ 38%] +tests/test_graphs.py::pytest_train_model[ci.json-PNAEq] FAILED [ 38%] +tests/test_graphs.py::pytest_train_model[ci.json-PAINN] FAILED [ 39%] +tests/test_graphs.py::pytest_train_model[ci.json-MACE] FAILED [ 40%] +tests/test_graphs.py::pytest_train_model[ci_multihead.json-SAGE] FAILED [ 40%] +tests/test_graphs.py::pytest_train_model[ci_multihead.json-GIN] FAILED [ 41%] +tests/test_graphs.py::pytest_train_model[ci_multihead.json-GAT] FAILED [ 41%] +tests/test_graphs.py::pytest_train_model[ci_multihead.json-MFC] FAILED [ 42%] +tests/test_graphs.py::pytest_train_model[ci_multihead.json-PNA] FAILED [ 43%] +tests/test_graphs.py::pytest_train_model[ci_multihead.json-PNAPlus] FAILED [ 43%] +tests/test_graphs.py::pytest_train_model[ci_multihead.json-CGCNN] FAILED [ 44%] +tests/test_graphs.py::pytest_train_model[ci_multihead.json-SchNet] FAILED [ 44%] +tests/test_graphs.py::pytest_train_model[ci_multihead.json-DimeNet] FAILED [ 45%] +tests/test_graphs.py::pytest_train_model[ci_multihead.json-EGNN] FAILED [ 45%] +tests/test_graphs.py::pytest_train_model[ci_multihead.json-PNAEq] FAILED [ 46%] +tests/test_graphs.py::pytest_train_model[ci_multihead.json-PAINN] FAILED [ 47%] +tests/test_graphs.py::pytest_train_model[ci_multihead.json-MACE] FAILED [ 47%] +tests/test_graphs.py::pytest_train_model_lengths[GAT] FAILED [ 48%] +tests/test_graphs.py::pytest_train_model_lengths[PNA] FAILED [ 48%] +tests/test_graphs.py::pytest_train_model_lengths[PNAPlus] FAILED [ 49%] +tests/test_graphs.py::pytest_train_model_lengths[CGCNN] FAILED [ 50%] +tests/test_graphs.py::pytest_train_model_lengths[SchNet] FAILED [ 50%] +tests/test_graphs.py::pytest_train_model_lengths[DimeNet] FAILED [ 51%] +tests/test_graphs.py::pytest_train_model_lengths[EGNN] FAILED [ 51%] +tests/test_graphs.py::pytest_train_model_lengths[PNAEq] FAILED [ 52%] +tests/test_graphs.py::pytest_train_model_lengths[PAINN] FAILED [ 52%] +tests/test_graphs.py::pytest_train_model_lengths_global_attention[GAT-multihead-GPS] FAILED [ 53%] +tests/test_graphs.py::pytest_train_model_lengths_global_attention[PNA-multihead-GPS] FAILED [ 54%] +tests/test_graphs.py::pytest_train_model_lengths_global_attention[PNAPlus-multihead-GPS] FAILED [ 54%] +tests/test_graphs.py::pytest_train_model_lengths_global_attention[CGCNN-multihead-GPS] FAILED [ 55%] +tests/test_graphs.py::pytest_train_model_lengths_global_attention[SchNet-multihead-GPS] FAILED [ 55%] +tests/test_graphs.py::pytest_train_model_lengths_global_attention[DimeNet-multihead-GPS] FAILED [ 56%] +tests/test_graphs.py::pytest_train_model_lengths_global_attention[EGNN-multihead-GPS] FAILED [ 56%] +tests/test_graphs.py::pytest_train_model_lengths_global_attention[PNAEq-multihead-GPS] FAILED [ 57%] +tests/test_graphs.py::pytest_train_model_lengths_global_attention[PAINN-multihead-GPS] FAILED [ 58%] +tests/test_graphs.py::pytest_train_mace_model_lengths[MACE] FAILED [ 58%] +tests/test_graphs.py::pytest_train_equivariant_model[EGNN] FAILED [ 59%] +tests/test_graphs.py::pytest_train_equivariant_model[SchNet] FAILED [ 59%] +tests/test_graphs.py::pytest_train_equivariant_model[PNAEq] FAILED [ 60%] +tests/test_graphs.py::pytest_train_equivariant_model[PAINN] FAILED [ 61%] +tests/test_graphs.py::pytest_train_equivariant_model[MACE] FAILED [ 61%] +tests/test_graphs.py::pytest_train_model_vectoroutput[GAT] FAILED [ 62%] +tests/test_graphs.py::pytest_train_model_vectoroutput[PNA] FAILED [ 62%] +tests/test_graphs.py::pytest_train_model_vectoroutput[PNAPlus] FAILED [ 63%] +tests/test_graphs.py::pytest_train_model_vectoroutput[SchNet] FAILED [ 63%] +tests/test_graphs.py::pytest_train_model_vectoroutput[DimeNet] FAILED [ 64%] +tests/test_graphs.py::pytest_train_model_vectoroutput[EGNN] FAILED [ 65%] +tests/test_graphs.py::pytest_train_model_vectoroutput[PNAEq] FAILED [ 65%] +tests/test_graphs.py::pytest_train_model_conv_head[SAGE] FAILED [ 66%] +tests/test_graphs.py::pytest_train_model_conv_head[GIN] FAILED [ 66%] +tests/test_graphs.py::pytest_train_model_conv_head[GAT] FAILED [ 67%] +tests/test_graphs.py::pytest_train_model_conv_head[MFC] FAILED [ 68%] +tests/test_graphs.py::pytest_train_model_conv_head[PNA] FAILED [ 68%] +tests/test_graphs.py::pytest_train_model_conv_head[PNAPlus] FAILED [ 69%] +tests/test_graphs.py::pytest_train_model_conv_head[SchNet] FAILED [ 69%] +tests/test_graphs.py::pytest_train_model_conv_head[DimeNet] FAILED [ 70%] +tests/test_graphs.py::pytest_train_model_conv_head[EGNN] FAILED [ 70%] +tests/test_graphs.py::pytest_train_model_conv_head[PNAEq] FAILED [ 71%] +tests/test_graphs.py::pytest_train_model_conv_head[PAINN] FAILED [ 72%] +tests/test_interatomic_potential.py::pytest_model_creation_with_enhancement PASSED [ 72%] +tests/test_interatomic_potential.py::pytest_forward_pass PASSED [ 73%] +tests/test_interatomic_potential.py::pytest_energy_force_consistency PASSED [ 73%] +tests/test_loss_and_activation_functions.py::pytest_loss_functions[mse] FAILED [ 74%] +tests/test_loss_and_activation_functions.py::pytest_loss_functions[mae] FAILED [ 75%] +tests/test_loss_and_activation_functions.py::pytest_loss_functions[rmse] FAILED [ 75%] +tests/test_loss_and_activation_functions.py::pytest_loss_functions[GaussianNLLLoss] FAILED [ 76%] +tests/test_loss_and_activation_functions.py::pytest_activation_functions_multihead[relu] FAILED [ 76%] +tests/test_loss_and_activation_functions.py::pytest_activation_functions_multihead[selu] FAILED [ 77%] +tests/test_loss_and_activation_functions.py::pytest_activation_functions_multihead[prelu] FAILED [ 77%] +tests/test_loss_and_activation_functions.py::pytest_activation_functions_multihead[elu] FAILED [ 78%] +tests/test_loss_and_activation_functions.py::pytest_activation_functions_multihead[lrelu_01] FAILED [ 79%] +tests/test_loss_and_activation_functions.py::pytest_activation_functions_multihead[lrelu_025] FAILED [ 79%] +tests/test_loss_and_activation_functions.py::pytest_activation_functions_multihead[lrelu_05] FAILED [ 80%] +tests/test_loss_and_activation_functions.py::pytest_activation_functions_vectoroutput[relu] FAILED [ 80%] +tests/test_loss_and_activation_functions.py::pytest_activation_functions_vectoroutput[selu] FAILED [ 81%] +tests/test_loss_and_activation_functions.py::pytest_activation_functions_vectoroutput[prelu] FAILED [ 81%] +tests/test_loss_and_activation_functions.py::pytest_activation_functions_vectoroutput[elu] FAILED [ 82%] +tests/test_loss_and_activation_functions.py::pytest_activation_functions_vectoroutput[lrelu_01] FAILED [ 83%] +tests/test_loss_and_activation_functions.py::pytest_activation_functions_vectoroutput[lrelu_025] FAILED [ 83%] +tests/test_loss_and_activation_functions.py::pytest_activation_functions_vectoroutput[lrelu_05] FAILED [ 84%] +tests/test_model_loadpred.py::pytest_model_loadpred FAILED [ 84%] +tests/test_optimizer.py::pytest_optimizers[False-SGD] FAILED [ 85%] +tests/test_optimizer.py::pytest_optimizers[False-Adam] FAILED [ 86%] +tests/test_optimizer.py::pytest_optimizers[False-Adadelta] FAILED [ 86%] +tests/test_optimizer.py::pytest_optimizers[False-Adagrad] FAILED [ 87%] +tests/test_optimizer.py::pytest_optimizers[False-Adamax] FAILED [ 87%] +tests/test_optimizer.py::pytest_optimizers[False-AdamW] FAILED [ 88%] +tests/test_optimizer.py::pytest_optimizers[False-RMSprop] FAILED [ 88%] +tests/test_optimizer.py::pytest_optimizers[True-SGD] FAILED [ 89%] +tests/test_optimizer.py::pytest_optimizers[True-Adam] FAILED [ 90%] +tests/test_optimizer.py::pytest_optimizers[True-Adadelta] FAILED [ 90%] +tests/test_optimizer.py::pytest_optimizers[True-Adagrad] FAILED [ 91%] +tests/test_optimizer.py::pytest_optimizers[True-Adamax] FAILED [ 91%] +tests/test_optimizer.py::pytest_optimizers[True-AdamW] FAILED [ 92%] +tests/test_optimizer.py::pytest_optimizers[True-RMSprop] FAILED [ 93%] +tests/test_periodic_boundary_conditions.py::pytest_periodic_h2 PASSED [ 93%] +tests/test_periodic_boundary_conditions.py::pytest_periodic_bcc_large PASSED [ 94%] +tests/test_radial_transforms.py::pytest_train_model_transforms[None-bessel-MACE] FAILED [ 94%] +tests/test_radial_transforms.py::pytest_train_model_transforms[None-gaussian-MACE] FAILED [ 95%] +tests/test_radial_transforms.py::pytest_train_model_transforms[None-chebyshev-MACE] FAILED [ 95%] +tests/test_radial_transforms.py::pytest_train_model_transforms[Agnesi-bessel-MACE] FAILED [ 96%] +tests/test_radial_transforms.py::pytest_train_model_transforms[Agnesi-gaussian-MACE] FAILED [ 97%] +tests/test_radial_transforms.py::pytest_train_model_transforms[Agnesi-chebyshev-MACE] FAILED [ 97%] +tests/test_radial_transforms.py::pytest_train_model_transforms[Soft-bessel-MACE] FAILED [ 98%] +tests/test_radial_transforms.py::pytest_train_model_transforms[Soft-gaussian-MACE] FAILED [ 98%] +tests/test_radial_transforms.py::pytest_train_model_transforms[Soft-chebyshev-MACE] FAILED [ 99%] +tests/test_rotational_invariance.py::pytest_rotational_invariance PASSED [100%] + +=================================== FAILURES =================================== +___________________________ pytest_atomicdescriptors ___________________________ + + @pytest.mark.mpi_skip() + def pytest_atomicdescriptors(): + file_path = os.path.join( + os.path.dirname(__file__), + "..", + "hydragnn/utils/descriptors_and_embeddings/atomicdescriptors.py", + ) + return_code = subprocess.call(["python", file_path]) + + # Check the file ran without error. +> assert return_code == 0 +E assert 1 == 0 + +tests/test_atomicdescriptors.py:28: AssertionError +----------------------------- Captured stderr call ----------------------------- +Traceback (most recent call last): + File "/Users/7ml/Documents/Codes/HydraGNN/tests/../hydragnn/utils/descriptors_and_embeddings/atomicdescriptors.py", line 231, in + atomicdescriptor = atomicdescriptors( + "./embedding.json", overwritten=True, element_types=["C", "H", "S"] + ) + File "/Users/7ml/Documents/Codes/HydraGNN/tests/../hydragnn/utils/descriptors_and_embeddings/atomicdescriptors.py", line 32, in __init__ + for ele in mendeleev.get_all_elements(): + ^^^^^^^^^ +NameError: name 'mendeleev' is not defined +________________ pytest_examples_energy[qm9-SAGE-multihead-GPS] ________________ + +example = 'qm9', mpnn_type = 'SAGE', global_attn_engine = 'GPS' +global_attn_type = 'multihead' + + @pytest.mark.parametrize( + "global_attn_engine", + ["GPS"], + ) + @pytest.mark.parametrize("global_attn_type", ["multihead"]) + @pytest.mark.parametrize( + "mpnn_type", + [ + "SAGE", + "GIN", + "GAT", + "MFC", + "PNA", + "PNAPlus", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + "PAINN", + ], + ) + @pytest.mark.parametrize("example", ["qm9", "md17"]) + @pytest.mark.mpi_skip() + def pytest_examples_energy(example, mpnn_type, global_attn_engine, global_attn_type): + path = os.path.join(os.path.dirname(__file__), "..", "examples", example) + file_path = os.path.join(path, example + ".py") + # Add the --mpnn_type argument for the subprocess call + return_code = subprocess.call( + [ + "python", + file_path, + "--mpnn_type", + mpnn_type, + "--global_attn_engine", + global_attn_engine, + "--global_attn_type", + global_attn_type, + ] + ) + + # Check the file ran without error. +> assert return_code == 0 +E assert 1 == 0 + +tests/test_examples.py:59: AssertionError +----------------------------- Captured stderr call ----------------------------- +Traceback (most recent call last): + File "/Users/7ml/Documents/Codes/HydraGNN/tests/../examples/qm9/qm9.py", line 16, in + import hydragnn +ModuleNotFoundError: No module named 'hydragnn' +________________ pytest_examples_energy[qm9-GIN-multihead-GPS] _________________ + +example = 'qm9', mpnn_type = 'GIN', global_attn_engine = 'GPS' +global_attn_type = 'multihead' + + @pytest.mark.parametrize( + "global_attn_engine", + ["GPS"], + ) + @pytest.mark.parametrize("global_attn_type", ["multihead"]) + @pytest.mark.parametrize( + "mpnn_type", + [ + "SAGE", + "GIN", + "GAT", + "MFC", + "PNA", + "PNAPlus", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + "PAINN", + ], + ) + @pytest.mark.parametrize("example", ["qm9", "md17"]) + @pytest.mark.mpi_skip() + def pytest_examples_energy(example, mpnn_type, global_attn_engine, global_attn_type): + path = os.path.join(os.path.dirname(__file__), "..", "examples", example) + file_path = os.path.join(path, example + ".py") + # Add the --mpnn_type argument for the subprocess call + return_code = subprocess.call( + [ + "python", + file_path, + "--mpnn_type", + mpnn_type, + "--global_attn_engine", + global_attn_engine, + "--global_attn_type", + global_attn_type, + ] + ) + + # Check the file ran without error. +> assert return_code == 0 +E assert 1 == 0 + +tests/test_examples.py:59: AssertionError +----------------------------- Captured stderr call ----------------------------- +Traceback (most recent call last): + File "/Users/7ml/Documents/Codes/HydraGNN/tests/../examples/qm9/qm9.py", line 16, in + import hydragnn +ModuleNotFoundError: No module named 'hydragnn' +________________ pytest_examples_energy[qm9-GAT-multihead-GPS] _________________ + +example = 'qm9', mpnn_type = 'GAT', global_attn_engine = 'GPS' +global_attn_type = 'multihead' + + @pytest.mark.parametrize( + "global_attn_engine", + ["GPS"], + ) + @pytest.mark.parametrize("global_attn_type", ["multihead"]) + @pytest.mark.parametrize( + "mpnn_type", + [ + "SAGE", + "GIN", + "GAT", + "MFC", + "PNA", + "PNAPlus", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + "PAINN", + ], + ) + @pytest.mark.parametrize("example", ["qm9", "md17"]) + @pytest.mark.mpi_skip() + def pytest_examples_energy(example, mpnn_type, global_attn_engine, global_attn_type): + path = os.path.join(os.path.dirname(__file__), "..", "examples", example) + file_path = os.path.join(path, example + ".py") + # Add the --mpnn_type argument for the subprocess call + return_code = subprocess.call( + [ + "python", + file_path, + "--mpnn_type", + mpnn_type, + "--global_attn_engine", + global_attn_engine, + "--global_attn_type", + global_attn_type, + ] + ) + + # Check the file ran without error. +> assert return_code == 0 +E assert 1 == 0 + +tests/test_examples.py:59: AssertionError +----------------------------- Captured stderr call ----------------------------- +Traceback (most recent call last): + File "/Users/7ml/Documents/Codes/HydraGNN/tests/../examples/qm9/qm9.py", line 16, in + import hydragnn +ModuleNotFoundError: No module named 'hydragnn' +________________ pytest_examples_energy[qm9-MFC-multihead-GPS] _________________ + +example = 'qm9', mpnn_type = 'MFC', global_attn_engine = 'GPS' +global_attn_type = 'multihead' + + @pytest.mark.parametrize( + "global_attn_engine", + ["GPS"], + ) + @pytest.mark.parametrize("global_attn_type", ["multihead"]) + @pytest.mark.parametrize( + "mpnn_type", + [ + "SAGE", + "GIN", + "GAT", + "MFC", + "PNA", + "PNAPlus", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + "PAINN", + ], + ) + @pytest.mark.parametrize("example", ["qm9", "md17"]) + @pytest.mark.mpi_skip() + def pytest_examples_energy(example, mpnn_type, global_attn_engine, global_attn_type): + path = os.path.join(os.path.dirname(__file__), "..", "examples", example) + file_path = os.path.join(path, example + ".py") + # Add the --mpnn_type argument for the subprocess call + return_code = subprocess.call( + [ + "python", + file_path, + "--mpnn_type", + mpnn_type, + "--global_attn_engine", + global_attn_engine, + "--global_attn_type", + global_attn_type, + ] + ) + + # Check the file ran without error. +> assert return_code == 0 +E assert 1 == 0 + +tests/test_examples.py:59: AssertionError +----------------------------- Captured stderr call ----------------------------- +Traceback (most recent call last): + File "/Users/7ml/Documents/Codes/HydraGNN/tests/../examples/qm9/qm9.py", line 16, in + import hydragnn +ModuleNotFoundError: No module named 'hydragnn' +________________ pytest_examples_energy[qm9-PNA-multihead-GPS] _________________ + +example = 'qm9', mpnn_type = 'PNA', global_attn_engine = 'GPS' +global_attn_type = 'multihead' + + @pytest.mark.parametrize( + "global_attn_engine", + ["GPS"], + ) + @pytest.mark.parametrize("global_attn_type", ["multihead"]) + @pytest.mark.parametrize( + "mpnn_type", + [ + "SAGE", + "GIN", + "GAT", + "MFC", + "PNA", + "PNAPlus", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + "PAINN", + ], + ) + @pytest.mark.parametrize("example", ["qm9", "md17"]) + @pytest.mark.mpi_skip() + def pytest_examples_energy(example, mpnn_type, global_attn_engine, global_attn_type): + path = os.path.join(os.path.dirname(__file__), "..", "examples", example) + file_path = os.path.join(path, example + ".py") + # Add the --mpnn_type argument for the subprocess call + return_code = subprocess.call( + [ + "python", + file_path, + "--mpnn_type", + mpnn_type, + "--global_attn_engine", + global_attn_engine, + "--global_attn_type", + global_attn_type, + ] + ) + + # Check the file ran without error. +> assert return_code == 0 +E assert 1 == 0 + +tests/test_examples.py:59: AssertionError +----------------------------- Captured stderr call ----------------------------- +Traceback (most recent call last): + File "/Users/7ml/Documents/Codes/HydraGNN/tests/../examples/qm9/qm9.py", line 16, in + import hydragnn +ModuleNotFoundError: No module named 'hydragnn' +______________ pytest_examples_energy[qm9-PNAPlus-multihead-GPS] _______________ + +example = 'qm9', mpnn_type = 'PNAPlus', global_attn_engine = 'GPS' +global_attn_type = 'multihead' + + @pytest.mark.parametrize( + "global_attn_engine", + ["GPS"], + ) + @pytest.mark.parametrize("global_attn_type", ["multihead"]) + @pytest.mark.parametrize( + "mpnn_type", + [ + "SAGE", + "GIN", + "GAT", + "MFC", + "PNA", + "PNAPlus", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + "PAINN", + ], + ) + @pytest.mark.parametrize("example", ["qm9", "md17"]) + @pytest.mark.mpi_skip() + def pytest_examples_energy(example, mpnn_type, global_attn_engine, global_attn_type): + path = os.path.join(os.path.dirname(__file__), "..", "examples", example) + file_path = os.path.join(path, example + ".py") + # Add the --mpnn_type argument for the subprocess call + return_code = subprocess.call( + [ + "python", + file_path, + "--mpnn_type", + mpnn_type, + "--global_attn_engine", + global_attn_engine, + "--global_attn_type", + global_attn_type, + ] + ) + + # Check the file ran without error. +> assert return_code == 0 +E assert 1 == 0 + +tests/test_examples.py:59: AssertionError +----------------------------- Captured stderr call ----------------------------- +Traceback (most recent call last): + File "/Users/7ml/Documents/Codes/HydraGNN/tests/../examples/qm9/qm9.py", line 16, in + import hydragnn +ModuleNotFoundError: No module named 'hydragnn' +_______________ pytest_examples_energy[qm9-SchNet-multihead-GPS] _______________ + +example = 'qm9', mpnn_type = 'SchNet', global_attn_engine = 'GPS' +global_attn_type = 'multihead' + + @pytest.mark.parametrize( + "global_attn_engine", + ["GPS"], + ) + @pytest.mark.parametrize("global_attn_type", ["multihead"]) + @pytest.mark.parametrize( + "mpnn_type", + [ + "SAGE", + "GIN", + "GAT", + "MFC", + "PNA", + "PNAPlus", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + "PAINN", + ], + ) + @pytest.mark.parametrize("example", ["qm9", "md17"]) + @pytest.mark.mpi_skip() + def pytest_examples_energy(example, mpnn_type, global_attn_engine, global_attn_type): + path = os.path.join(os.path.dirname(__file__), "..", "examples", example) + file_path = os.path.join(path, example + ".py") + # Add the --mpnn_type argument for the subprocess call + return_code = subprocess.call( + [ + "python", + file_path, + "--mpnn_type", + mpnn_type, + "--global_attn_engine", + global_attn_engine, + "--global_attn_type", + global_attn_type, + ] + ) + + # Check the file ran without error. +> assert return_code == 0 +E assert 1 == 0 + +tests/test_examples.py:59: AssertionError +----------------------------- Captured stderr call ----------------------------- +Traceback (most recent call last): + File "/Users/7ml/Documents/Codes/HydraGNN/tests/../examples/qm9/qm9.py", line 16, in + import hydragnn +ModuleNotFoundError: No module named 'hydragnn' +______________ pytest_examples_energy[qm9-DimeNet-multihead-GPS] _______________ + +example = 'qm9', mpnn_type = 'DimeNet', global_attn_engine = 'GPS' +global_attn_type = 'multihead' + + @pytest.mark.parametrize( + "global_attn_engine", + ["GPS"], + ) + @pytest.mark.parametrize("global_attn_type", ["multihead"]) + @pytest.mark.parametrize( + "mpnn_type", + [ + "SAGE", + "GIN", + "GAT", + "MFC", + "PNA", + "PNAPlus", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + "PAINN", + ], + ) + @pytest.mark.parametrize("example", ["qm9", "md17"]) + @pytest.mark.mpi_skip() + def pytest_examples_energy(example, mpnn_type, global_attn_engine, global_attn_type): + path = os.path.join(os.path.dirname(__file__), "..", "examples", example) + file_path = os.path.join(path, example + ".py") + # Add the --mpnn_type argument for the subprocess call + return_code = subprocess.call( + [ + "python", + file_path, + "--mpnn_type", + mpnn_type, + "--global_attn_engine", + global_attn_engine, + "--global_attn_type", + global_attn_type, + ] + ) + + # Check the file ran without error. +> assert return_code == 0 +E assert 1 == 0 + +tests/test_examples.py:59: AssertionError +----------------------------- Captured stderr call ----------------------------- +Traceback (most recent call last): + File "/Users/7ml/Documents/Codes/HydraGNN/tests/../examples/qm9/qm9.py", line 16, in + import hydragnn +ModuleNotFoundError: No module named 'hydragnn' +________________ pytest_examples_energy[qm9-EGNN-multihead-GPS] ________________ + +example = 'qm9', mpnn_type = 'EGNN', global_attn_engine = 'GPS' +global_attn_type = 'multihead' + + @pytest.mark.parametrize( + "global_attn_engine", + ["GPS"], + ) + @pytest.mark.parametrize("global_attn_type", ["multihead"]) + @pytest.mark.parametrize( + "mpnn_type", + [ + "SAGE", + "GIN", + "GAT", + "MFC", + "PNA", + "PNAPlus", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + "PAINN", + ], + ) + @pytest.mark.parametrize("example", ["qm9", "md17"]) + @pytest.mark.mpi_skip() + def pytest_examples_energy(example, mpnn_type, global_attn_engine, global_attn_type): + path = os.path.join(os.path.dirname(__file__), "..", "examples", example) + file_path = os.path.join(path, example + ".py") + # Add the --mpnn_type argument for the subprocess call + return_code = subprocess.call( + [ + "python", + file_path, + "--mpnn_type", + mpnn_type, + "--global_attn_engine", + global_attn_engine, + "--global_attn_type", + global_attn_type, + ] + ) + + # Check the file ran without error. +> assert return_code == 0 +E assert 1 == 0 + +tests/test_examples.py:59: AssertionError +----------------------------- Captured stderr call ----------------------------- +Traceback (most recent call last): + File "/Users/7ml/Documents/Codes/HydraGNN/tests/../examples/qm9/qm9.py", line 16, in + import hydragnn +ModuleNotFoundError: No module named 'hydragnn' +_______________ pytest_examples_energy[qm9-PNAEq-multihead-GPS] ________________ + +example = 'qm9', mpnn_type = 'PNAEq', global_attn_engine = 'GPS' +global_attn_type = 'multihead' + + @pytest.mark.parametrize( + "global_attn_engine", + ["GPS"], + ) + @pytest.mark.parametrize("global_attn_type", ["multihead"]) + @pytest.mark.parametrize( + "mpnn_type", + [ + "SAGE", + "GIN", + "GAT", + "MFC", + "PNA", + "PNAPlus", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + "PAINN", + ], + ) + @pytest.mark.parametrize("example", ["qm9", "md17"]) + @pytest.mark.mpi_skip() + def pytest_examples_energy(example, mpnn_type, global_attn_engine, global_attn_type): + path = os.path.join(os.path.dirname(__file__), "..", "examples", example) + file_path = os.path.join(path, example + ".py") + # Add the --mpnn_type argument for the subprocess call + return_code = subprocess.call( + [ + "python", + file_path, + "--mpnn_type", + mpnn_type, + "--global_attn_engine", + global_attn_engine, + "--global_attn_type", + global_attn_type, + ] + ) + + # Check the file ran without error. +> assert return_code == 0 +E assert 1 == 0 + +tests/test_examples.py:59: AssertionError +----------------------------- Captured stderr call ----------------------------- +Traceback (most recent call last): + File "/Users/7ml/Documents/Codes/HydraGNN/tests/../examples/qm9/qm9.py", line 16, in + import hydragnn +ModuleNotFoundError: No module named 'hydragnn' +_______________ pytest_examples_energy[qm9-PAINN-multihead-GPS] ________________ + +example = 'qm9', mpnn_type = 'PAINN', global_attn_engine = 'GPS' +global_attn_type = 'multihead' + + @pytest.mark.parametrize( + "global_attn_engine", + ["GPS"], + ) + @pytest.mark.parametrize("global_attn_type", ["multihead"]) + @pytest.mark.parametrize( + "mpnn_type", + [ + "SAGE", + "GIN", + "GAT", + "MFC", + "PNA", + "PNAPlus", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + "PAINN", + ], + ) + @pytest.mark.parametrize("example", ["qm9", "md17"]) + @pytest.mark.mpi_skip() + def pytest_examples_energy(example, mpnn_type, global_attn_engine, global_attn_type): + path = os.path.join(os.path.dirname(__file__), "..", "examples", example) + file_path = os.path.join(path, example + ".py") + # Add the --mpnn_type argument for the subprocess call + return_code = subprocess.call( + [ + "python", + file_path, + "--mpnn_type", + mpnn_type, + "--global_attn_engine", + global_attn_engine, + "--global_attn_type", + global_attn_type, + ] + ) + + # Check the file ran without error. +> assert return_code == 0 +E assert 1 == 0 + +tests/test_examples.py:59: AssertionError +----------------------------- Captured stderr call ----------------------------- +Traceback (most recent call last): + File "/Users/7ml/Documents/Codes/HydraGNN/tests/../examples/qm9/qm9.py", line 16, in + import hydragnn +ModuleNotFoundError: No module named 'hydragnn' +_______________ pytest_examples_energy[md17-SAGE-multihead-GPS] ________________ + +example = 'md17', mpnn_type = 'SAGE', global_attn_engine = 'GPS' +global_attn_type = 'multihead' + + @pytest.mark.parametrize( + "global_attn_engine", + ["GPS"], + ) + @pytest.mark.parametrize("global_attn_type", ["multihead"]) + @pytest.mark.parametrize( + "mpnn_type", + [ + "SAGE", + "GIN", + "GAT", + "MFC", + "PNA", + "PNAPlus", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + "PAINN", + ], + ) + @pytest.mark.parametrize("example", ["qm9", "md17"]) + @pytest.mark.mpi_skip() + def pytest_examples_energy(example, mpnn_type, global_attn_engine, global_attn_type): + path = os.path.join(os.path.dirname(__file__), "..", "examples", example) + file_path = os.path.join(path, example + ".py") + # Add the --mpnn_type argument for the subprocess call + return_code = subprocess.call( + [ + "python", + file_path, + "--mpnn_type", + mpnn_type, + "--global_attn_engine", + global_attn_engine, + "--global_attn_type", + global_attn_type, + ] + ) + + # Check the file ran without error. +> assert return_code == 0 +E assert 1 == 0 + +tests/test_examples.py:59: AssertionError +----------------------------- Captured stderr call ----------------------------- +Traceback (most recent call last): + File "/Users/7ml/Documents/Codes/HydraGNN/tests/../examples/md17/md17.py", line 17, in + import hydragnn +ModuleNotFoundError: No module named 'hydragnn' +________________ pytest_examples_energy[md17-GIN-multihead-GPS] ________________ + +example = 'md17', mpnn_type = 'GIN', global_attn_engine = 'GPS' +global_attn_type = 'multihead' + + @pytest.mark.parametrize( + "global_attn_engine", + ["GPS"], + ) + @pytest.mark.parametrize("global_attn_type", ["multihead"]) + @pytest.mark.parametrize( + "mpnn_type", + [ + "SAGE", + "GIN", + "GAT", + "MFC", + "PNA", + "PNAPlus", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + "PAINN", + ], + ) + @pytest.mark.parametrize("example", ["qm9", "md17"]) + @pytest.mark.mpi_skip() + def pytest_examples_energy(example, mpnn_type, global_attn_engine, global_attn_type): + path = os.path.join(os.path.dirname(__file__), "..", "examples", example) + file_path = os.path.join(path, example + ".py") + # Add the --mpnn_type argument for the subprocess call + return_code = subprocess.call( + [ + "python", + file_path, + "--mpnn_type", + mpnn_type, + "--global_attn_engine", + global_attn_engine, + "--global_attn_type", + global_attn_type, + ] + ) + + # Check the file ran without error. +> assert return_code == 0 +E assert 1 == 0 + +tests/test_examples.py:59: AssertionError +----------------------------- Captured stderr call ----------------------------- +Traceback (most recent call last): + File "/Users/7ml/Documents/Codes/HydraGNN/tests/../examples/md17/md17.py", line 17, in + import hydragnn +ModuleNotFoundError: No module named 'hydragnn' +________________ pytest_examples_energy[md17-GAT-multihead-GPS] ________________ + +example = 'md17', mpnn_type = 'GAT', global_attn_engine = 'GPS' +global_attn_type = 'multihead' + + @pytest.mark.parametrize( + "global_attn_engine", + ["GPS"], + ) + @pytest.mark.parametrize("global_attn_type", ["multihead"]) + @pytest.mark.parametrize( + "mpnn_type", + [ + "SAGE", + "GIN", + "GAT", + "MFC", + "PNA", + "PNAPlus", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + "PAINN", + ], + ) + @pytest.mark.parametrize("example", ["qm9", "md17"]) + @pytest.mark.mpi_skip() + def pytest_examples_energy(example, mpnn_type, global_attn_engine, global_attn_type): + path = os.path.join(os.path.dirname(__file__), "..", "examples", example) + file_path = os.path.join(path, example + ".py") + # Add the --mpnn_type argument for the subprocess call + return_code = subprocess.call( + [ + "python", + file_path, + "--mpnn_type", + mpnn_type, + "--global_attn_engine", + global_attn_engine, + "--global_attn_type", + global_attn_type, + ] + ) + + # Check the file ran without error. +> assert return_code == 0 +E assert 1 == 0 + +tests/test_examples.py:59: AssertionError +----------------------------- Captured stderr call ----------------------------- +Traceback (most recent call last): + File "/Users/7ml/Documents/Codes/HydraGNN/tests/../examples/md17/md17.py", line 17, in + import hydragnn +ModuleNotFoundError: No module named 'hydragnn' +________________ pytest_examples_energy[md17-MFC-multihead-GPS] ________________ + +example = 'md17', mpnn_type = 'MFC', global_attn_engine = 'GPS' +global_attn_type = 'multihead' + + @pytest.mark.parametrize( + "global_attn_engine", + ["GPS"], + ) + @pytest.mark.parametrize("global_attn_type", ["multihead"]) + @pytest.mark.parametrize( + "mpnn_type", + [ + "SAGE", + "GIN", + "GAT", + "MFC", + "PNA", + "PNAPlus", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + "PAINN", + ], + ) + @pytest.mark.parametrize("example", ["qm9", "md17"]) + @pytest.mark.mpi_skip() + def pytest_examples_energy(example, mpnn_type, global_attn_engine, global_attn_type): + path = os.path.join(os.path.dirname(__file__), "..", "examples", example) + file_path = os.path.join(path, example + ".py") + # Add the --mpnn_type argument for the subprocess call + return_code = subprocess.call( + [ + "python", + file_path, + "--mpnn_type", + mpnn_type, + "--global_attn_engine", + global_attn_engine, + "--global_attn_type", + global_attn_type, + ] + ) + + # Check the file ran without error. +> assert return_code == 0 +E assert 1 == 0 + +tests/test_examples.py:59: AssertionError +----------------------------- Captured stderr call ----------------------------- +Traceback (most recent call last): + File "/Users/7ml/Documents/Codes/HydraGNN/tests/../examples/md17/md17.py", line 17, in + import hydragnn +ModuleNotFoundError: No module named 'hydragnn' +________________ pytest_examples_energy[md17-PNA-multihead-GPS] ________________ + +example = 'md17', mpnn_type = 'PNA', global_attn_engine = 'GPS' +global_attn_type = 'multihead' + + @pytest.mark.parametrize( + "global_attn_engine", + ["GPS"], + ) + @pytest.mark.parametrize("global_attn_type", ["multihead"]) + @pytest.mark.parametrize( + "mpnn_type", + [ + "SAGE", + "GIN", + "GAT", + "MFC", + "PNA", + "PNAPlus", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + "PAINN", + ], + ) + @pytest.mark.parametrize("example", ["qm9", "md17"]) + @pytest.mark.mpi_skip() + def pytest_examples_energy(example, mpnn_type, global_attn_engine, global_attn_type): + path = os.path.join(os.path.dirname(__file__), "..", "examples", example) + file_path = os.path.join(path, example + ".py") + # Add the --mpnn_type argument for the subprocess call + return_code = subprocess.call( + [ + "python", + file_path, + "--mpnn_type", + mpnn_type, + "--global_attn_engine", + global_attn_engine, + "--global_attn_type", + global_attn_type, + ] + ) + + # Check the file ran without error. +> assert return_code == 0 +E assert 1 == 0 + +tests/test_examples.py:59: AssertionError +----------------------------- Captured stderr call ----------------------------- +Traceback (most recent call last): + File "/Users/7ml/Documents/Codes/HydraGNN/tests/../examples/md17/md17.py", line 17, in + import hydragnn +ModuleNotFoundError: No module named 'hydragnn' +______________ pytest_examples_energy[md17-PNAPlus-multihead-GPS] ______________ + +example = 'md17', mpnn_type = 'PNAPlus', global_attn_engine = 'GPS' +global_attn_type = 'multihead' + + @pytest.mark.parametrize( + "global_attn_engine", + ["GPS"], + ) + @pytest.mark.parametrize("global_attn_type", ["multihead"]) + @pytest.mark.parametrize( + "mpnn_type", + [ + "SAGE", + "GIN", + "GAT", + "MFC", + "PNA", + "PNAPlus", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + "PAINN", + ], + ) + @pytest.mark.parametrize("example", ["qm9", "md17"]) + @pytest.mark.mpi_skip() + def pytest_examples_energy(example, mpnn_type, global_attn_engine, global_attn_type): + path = os.path.join(os.path.dirname(__file__), "..", "examples", example) + file_path = os.path.join(path, example + ".py") + # Add the --mpnn_type argument for the subprocess call + return_code = subprocess.call( + [ + "python", + file_path, + "--mpnn_type", + mpnn_type, + "--global_attn_engine", + global_attn_engine, + "--global_attn_type", + global_attn_type, + ] + ) + + # Check the file ran without error. +> assert return_code == 0 +E assert 1 == 0 + +tests/test_examples.py:59: AssertionError +----------------------------- Captured stderr call ----------------------------- +Traceback (most recent call last): + File "/Users/7ml/Documents/Codes/HydraGNN/tests/../examples/md17/md17.py", line 17, in + import hydragnn +ModuleNotFoundError: No module named 'hydragnn' +______________ pytest_examples_energy[md17-SchNet-multihead-GPS] _______________ + +example = 'md17', mpnn_type = 'SchNet', global_attn_engine = 'GPS' +global_attn_type = 'multihead' + + @pytest.mark.parametrize( + "global_attn_engine", + ["GPS"], + ) + @pytest.mark.parametrize("global_attn_type", ["multihead"]) + @pytest.mark.parametrize( + "mpnn_type", + [ + "SAGE", + "GIN", + "GAT", + "MFC", + "PNA", + "PNAPlus", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + "PAINN", + ], + ) + @pytest.mark.parametrize("example", ["qm9", "md17"]) + @pytest.mark.mpi_skip() + def pytest_examples_energy(example, mpnn_type, global_attn_engine, global_attn_type): + path = os.path.join(os.path.dirname(__file__), "..", "examples", example) + file_path = os.path.join(path, example + ".py") + # Add the --mpnn_type argument for the subprocess call + return_code = subprocess.call( + [ + "python", + file_path, + "--mpnn_type", + mpnn_type, + "--global_attn_engine", + global_attn_engine, + "--global_attn_type", + global_attn_type, + ] + ) + + # Check the file ran without error. +> assert return_code == 0 +E assert 1 == 0 + +tests/test_examples.py:59: AssertionError +----------------------------- Captured stderr call ----------------------------- +Traceback (most recent call last): + File "/Users/7ml/Documents/Codes/HydraGNN/tests/../examples/md17/md17.py", line 17, in + import hydragnn +ModuleNotFoundError: No module named 'hydragnn' +______________ pytest_examples_energy[md17-DimeNet-multihead-GPS] ______________ + +example = 'md17', mpnn_type = 'DimeNet', global_attn_engine = 'GPS' +global_attn_type = 'multihead' + + @pytest.mark.parametrize( + "global_attn_engine", + ["GPS"], + ) + @pytest.mark.parametrize("global_attn_type", ["multihead"]) + @pytest.mark.parametrize( + "mpnn_type", + [ + "SAGE", + "GIN", + "GAT", + "MFC", + "PNA", + "PNAPlus", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + "PAINN", + ], + ) + @pytest.mark.parametrize("example", ["qm9", "md17"]) + @pytest.mark.mpi_skip() + def pytest_examples_energy(example, mpnn_type, global_attn_engine, global_attn_type): + path = os.path.join(os.path.dirname(__file__), "..", "examples", example) + file_path = os.path.join(path, example + ".py") + # Add the --mpnn_type argument for the subprocess call + return_code = subprocess.call( + [ + "python", + file_path, + "--mpnn_type", + mpnn_type, + "--global_attn_engine", + global_attn_engine, + "--global_attn_type", + global_attn_type, + ] + ) + + # Check the file ran without error. +> assert return_code == 0 +E assert 1 == 0 + +tests/test_examples.py:59: AssertionError +----------------------------- Captured stderr call ----------------------------- +Traceback (most recent call last): + File "/Users/7ml/Documents/Codes/HydraGNN/tests/../examples/md17/md17.py", line 17, in + import hydragnn +ModuleNotFoundError: No module named 'hydragnn' +_______________ pytest_examples_energy[md17-EGNN-multihead-GPS] ________________ + +example = 'md17', mpnn_type = 'EGNN', global_attn_engine = 'GPS' +global_attn_type = 'multihead' + + @pytest.mark.parametrize( + "global_attn_engine", + ["GPS"], + ) + @pytest.mark.parametrize("global_attn_type", ["multihead"]) + @pytest.mark.parametrize( + "mpnn_type", + [ + "SAGE", + "GIN", + "GAT", + "MFC", + "PNA", + "PNAPlus", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + "PAINN", + ], + ) + @pytest.mark.parametrize("example", ["qm9", "md17"]) + @pytest.mark.mpi_skip() + def pytest_examples_energy(example, mpnn_type, global_attn_engine, global_attn_type): + path = os.path.join(os.path.dirname(__file__), "..", "examples", example) + file_path = os.path.join(path, example + ".py") + # Add the --mpnn_type argument for the subprocess call + return_code = subprocess.call( + [ + "python", + file_path, + "--mpnn_type", + mpnn_type, + "--global_attn_engine", + global_attn_engine, + "--global_attn_type", + global_attn_type, + ] + ) + + # Check the file ran without error. +> assert return_code == 0 +E assert 1 == 0 + +tests/test_examples.py:59: AssertionError +----------------------------- Captured stderr call ----------------------------- +Traceback (most recent call last): + File "/Users/7ml/Documents/Codes/HydraGNN/tests/../examples/md17/md17.py", line 17, in + import hydragnn +ModuleNotFoundError: No module named 'hydragnn' +_______________ pytest_examples_energy[md17-PNAEq-multihead-GPS] _______________ + +example = 'md17', mpnn_type = 'PNAEq', global_attn_engine = 'GPS' +global_attn_type = 'multihead' + + @pytest.mark.parametrize( + "global_attn_engine", + ["GPS"], + ) + @pytest.mark.parametrize("global_attn_type", ["multihead"]) + @pytest.mark.parametrize( + "mpnn_type", + [ + "SAGE", + "GIN", + "GAT", + "MFC", + "PNA", + "PNAPlus", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + "PAINN", + ], + ) + @pytest.mark.parametrize("example", ["qm9", "md17"]) + @pytest.mark.mpi_skip() + def pytest_examples_energy(example, mpnn_type, global_attn_engine, global_attn_type): + path = os.path.join(os.path.dirname(__file__), "..", "examples", example) + file_path = os.path.join(path, example + ".py") + # Add the --mpnn_type argument for the subprocess call + return_code = subprocess.call( + [ + "python", + file_path, + "--mpnn_type", + mpnn_type, + "--global_attn_engine", + global_attn_engine, + "--global_attn_type", + global_attn_type, + ] + ) + + # Check the file ran without error. +> assert return_code == 0 +E assert 1 == 0 + +tests/test_examples.py:59: AssertionError +----------------------------- Captured stderr call ----------------------------- +Traceback (most recent call last): + File "/Users/7ml/Documents/Codes/HydraGNN/tests/../examples/md17/md17.py", line 17, in + import hydragnn +ModuleNotFoundError: No module named 'hydragnn' +_______________ pytest_examples_energy[md17-PAINN-multihead-GPS] _______________ + +example = 'md17', mpnn_type = 'PAINN', global_attn_engine = 'GPS' +global_attn_type = 'multihead' + + @pytest.mark.parametrize( + "global_attn_engine", + ["GPS"], + ) + @pytest.mark.parametrize("global_attn_type", ["multihead"]) + @pytest.mark.parametrize( + "mpnn_type", + [ + "SAGE", + "GIN", + "GAT", + "MFC", + "PNA", + "PNAPlus", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + "PAINN", + ], + ) + @pytest.mark.parametrize("example", ["qm9", "md17"]) + @pytest.mark.mpi_skip() + def pytest_examples_energy(example, mpnn_type, global_attn_engine, global_attn_type): + path = os.path.join(os.path.dirname(__file__), "..", "examples", example) + file_path = os.path.join(path, example + ".py") + # Add the --mpnn_type argument for the subprocess call + return_code = subprocess.call( + [ + "python", + file_path, + "--mpnn_type", + mpnn_type, + "--global_attn_engine", + global_attn_engine, + "--global_attn_type", + global_attn_type, + ] + ) + + # Check the file ran without error. +> assert return_code == 0 +E assert 1 == 0 + +tests/test_examples.py:59: AssertionError +----------------------------- Captured stderr call ----------------------------- +Traceback (most recent call last): + File "/Users/7ml/Documents/Codes/HydraGNN/tests/../examples/md17/md17.py", line 17, in + import hydragnn +ModuleNotFoundError: No module named 'hydragnn' +______________ pytest_examples_grad_forces[LennardJones-PNAPlus] _______________ + +example = 'LennardJones', mpnn_type = 'PNAPlus' + + @pytest.mark.parametrize( + "mpnn_type", + [ + "PNAPlus", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + "PAINN", + "MACE", + ], + ) + @pytest.mark.parametrize("example", ["LennardJones"]) + @pytest.mark.mpi_skip() + def pytest_examples_grad_forces(example, mpnn_type): + path = os.path.join(os.path.dirname(__file__), "..", "examples", example) + file_path = os.path.join(path, example + ".py") + + # Add the --mpnn_type argument for the subprocess call + return_code = subprocess.call(["python", file_path, "--mpnn_type", mpnn_type]) + + # Check the file ran without error. +> assert return_code == 0 +E assert 1 == 0 + +tests/test_examples.py:87: AssertionError +----------------------------- Captured stderr call ----------------------------- +Traceback (most recent call last): + File "/Users/7ml/Documents/Codes/HydraGNN/tests/../examples/LennardJones/LennardJones.py", line 33, in + import hydragnn +ModuleNotFoundError: No module named 'hydragnn' +_______________ pytest_examples_grad_forces[LennardJones-SchNet] _______________ + +example = 'LennardJones', mpnn_type = 'SchNet' + + @pytest.mark.parametrize( + "mpnn_type", + [ + "PNAPlus", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + "PAINN", + "MACE", + ], + ) + @pytest.mark.parametrize("example", ["LennardJones"]) + @pytest.mark.mpi_skip() + def pytest_examples_grad_forces(example, mpnn_type): + path = os.path.join(os.path.dirname(__file__), "..", "examples", example) + file_path = os.path.join(path, example + ".py") + + # Add the --mpnn_type argument for the subprocess call + return_code = subprocess.call(["python", file_path, "--mpnn_type", mpnn_type]) + + # Check the file ran without error. +> assert return_code == 0 +E assert 1 == 0 + +tests/test_examples.py:87: AssertionError +----------------------------- Captured stderr call ----------------------------- +Traceback (most recent call last): + File "/Users/7ml/Documents/Codes/HydraGNN/tests/../examples/LennardJones/LennardJones.py", line 33, in + import hydragnn +ModuleNotFoundError: No module named 'hydragnn' +______________ pytest_examples_grad_forces[LennardJones-DimeNet] _______________ + +example = 'LennardJones', mpnn_type = 'DimeNet' + + @pytest.mark.parametrize( + "mpnn_type", + [ + "PNAPlus", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + "PAINN", + "MACE", + ], + ) + @pytest.mark.parametrize("example", ["LennardJones"]) + @pytest.mark.mpi_skip() + def pytest_examples_grad_forces(example, mpnn_type): + path = os.path.join(os.path.dirname(__file__), "..", "examples", example) + file_path = os.path.join(path, example + ".py") + + # Add the --mpnn_type argument for the subprocess call + return_code = subprocess.call(["python", file_path, "--mpnn_type", mpnn_type]) + + # Check the file ran without error. +> assert return_code == 0 +E assert 1 == 0 + +tests/test_examples.py:87: AssertionError +----------------------------- Captured stderr call ----------------------------- +Traceback (most recent call last): + File "/Users/7ml/Documents/Codes/HydraGNN/tests/../examples/LennardJones/LennardJones.py", line 33, in + import hydragnn +ModuleNotFoundError: No module named 'hydragnn' +________________ pytest_examples_grad_forces[LennardJones-EGNN] ________________ + +example = 'LennardJones', mpnn_type = 'EGNN' + + @pytest.mark.parametrize( + "mpnn_type", + [ + "PNAPlus", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + "PAINN", + "MACE", + ], + ) + @pytest.mark.parametrize("example", ["LennardJones"]) + @pytest.mark.mpi_skip() + def pytest_examples_grad_forces(example, mpnn_type): + path = os.path.join(os.path.dirname(__file__), "..", "examples", example) + file_path = os.path.join(path, example + ".py") + + # Add the --mpnn_type argument for the subprocess call + return_code = subprocess.call(["python", file_path, "--mpnn_type", mpnn_type]) + + # Check the file ran without error. +> assert return_code == 0 +E assert 1 == 0 + +tests/test_examples.py:87: AssertionError +----------------------------- Captured stderr call ----------------------------- +Traceback (most recent call last): + File "/Users/7ml/Documents/Codes/HydraGNN/tests/../examples/LennardJones/LennardJones.py", line 33, in + import hydragnn +ModuleNotFoundError: No module named 'hydragnn' +_______________ pytest_examples_grad_forces[LennardJones-PNAEq] ________________ + +example = 'LennardJones', mpnn_type = 'PNAEq' + + @pytest.mark.parametrize( + "mpnn_type", + [ + "PNAPlus", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + "PAINN", + "MACE", + ], + ) + @pytest.mark.parametrize("example", ["LennardJones"]) + @pytest.mark.mpi_skip() + def pytest_examples_grad_forces(example, mpnn_type): + path = os.path.join(os.path.dirname(__file__), "..", "examples", example) + file_path = os.path.join(path, example + ".py") + + # Add the --mpnn_type argument for the subprocess call + return_code = subprocess.call(["python", file_path, "--mpnn_type", mpnn_type]) + + # Check the file ran without error. +> assert return_code == 0 +E assert 1 == 0 + +tests/test_examples.py:87: AssertionError +----------------------------- Captured stderr call ----------------------------- +Traceback (most recent call last): + File "/Users/7ml/Documents/Codes/HydraGNN/tests/../examples/LennardJones/LennardJones.py", line 33, in + import hydragnn +ModuleNotFoundError: No module named 'hydragnn' +_______________ pytest_examples_grad_forces[LennardJones-PAINN] ________________ + +example = 'LennardJones', mpnn_type = 'PAINN' + + @pytest.mark.parametrize( + "mpnn_type", + [ + "PNAPlus", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + "PAINN", + "MACE", + ], + ) + @pytest.mark.parametrize("example", ["LennardJones"]) + @pytest.mark.mpi_skip() + def pytest_examples_grad_forces(example, mpnn_type): + path = os.path.join(os.path.dirname(__file__), "..", "examples", example) + file_path = os.path.join(path, example + ".py") + + # Add the --mpnn_type argument for the subprocess call + return_code = subprocess.call(["python", file_path, "--mpnn_type", mpnn_type]) + + # Check the file ran without error. +> assert return_code == 0 +E assert 1 == 0 + +tests/test_examples.py:87: AssertionError +----------------------------- Captured stderr call ----------------------------- +Traceback (most recent call last): + File "/Users/7ml/Documents/Codes/HydraGNN/tests/../examples/LennardJones/LennardJones.py", line 33, in + import hydragnn +ModuleNotFoundError: No module named 'hydragnn' +________________ pytest_examples_grad_forces[LennardJones-MACE] ________________ + +example = 'LennardJones', mpnn_type = 'MACE' + + @pytest.mark.parametrize( + "mpnn_type", + [ + "PNAPlus", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + "PAINN", + "MACE", + ], + ) + @pytest.mark.parametrize("example", ["LennardJones"]) + @pytest.mark.mpi_skip() + def pytest_examples_grad_forces(example, mpnn_type): + path = os.path.join(os.path.dirname(__file__), "..", "examples", example) + file_path = os.path.join(path, example + ".py") + + # Add the --mpnn_type argument for the subprocess call + return_code = subprocess.call(["python", file_path, "--mpnn_type", mpnn_type]) + + # Check the file ran without error. +> assert return_code == 0 +E assert 1 == 0 + +tests/test_examples.py:87: AssertionError +----------------------------- Captured stderr call ----------------------------- +Traceback (most recent call last): + File "/Users/7ml/Documents/Codes/HydraGNN/tests/../examples/LennardJones/LennardJones.py", line 33, in + import hydragnn +ModuleNotFoundError: No module named 'hydragnn' +_____________________ pytest_examples[SchNet-LennardJones] _____________________ + +example = 'LennardJones', mpnn_type = 'SchNet' + + @pytest.mark.parametrize("example", ["LennardJones"]) + @pytest.mark.parametrize( + "mpnn_type", ["SchNet", "EGNN", "DimeNet", "PAINN", "PNAPlus", "MACE"] + ) + @pytest.mark.mpi_skip() + def pytest_examples(example, mpnn_type): + path = os.path.join(os.path.dirname(__file__), "..", "examples", example) + file_path = os.path.join(path, example + ".py") # Assuming different model scripts + return_code = subprocess.call(["python", file_path, "--mpnn_type", mpnn_type]) + + # Check the file ran without error. +> assert return_code == 0 +E assert 1 == 0 + +tests/test_forces_equivariant.py:29: AssertionError +----------------------------- Captured stderr call ----------------------------- +Traceback (most recent call last): + File "/Users/7ml/Documents/Codes/HydraGNN/tests/../examples/LennardJones/LennardJones.py", line 33, in + import hydragnn +ModuleNotFoundError: No module named 'hydragnn' +______________________ pytest_examples[EGNN-LennardJones] ______________________ + +example = 'LennardJones', mpnn_type = 'EGNN' + + @pytest.mark.parametrize("example", ["LennardJones"]) + @pytest.mark.parametrize( + "mpnn_type", ["SchNet", "EGNN", "DimeNet", "PAINN", "PNAPlus", "MACE"] + ) + @pytest.mark.mpi_skip() + def pytest_examples(example, mpnn_type): + path = os.path.join(os.path.dirname(__file__), "..", "examples", example) + file_path = os.path.join(path, example + ".py") # Assuming different model scripts + return_code = subprocess.call(["python", file_path, "--mpnn_type", mpnn_type]) + + # Check the file ran without error. +> assert return_code == 0 +E assert 1 == 0 + +tests/test_forces_equivariant.py:29: AssertionError +----------------------------- Captured stderr call ----------------------------- +Traceback (most recent call last): + File "/Users/7ml/Documents/Codes/HydraGNN/tests/../examples/LennardJones/LennardJones.py", line 33, in + import hydragnn +ModuleNotFoundError: No module named 'hydragnn' +____________________ pytest_examples[DimeNet-LennardJones] _____________________ + +example = 'LennardJones', mpnn_type = 'DimeNet' + + @pytest.mark.parametrize("example", ["LennardJones"]) + @pytest.mark.parametrize( + "mpnn_type", ["SchNet", "EGNN", "DimeNet", "PAINN", "PNAPlus", "MACE"] + ) + @pytest.mark.mpi_skip() + def pytest_examples(example, mpnn_type): + path = os.path.join(os.path.dirname(__file__), "..", "examples", example) + file_path = os.path.join(path, example + ".py") # Assuming different model scripts + return_code = subprocess.call(["python", file_path, "--mpnn_type", mpnn_type]) + + # Check the file ran without error. +> assert return_code == 0 +E assert 1 == 0 + +tests/test_forces_equivariant.py:29: AssertionError +----------------------------- Captured stderr call ----------------------------- +Traceback (most recent call last): + File "/Users/7ml/Documents/Codes/HydraGNN/tests/../examples/LennardJones/LennardJones.py", line 33, in + import hydragnn +ModuleNotFoundError: No module named 'hydragnn' +_____________________ pytest_examples[PAINN-LennardJones] ______________________ + +example = 'LennardJones', mpnn_type = 'PAINN' + + @pytest.mark.parametrize("example", ["LennardJones"]) + @pytest.mark.parametrize( + "mpnn_type", ["SchNet", "EGNN", "DimeNet", "PAINN", "PNAPlus", "MACE"] + ) + @pytest.mark.mpi_skip() + def pytest_examples(example, mpnn_type): + path = os.path.join(os.path.dirname(__file__), "..", "examples", example) + file_path = os.path.join(path, example + ".py") # Assuming different model scripts + return_code = subprocess.call(["python", file_path, "--mpnn_type", mpnn_type]) + + # Check the file ran without error. +> assert return_code == 0 +E assert 1 == 0 + +tests/test_forces_equivariant.py:29: AssertionError +----------------------------- Captured stderr call ----------------------------- +Traceback (most recent call last): + File "/Users/7ml/Documents/Codes/HydraGNN/tests/../examples/LennardJones/LennardJones.py", line 33, in + import hydragnn +ModuleNotFoundError: No module named 'hydragnn' +____________________ pytest_examples[PNAPlus-LennardJones] _____________________ + +example = 'LennardJones', mpnn_type = 'PNAPlus' + + @pytest.mark.parametrize("example", ["LennardJones"]) + @pytest.mark.parametrize( + "mpnn_type", ["SchNet", "EGNN", "DimeNet", "PAINN", "PNAPlus", "MACE"] + ) + @pytest.mark.mpi_skip() + def pytest_examples(example, mpnn_type): + path = os.path.join(os.path.dirname(__file__), "..", "examples", example) + file_path = os.path.join(path, example + ".py") # Assuming different model scripts + return_code = subprocess.call(["python", file_path, "--mpnn_type", mpnn_type]) + + # Check the file ran without error. +> assert return_code == 0 +E assert 1 == 0 + +tests/test_forces_equivariant.py:29: AssertionError +----------------------------- Captured stderr call ----------------------------- +Traceback (most recent call last): + File "/Users/7ml/Documents/Codes/HydraGNN/tests/../examples/LennardJones/LennardJones.py", line 33, in + import hydragnn +ModuleNotFoundError: No module named 'hydragnn' +______________________ pytest_examples[MACE-LennardJones] ______________________ + +example = 'LennardJones', mpnn_type = 'MACE' + + @pytest.mark.parametrize("example", ["LennardJones"]) + @pytest.mark.parametrize( + "mpnn_type", ["SchNet", "EGNN", "DimeNet", "PAINN", "PNAPlus", "MACE"] + ) + @pytest.mark.mpi_skip() + def pytest_examples(example, mpnn_type): + path = os.path.join(os.path.dirname(__file__), "..", "examples", example) + file_path = os.path.join(path, example + ".py") # Assuming different model scripts + return_code = subprocess.call(["python", file_path, "--mpnn_type", mpnn_type]) + + # Check the file ran without error. +> assert return_code == 0 +E assert 1 == 0 + +tests/test_forces_equivariant.py:29: AssertionError +----------------------------- Captured stderr call ----------------------------- +Traceback (most recent call last): + File "/Users/7ml/Documents/Codes/HydraGNN/tests/../examples/LennardJones/LennardJones.py", line 33, in + import hydragnn +ModuleNotFoundError: No module named 'hydragnn' +_______________________ pytest_train_model[ci.json-SAGE] _______________________ + +mpnn_type = 'SAGE', ci_input = 'ci.json', overwrite_data = False + + @pytest.mark.parametrize( + "mpnn_type", + [ + "SAGE", + "GIN", + "GAT", + "MFC", + "PNA", + "PNAPlus", + "CGCNN", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + "PAINN", + "MACE", + ], + ) + @pytest.mark.parametrize("ci_input", ["ci.json", "ci_multihead.json"]) + def pytest_train_model(mpnn_type, ci_input, overwrite_data=False): +> unittest_train_model(mpnn_type, None, None, ci_input, False, overwrite_data) + +tests/test_graphs.py:223: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_graphs.py:134: in unittest_train_model + hydragnn.run_training(config, use_deepspeed) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:80: in _ + world_size, world_rank = setup_ddp(use_deepspeed=use_deepspeed) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/utils/distributed/distributed.py:206: in setup_ddp + dist.init_process_group( +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:81: in wrapper + return func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:95: in wrapper + func_return = func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/distributed_c10d.py:1757: in init_process_group + store, rank, world_size = next(rendezvous_iterator) + ^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:278: in _env_rendezvous_handler + store = _create_c10d_store( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +hostname = '127.0.0.1', port = 8889, rank = 0, world_size = 1 +timeout = datetime.timedelta(seconds=1800), use_libuv = True + + def _create_c10d_store( + hostname, port, rank, world_size, timeout, use_libuv=True + ) -> Store: + """ + Smartly creates a c10d Store object on ``rank`` based on whether we need to reuse agent store. + + The TCPStore server is assumed to be hosted + on ``hostname:port``. + + By default, the TCPStore server uses the asynchronous implementation + ``LibUVStoreDaemon`` which utilizes libuv. + + If ``torchelastic_use_agent_store()`` is ``True``, then it is assumed that + the agent leader (node rank 0) hosts the TCPStore server (for which the + endpoint is specified by the given ``hostname:port``). Hence + ALL ranks will create and return a TCPStore client (e.g. ``start_daemon=False``). + + If ``torchelastic_use_agent_store()`` is ``False``, then rank 0 will host + the TCPStore (with multi-tenancy) and it is assumed that rank 0's hostname + and port are correctly passed via ``hostname`` and ``port``. All + non-zero ranks will create and return a TCPStore client. + """ + # check if port is uint16_t + if not 0 <= port < 2**16: + raise ValueError(f"port must have value from 0 to 65535 but was {port}.") + + if _torchelastic_use_agent_store(): + # We create a new TCPStore for every retry so no need to add prefix for each attempt. + return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=False, + timeout=timeout, + ) + else: + start_daemon = rank == 0 +> return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=start_daemon, + timeout=timeout, + multi_tenant=True, + use_libuv=use_libuv, + ) +E torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 8889, useIpv6: false, code: -48, name: EADDRINUSE, message: address already in use + +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:198: DistNetworkError +----------------------------- Captured stdout call ----------------------------- +Distributed data parallel: gloo master at 127.0.0.1:8889 +_______________________ pytest_train_model[ci.json-GIN] ________________________ + +mpnn_type = 'GIN', ci_input = 'ci.json', overwrite_data = False + + @pytest.mark.parametrize( + "mpnn_type", + [ + "SAGE", + "GIN", + "GAT", + "MFC", + "PNA", + "PNAPlus", + "CGCNN", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + "PAINN", + "MACE", + ], + ) + @pytest.mark.parametrize("ci_input", ["ci.json", "ci_multihead.json"]) + def pytest_train_model(mpnn_type, ci_input, overwrite_data=False): +> unittest_train_model(mpnn_type, None, None, ci_input, False, overwrite_data) + +tests/test_graphs.py:223: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_graphs.py:134: in unittest_train_model + hydragnn.run_training(config, use_deepspeed) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:80: in _ + world_size, world_rank = setup_ddp(use_deepspeed=use_deepspeed) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/utils/distributed/distributed.py:206: in setup_ddp + dist.init_process_group( +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:81: in wrapper + return func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:95: in wrapper + func_return = func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/distributed_c10d.py:1757: in init_process_group + store, rank, world_size = next(rendezvous_iterator) + ^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:278: in _env_rendezvous_handler + store = _create_c10d_store( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +hostname = '127.0.0.1', port = 8889, rank = 0, world_size = 1 +timeout = datetime.timedelta(seconds=1800), use_libuv = True + + def _create_c10d_store( + hostname, port, rank, world_size, timeout, use_libuv=True + ) -> Store: + """ + Smartly creates a c10d Store object on ``rank`` based on whether we need to reuse agent store. + + The TCPStore server is assumed to be hosted + on ``hostname:port``. + + By default, the TCPStore server uses the asynchronous implementation + ``LibUVStoreDaemon`` which utilizes libuv. + + If ``torchelastic_use_agent_store()`` is ``True``, then it is assumed that + the agent leader (node rank 0) hosts the TCPStore server (for which the + endpoint is specified by the given ``hostname:port``). Hence + ALL ranks will create and return a TCPStore client (e.g. ``start_daemon=False``). + + If ``torchelastic_use_agent_store()`` is ``False``, then rank 0 will host + the TCPStore (with multi-tenancy) and it is assumed that rank 0's hostname + and port are correctly passed via ``hostname`` and ``port``. All + non-zero ranks will create and return a TCPStore client. + """ + # check if port is uint16_t + if not 0 <= port < 2**16: + raise ValueError(f"port must have value from 0 to 65535 but was {port}.") + + if _torchelastic_use_agent_store(): + # We create a new TCPStore for every retry so no need to add prefix for each attempt. + return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=False, + timeout=timeout, + ) + else: + start_daemon = rank == 0 +> return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=start_daemon, + timeout=timeout, + multi_tenant=True, + use_libuv=use_libuv, + ) +E torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 8889, useIpv6: false, code: -48, name: EADDRINUSE, message: address already in use + +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:198: DistNetworkError +----------------------------- Captured stdout call ----------------------------- +Distributed data parallel: gloo master at 127.0.0.1:8889 +_______________________ pytest_train_model[ci.json-GAT] ________________________ + +mpnn_type = 'GAT', ci_input = 'ci.json', overwrite_data = False + + @pytest.mark.parametrize( + "mpnn_type", + [ + "SAGE", + "GIN", + "GAT", + "MFC", + "PNA", + "PNAPlus", + "CGCNN", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + "PAINN", + "MACE", + ], + ) + @pytest.mark.parametrize("ci_input", ["ci.json", "ci_multihead.json"]) + def pytest_train_model(mpnn_type, ci_input, overwrite_data=False): +> unittest_train_model(mpnn_type, None, None, ci_input, False, overwrite_data) + +tests/test_graphs.py:223: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_graphs.py:134: in unittest_train_model + hydragnn.run_training(config, use_deepspeed) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:80: in _ + world_size, world_rank = setup_ddp(use_deepspeed=use_deepspeed) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/utils/distributed/distributed.py:206: in setup_ddp + dist.init_process_group( +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:81: in wrapper + return func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:95: in wrapper + func_return = func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/distributed_c10d.py:1757: in init_process_group + store, rank, world_size = next(rendezvous_iterator) + ^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:278: in _env_rendezvous_handler + store = _create_c10d_store( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +hostname = '127.0.0.1', port = 8889, rank = 0, world_size = 1 +timeout = datetime.timedelta(seconds=1800), use_libuv = True + + def _create_c10d_store( + hostname, port, rank, world_size, timeout, use_libuv=True + ) -> Store: + """ + Smartly creates a c10d Store object on ``rank`` based on whether we need to reuse agent store. + + The TCPStore server is assumed to be hosted + on ``hostname:port``. + + By default, the TCPStore server uses the asynchronous implementation + ``LibUVStoreDaemon`` which utilizes libuv. + + If ``torchelastic_use_agent_store()`` is ``True``, then it is assumed that + the agent leader (node rank 0) hosts the TCPStore server (for which the + endpoint is specified by the given ``hostname:port``). Hence + ALL ranks will create and return a TCPStore client (e.g. ``start_daemon=False``). + + If ``torchelastic_use_agent_store()`` is ``False``, then rank 0 will host + the TCPStore (with multi-tenancy) and it is assumed that rank 0's hostname + and port are correctly passed via ``hostname`` and ``port``. All + non-zero ranks will create and return a TCPStore client. + """ + # check if port is uint16_t + if not 0 <= port < 2**16: + raise ValueError(f"port must have value from 0 to 65535 but was {port}.") + + if _torchelastic_use_agent_store(): + # We create a new TCPStore for every retry so no need to add prefix for each attempt. + return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=False, + timeout=timeout, + ) + else: + start_daemon = rank == 0 +> return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=start_daemon, + timeout=timeout, + multi_tenant=True, + use_libuv=use_libuv, + ) +E torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 8889, useIpv6: false, code: -48, name: EADDRINUSE, message: address already in use + +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:198: DistNetworkError +----------------------------- Captured stdout call ----------------------------- +Distributed data parallel: gloo master at 127.0.0.1:8889 +_______________________ pytest_train_model[ci.json-MFC] ________________________ + +mpnn_type = 'MFC', ci_input = 'ci.json', overwrite_data = False + + @pytest.mark.parametrize( + "mpnn_type", + [ + "SAGE", + "GIN", + "GAT", + "MFC", + "PNA", + "PNAPlus", + "CGCNN", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + "PAINN", + "MACE", + ], + ) + @pytest.mark.parametrize("ci_input", ["ci.json", "ci_multihead.json"]) + def pytest_train_model(mpnn_type, ci_input, overwrite_data=False): +> unittest_train_model(mpnn_type, None, None, ci_input, False, overwrite_data) + +tests/test_graphs.py:223: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_graphs.py:134: in unittest_train_model + hydragnn.run_training(config, use_deepspeed) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:80: in _ + world_size, world_rank = setup_ddp(use_deepspeed=use_deepspeed) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/utils/distributed/distributed.py:206: in setup_ddp + dist.init_process_group( +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:81: in wrapper + return func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:95: in wrapper + func_return = func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/distributed_c10d.py:1757: in init_process_group + store, rank, world_size = next(rendezvous_iterator) + ^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:278: in _env_rendezvous_handler + store = _create_c10d_store( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +hostname = '127.0.0.1', port = 8889, rank = 0, world_size = 1 +timeout = datetime.timedelta(seconds=1800), use_libuv = True + + def _create_c10d_store( + hostname, port, rank, world_size, timeout, use_libuv=True + ) -> Store: + """ + Smartly creates a c10d Store object on ``rank`` based on whether we need to reuse agent store. + + The TCPStore server is assumed to be hosted + on ``hostname:port``. + + By default, the TCPStore server uses the asynchronous implementation + ``LibUVStoreDaemon`` which utilizes libuv. + + If ``torchelastic_use_agent_store()`` is ``True``, then it is assumed that + the agent leader (node rank 0) hosts the TCPStore server (for which the + endpoint is specified by the given ``hostname:port``). Hence + ALL ranks will create and return a TCPStore client (e.g. ``start_daemon=False``). + + If ``torchelastic_use_agent_store()`` is ``False``, then rank 0 will host + the TCPStore (with multi-tenancy) and it is assumed that rank 0's hostname + and port are correctly passed via ``hostname`` and ``port``. All + non-zero ranks will create and return a TCPStore client. + """ + # check if port is uint16_t + if not 0 <= port < 2**16: + raise ValueError(f"port must have value from 0 to 65535 but was {port}.") + + if _torchelastic_use_agent_store(): + # We create a new TCPStore for every retry so no need to add prefix for each attempt. + return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=False, + timeout=timeout, + ) + else: + start_daemon = rank == 0 +> return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=start_daemon, + timeout=timeout, + multi_tenant=True, + use_libuv=use_libuv, + ) +E torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 8889, useIpv6: false, code: -48, name: EADDRINUSE, message: address already in use + +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:198: DistNetworkError +----------------------------- Captured stdout call ----------------------------- +Distributed data parallel: gloo master at 127.0.0.1:8889 +_______________________ pytest_train_model[ci.json-PNA] ________________________ + +mpnn_type = 'PNA', ci_input = 'ci.json', overwrite_data = False + + @pytest.mark.parametrize( + "mpnn_type", + [ + "SAGE", + "GIN", + "GAT", + "MFC", + "PNA", + "PNAPlus", + "CGCNN", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + "PAINN", + "MACE", + ], + ) + @pytest.mark.parametrize("ci_input", ["ci.json", "ci_multihead.json"]) + def pytest_train_model(mpnn_type, ci_input, overwrite_data=False): +> unittest_train_model(mpnn_type, None, None, ci_input, False, overwrite_data) + +tests/test_graphs.py:223: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_graphs.py:132: in unittest_train_model + hydragnn.run_training(config_file, use_deepspeed) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:62: in _ + run_training(config) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:80: in _ + world_size, world_rank = setup_ddp(use_deepspeed=use_deepspeed) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/utils/distributed/distributed.py:206: in setup_ddp + dist.init_process_group( +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:81: in wrapper + return func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:95: in wrapper + func_return = func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/distributed_c10d.py:1757: in init_process_group + store, rank, world_size = next(rendezvous_iterator) + ^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:278: in _env_rendezvous_handler + store = _create_c10d_store( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +hostname = '127.0.0.1', port = 8889, rank = 0, world_size = 1 +timeout = datetime.timedelta(seconds=1800), use_libuv = True + + def _create_c10d_store( + hostname, port, rank, world_size, timeout, use_libuv=True + ) -> Store: + """ + Smartly creates a c10d Store object on ``rank`` based on whether we need to reuse agent store. + + The TCPStore server is assumed to be hosted + on ``hostname:port``. + + By default, the TCPStore server uses the asynchronous implementation + ``LibUVStoreDaemon`` which utilizes libuv. + + If ``torchelastic_use_agent_store()`` is ``True``, then it is assumed that + the agent leader (node rank 0) hosts the TCPStore server (for which the + endpoint is specified by the given ``hostname:port``). Hence + ALL ranks will create and return a TCPStore client (e.g. ``start_daemon=False``). + + If ``torchelastic_use_agent_store()`` is ``False``, then rank 0 will host + the TCPStore (with multi-tenancy) and it is assumed that rank 0's hostname + and port are correctly passed via ``hostname`` and ``port``. All + non-zero ranks will create and return a TCPStore client. + """ + # check if port is uint16_t + if not 0 <= port < 2**16: + raise ValueError(f"port must have value from 0 to 65535 but was {port}.") + + if _torchelastic_use_agent_store(): + # We create a new TCPStore for every retry so no need to add prefix for each attempt. + return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=False, + timeout=timeout, + ) + else: + start_daemon = rank == 0 +> return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=start_daemon, + timeout=timeout, + multi_tenant=True, + use_libuv=use_libuv, + ) +E torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 8889, useIpv6: false, code: -48, name: EADDRINUSE, message: address already in use + +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:198: DistNetworkError +----------------------------- Captured stdout call ----------------------------- +Distributed data parallel: gloo master at 127.0.0.1:8889 +_____________________ pytest_train_model[ci.json-PNAPlus] ______________________ + +mpnn_type = 'PNAPlus', ci_input = 'ci.json', overwrite_data = False + + @pytest.mark.parametrize( + "mpnn_type", + [ + "SAGE", + "GIN", + "GAT", + "MFC", + "PNA", + "PNAPlus", + "CGCNN", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + "PAINN", + "MACE", + ], + ) + @pytest.mark.parametrize("ci_input", ["ci.json", "ci_multihead.json"]) + def pytest_train_model(mpnn_type, ci_input, overwrite_data=False): +> unittest_train_model(mpnn_type, None, None, ci_input, False, overwrite_data) + +tests/test_graphs.py:223: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_graphs.py:134: in unittest_train_model + hydragnn.run_training(config, use_deepspeed) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:80: in _ + world_size, world_rank = setup_ddp(use_deepspeed=use_deepspeed) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/utils/distributed/distributed.py:206: in setup_ddp + dist.init_process_group( +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:81: in wrapper + return func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:95: in wrapper + func_return = func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/distributed_c10d.py:1757: in init_process_group + store, rank, world_size = next(rendezvous_iterator) + ^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:278: in _env_rendezvous_handler + store = _create_c10d_store( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +hostname = '127.0.0.1', port = 8889, rank = 0, world_size = 1 +timeout = datetime.timedelta(seconds=1800), use_libuv = True + + def _create_c10d_store( + hostname, port, rank, world_size, timeout, use_libuv=True + ) -> Store: + """ + Smartly creates a c10d Store object on ``rank`` based on whether we need to reuse agent store. + + The TCPStore server is assumed to be hosted + on ``hostname:port``. + + By default, the TCPStore server uses the asynchronous implementation + ``LibUVStoreDaemon`` which utilizes libuv. + + If ``torchelastic_use_agent_store()`` is ``True``, then it is assumed that + the agent leader (node rank 0) hosts the TCPStore server (for which the + endpoint is specified by the given ``hostname:port``). Hence + ALL ranks will create and return a TCPStore client (e.g. ``start_daemon=False``). + + If ``torchelastic_use_agent_store()`` is ``False``, then rank 0 will host + the TCPStore (with multi-tenancy) and it is assumed that rank 0's hostname + and port are correctly passed via ``hostname`` and ``port``. All + non-zero ranks will create and return a TCPStore client. + """ + # check if port is uint16_t + if not 0 <= port < 2**16: + raise ValueError(f"port must have value from 0 to 65535 but was {port}.") + + if _torchelastic_use_agent_store(): + # We create a new TCPStore for every retry so no need to add prefix for each attempt. + return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=False, + timeout=timeout, + ) + else: + start_daemon = rank == 0 +> return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=start_daemon, + timeout=timeout, + multi_tenant=True, + use_libuv=use_libuv, + ) +E torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 8889, useIpv6: false, code: -48, name: EADDRINUSE, message: address already in use + +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:198: DistNetworkError +----------------------------- Captured stdout call ----------------------------- +Distributed data parallel: gloo master at 127.0.0.1:8889 +______________________ pytest_train_model[ci.json-CGCNN] _______________________ + +mpnn_type = 'CGCNN', ci_input = 'ci.json', overwrite_data = False + + @pytest.mark.parametrize( + "mpnn_type", + [ + "SAGE", + "GIN", + "GAT", + "MFC", + "PNA", + "PNAPlus", + "CGCNN", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + "PAINN", + "MACE", + ], + ) + @pytest.mark.parametrize("ci_input", ["ci.json", "ci_multihead.json"]) + def pytest_train_model(mpnn_type, ci_input, overwrite_data=False): +> unittest_train_model(mpnn_type, None, None, ci_input, False, overwrite_data) + +tests/test_graphs.py:223: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_graphs.py:134: in unittest_train_model + hydragnn.run_training(config, use_deepspeed) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:80: in _ + world_size, world_rank = setup_ddp(use_deepspeed=use_deepspeed) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/utils/distributed/distributed.py:206: in setup_ddp + dist.init_process_group( +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:81: in wrapper + return func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:95: in wrapper + func_return = func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/distributed_c10d.py:1757: in init_process_group + store, rank, world_size = next(rendezvous_iterator) + ^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:278: in _env_rendezvous_handler + store = _create_c10d_store( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +hostname = '127.0.0.1', port = 8889, rank = 0, world_size = 1 +timeout = datetime.timedelta(seconds=1800), use_libuv = True + + def _create_c10d_store( + hostname, port, rank, world_size, timeout, use_libuv=True + ) -> Store: + """ + Smartly creates a c10d Store object on ``rank`` based on whether we need to reuse agent store. + + The TCPStore server is assumed to be hosted + on ``hostname:port``. + + By default, the TCPStore server uses the asynchronous implementation + ``LibUVStoreDaemon`` which utilizes libuv. + + If ``torchelastic_use_agent_store()`` is ``True``, then it is assumed that + the agent leader (node rank 0) hosts the TCPStore server (for which the + endpoint is specified by the given ``hostname:port``). Hence + ALL ranks will create and return a TCPStore client (e.g. ``start_daemon=False``). + + If ``torchelastic_use_agent_store()`` is ``False``, then rank 0 will host + the TCPStore (with multi-tenancy) and it is assumed that rank 0's hostname + and port are correctly passed via ``hostname`` and ``port``. All + non-zero ranks will create and return a TCPStore client. + """ + # check if port is uint16_t + if not 0 <= port < 2**16: + raise ValueError(f"port must have value from 0 to 65535 but was {port}.") + + if _torchelastic_use_agent_store(): + # We create a new TCPStore for every retry so no need to add prefix for each attempt. + return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=False, + timeout=timeout, + ) + else: + start_daemon = rank == 0 +> return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=start_daemon, + timeout=timeout, + multi_tenant=True, + use_libuv=use_libuv, + ) +E torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 8889, useIpv6: false, code: -48, name: EADDRINUSE, message: address already in use + +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:198: DistNetworkError +----------------------------- Captured stdout call ----------------------------- +Distributed data parallel: gloo master at 127.0.0.1:8889 +______________________ pytest_train_model[ci.json-SchNet] ______________________ + +mpnn_type = 'SchNet', ci_input = 'ci.json', overwrite_data = False + + @pytest.mark.parametrize( + "mpnn_type", + [ + "SAGE", + "GIN", + "GAT", + "MFC", + "PNA", + "PNAPlus", + "CGCNN", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + "PAINN", + "MACE", + ], + ) + @pytest.mark.parametrize("ci_input", ["ci.json", "ci_multihead.json"]) + def pytest_train_model(mpnn_type, ci_input, overwrite_data=False): +> unittest_train_model(mpnn_type, None, None, ci_input, False, overwrite_data) + +tests/test_graphs.py:223: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_graphs.py:134: in unittest_train_model + hydragnn.run_training(config, use_deepspeed) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:80: in _ + world_size, world_rank = setup_ddp(use_deepspeed=use_deepspeed) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/utils/distributed/distributed.py:206: in setup_ddp + dist.init_process_group( +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:81: in wrapper + return func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:95: in wrapper + func_return = func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/distributed_c10d.py:1757: in init_process_group + store, rank, world_size = next(rendezvous_iterator) + ^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:278: in _env_rendezvous_handler + store = _create_c10d_store( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +hostname = '127.0.0.1', port = 8889, rank = 0, world_size = 1 +timeout = datetime.timedelta(seconds=1800), use_libuv = True + + def _create_c10d_store( + hostname, port, rank, world_size, timeout, use_libuv=True + ) -> Store: + """ + Smartly creates a c10d Store object on ``rank`` based on whether we need to reuse agent store. + + The TCPStore server is assumed to be hosted + on ``hostname:port``. + + By default, the TCPStore server uses the asynchronous implementation + ``LibUVStoreDaemon`` which utilizes libuv. + + If ``torchelastic_use_agent_store()`` is ``True``, then it is assumed that + the agent leader (node rank 0) hosts the TCPStore server (for which the + endpoint is specified by the given ``hostname:port``). Hence + ALL ranks will create and return a TCPStore client (e.g. ``start_daemon=False``). + + If ``torchelastic_use_agent_store()`` is ``False``, then rank 0 will host + the TCPStore (with multi-tenancy) and it is assumed that rank 0's hostname + and port are correctly passed via ``hostname`` and ``port``. All + non-zero ranks will create and return a TCPStore client. + """ + # check if port is uint16_t + if not 0 <= port < 2**16: + raise ValueError(f"port must have value from 0 to 65535 but was {port}.") + + if _torchelastic_use_agent_store(): + # We create a new TCPStore for every retry so no need to add prefix for each attempt. + return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=False, + timeout=timeout, + ) + else: + start_daemon = rank == 0 +> return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=start_daemon, + timeout=timeout, + multi_tenant=True, + use_libuv=use_libuv, + ) +E torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 8889, useIpv6: false, code: -48, name: EADDRINUSE, message: address already in use + +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:198: DistNetworkError +----------------------------- Captured stdout call ----------------------------- +Distributed data parallel: gloo master at 127.0.0.1:8889 +_____________________ pytest_train_model[ci.json-DimeNet] ______________________ + +mpnn_type = 'DimeNet', ci_input = 'ci.json', overwrite_data = False + + @pytest.mark.parametrize( + "mpnn_type", + [ + "SAGE", + "GIN", + "GAT", + "MFC", + "PNA", + "PNAPlus", + "CGCNN", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + "PAINN", + "MACE", + ], + ) + @pytest.mark.parametrize("ci_input", ["ci.json", "ci_multihead.json"]) + def pytest_train_model(mpnn_type, ci_input, overwrite_data=False): +> unittest_train_model(mpnn_type, None, None, ci_input, False, overwrite_data) + +tests/test_graphs.py:223: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_graphs.py:134: in unittest_train_model + hydragnn.run_training(config, use_deepspeed) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:80: in _ + world_size, world_rank = setup_ddp(use_deepspeed=use_deepspeed) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/utils/distributed/distributed.py:206: in setup_ddp + dist.init_process_group( +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:81: in wrapper + return func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:95: in wrapper + func_return = func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/distributed_c10d.py:1757: in init_process_group + store, rank, world_size = next(rendezvous_iterator) + ^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:278: in _env_rendezvous_handler + store = _create_c10d_store( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +hostname = '127.0.0.1', port = 8889, rank = 0, world_size = 1 +timeout = datetime.timedelta(seconds=1800), use_libuv = True + + def _create_c10d_store( + hostname, port, rank, world_size, timeout, use_libuv=True + ) -> Store: + """ + Smartly creates a c10d Store object on ``rank`` based on whether we need to reuse agent store. + + The TCPStore server is assumed to be hosted + on ``hostname:port``. + + By default, the TCPStore server uses the asynchronous implementation + ``LibUVStoreDaemon`` which utilizes libuv. + + If ``torchelastic_use_agent_store()`` is ``True``, then it is assumed that + the agent leader (node rank 0) hosts the TCPStore server (for which the + endpoint is specified by the given ``hostname:port``). Hence + ALL ranks will create and return a TCPStore client (e.g. ``start_daemon=False``). + + If ``torchelastic_use_agent_store()`` is ``False``, then rank 0 will host + the TCPStore (with multi-tenancy) and it is assumed that rank 0's hostname + and port are correctly passed via ``hostname`` and ``port``. All + non-zero ranks will create and return a TCPStore client. + """ + # check if port is uint16_t + if not 0 <= port < 2**16: + raise ValueError(f"port must have value from 0 to 65535 but was {port}.") + + if _torchelastic_use_agent_store(): + # We create a new TCPStore for every retry so no need to add prefix for each attempt. + return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=False, + timeout=timeout, + ) + else: + start_daemon = rank == 0 +> return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=start_daemon, + timeout=timeout, + multi_tenant=True, + use_libuv=use_libuv, + ) +E torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 8889, useIpv6: false, code: -48, name: EADDRINUSE, message: address already in use + +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:198: DistNetworkError +----------------------------- Captured stdout call ----------------------------- +Distributed data parallel: gloo master at 127.0.0.1:8889 +_______________________ pytest_train_model[ci.json-EGNN] _______________________ + +mpnn_type = 'EGNN', ci_input = 'ci.json', overwrite_data = False + + @pytest.mark.parametrize( + "mpnn_type", + [ + "SAGE", + "GIN", + "GAT", + "MFC", + "PNA", + "PNAPlus", + "CGCNN", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + "PAINN", + "MACE", + ], + ) + @pytest.mark.parametrize("ci_input", ["ci.json", "ci_multihead.json"]) + def pytest_train_model(mpnn_type, ci_input, overwrite_data=False): +> unittest_train_model(mpnn_type, None, None, ci_input, False, overwrite_data) + +tests/test_graphs.py:223: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_graphs.py:134: in unittest_train_model + hydragnn.run_training(config, use_deepspeed) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:80: in _ + world_size, world_rank = setup_ddp(use_deepspeed=use_deepspeed) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/utils/distributed/distributed.py:206: in setup_ddp + dist.init_process_group( +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:81: in wrapper + return func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:95: in wrapper + func_return = func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/distributed_c10d.py:1757: in init_process_group + store, rank, world_size = next(rendezvous_iterator) + ^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:278: in _env_rendezvous_handler + store = _create_c10d_store( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +hostname = '127.0.0.1', port = 8889, rank = 0, world_size = 1 +timeout = datetime.timedelta(seconds=1800), use_libuv = True + + def _create_c10d_store( + hostname, port, rank, world_size, timeout, use_libuv=True + ) -> Store: + """ + Smartly creates a c10d Store object on ``rank`` based on whether we need to reuse agent store. + + The TCPStore server is assumed to be hosted + on ``hostname:port``. + + By default, the TCPStore server uses the asynchronous implementation + ``LibUVStoreDaemon`` which utilizes libuv. + + If ``torchelastic_use_agent_store()`` is ``True``, then it is assumed that + the agent leader (node rank 0) hosts the TCPStore server (for which the + endpoint is specified by the given ``hostname:port``). Hence + ALL ranks will create and return a TCPStore client (e.g. ``start_daemon=False``). + + If ``torchelastic_use_agent_store()`` is ``False``, then rank 0 will host + the TCPStore (with multi-tenancy) and it is assumed that rank 0's hostname + and port are correctly passed via ``hostname`` and ``port``. All + non-zero ranks will create and return a TCPStore client. + """ + # check if port is uint16_t + if not 0 <= port < 2**16: + raise ValueError(f"port must have value from 0 to 65535 but was {port}.") + + if _torchelastic_use_agent_store(): + # We create a new TCPStore for every retry so no need to add prefix for each attempt. + return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=False, + timeout=timeout, + ) + else: + start_daemon = rank == 0 +> return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=start_daemon, + timeout=timeout, + multi_tenant=True, + use_libuv=use_libuv, + ) +E torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 8889, useIpv6: false, code: -48, name: EADDRINUSE, message: address already in use + +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:198: DistNetworkError +----------------------------- Captured stdout call ----------------------------- +Distributed data parallel: gloo master at 127.0.0.1:8889 +______________________ pytest_train_model[ci.json-PNAEq] _______________________ + +mpnn_type = 'PNAEq', ci_input = 'ci.json', overwrite_data = False + + @pytest.mark.parametrize( + "mpnn_type", + [ + "SAGE", + "GIN", + "GAT", + "MFC", + "PNA", + "PNAPlus", + "CGCNN", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + "PAINN", + "MACE", + ], + ) + @pytest.mark.parametrize("ci_input", ["ci.json", "ci_multihead.json"]) + def pytest_train_model(mpnn_type, ci_input, overwrite_data=False): +> unittest_train_model(mpnn_type, None, None, ci_input, False, overwrite_data) + +tests/test_graphs.py:223: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_graphs.py:134: in unittest_train_model + hydragnn.run_training(config, use_deepspeed) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:80: in _ + world_size, world_rank = setup_ddp(use_deepspeed=use_deepspeed) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/utils/distributed/distributed.py:206: in setup_ddp + dist.init_process_group( +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:81: in wrapper + return func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:95: in wrapper + func_return = func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/distributed_c10d.py:1757: in init_process_group + store, rank, world_size = next(rendezvous_iterator) + ^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:278: in _env_rendezvous_handler + store = _create_c10d_store( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +hostname = '127.0.0.1', port = 8889, rank = 0, world_size = 1 +timeout = datetime.timedelta(seconds=1800), use_libuv = True + + def _create_c10d_store( + hostname, port, rank, world_size, timeout, use_libuv=True + ) -> Store: + """ + Smartly creates a c10d Store object on ``rank`` based on whether we need to reuse agent store. + + The TCPStore server is assumed to be hosted + on ``hostname:port``. + + By default, the TCPStore server uses the asynchronous implementation + ``LibUVStoreDaemon`` which utilizes libuv. + + If ``torchelastic_use_agent_store()`` is ``True``, then it is assumed that + the agent leader (node rank 0) hosts the TCPStore server (for which the + endpoint is specified by the given ``hostname:port``). Hence + ALL ranks will create and return a TCPStore client (e.g. ``start_daemon=False``). + + If ``torchelastic_use_agent_store()`` is ``False``, then rank 0 will host + the TCPStore (with multi-tenancy) and it is assumed that rank 0's hostname + and port are correctly passed via ``hostname`` and ``port``. All + non-zero ranks will create and return a TCPStore client. + """ + # check if port is uint16_t + if not 0 <= port < 2**16: + raise ValueError(f"port must have value from 0 to 65535 but was {port}.") + + if _torchelastic_use_agent_store(): + # We create a new TCPStore for every retry so no need to add prefix for each attempt. + return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=False, + timeout=timeout, + ) + else: + start_daemon = rank == 0 +> return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=start_daemon, + timeout=timeout, + multi_tenant=True, + use_libuv=use_libuv, + ) +E torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 8889, useIpv6: false, code: -48, name: EADDRINUSE, message: address already in use + +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:198: DistNetworkError +----------------------------- Captured stdout call ----------------------------- +Distributed data parallel: gloo master at 127.0.0.1:8889 +______________________ pytest_train_model[ci.json-PAINN] _______________________ + +mpnn_type = 'PAINN', ci_input = 'ci.json', overwrite_data = False + + @pytest.mark.parametrize( + "mpnn_type", + [ + "SAGE", + "GIN", + "GAT", + "MFC", + "PNA", + "PNAPlus", + "CGCNN", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + "PAINN", + "MACE", + ], + ) + @pytest.mark.parametrize("ci_input", ["ci.json", "ci_multihead.json"]) + def pytest_train_model(mpnn_type, ci_input, overwrite_data=False): +> unittest_train_model(mpnn_type, None, None, ci_input, False, overwrite_data) + +tests/test_graphs.py:223: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_graphs.py:134: in unittest_train_model + hydragnn.run_training(config, use_deepspeed) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:80: in _ + world_size, world_rank = setup_ddp(use_deepspeed=use_deepspeed) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/utils/distributed/distributed.py:206: in setup_ddp + dist.init_process_group( +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:81: in wrapper + return func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:95: in wrapper + func_return = func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/distributed_c10d.py:1757: in init_process_group + store, rank, world_size = next(rendezvous_iterator) + ^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:278: in _env_rendezvous_handler + store = _create_c10d_store( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +hostname = '127.0.0.1', port = 8889, rank = 0, world_size = 1 +timeout = datetime.timedelta(seconds=1800), use_libuv = True + + def _create_c10d_store( + hostname, port, rank, world_size, timeout, use_libuv=True + ) -> Store: + """ + Smartly creates a c10d Store object on ``rank`` based on whether we need to reuse agent store. + + The TCPStore server is assumed to be hosted + on ``hostname:port``. + + By default, the TCPStore server uses the asynchronous implementation + ``LibUVStoreDaemon`` which utilizes libuv. + + If ``torchelastic_use_agent_store()`` is ``True``, then it is assumed that + the agent leader (node rank 0) hosts the TCPStore server (for which the + endpoint is specified by the given ``hostname:port``). Hence + ALL ranks will create and return a TCPStore client (e.g. ``start_daemon=False``). + + If ``torchelastic_use_agent_store()`` is ``False``, then rank 0 will host + the TCPStore (with multi-tenancy) and it is assumed that rank 0's hostname + and port are correctly passed via ``hostname`` and ``port``. All + non-zero ranks will create and return a TCPStore client. + """ + # check if port is uint16_t + if not 0 <= port < 2**16: + raise ValueError(f"port must have value from 0 to 65535 but was {port}.") + + if _torchelastic_use_agent_store(): + # We create a new TCPStore for every retry so no need to add prefix for each attempt. + return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=False, + timeout=timeout, + ) + else: + start_daemon = rank == 0 +> return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=start_daemon, + timeout=timeout, + multi_tenant=True, + use_libuv=use_libuv, + ) +E torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 8889, useIpv6: false, code: -48, name: EADDRINUSE, message: address already in use + +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:198: DistNetworkError +----------------------------- Captured stdout call ----------------------------- +Distributed data parallel: gloo master at 127.0.0.1:8889 +_______________________ pytest_train_model[ci.json-MACE] _______________________ + +mpnn_type = 'MACE', ci_input = 'ci.json', overwrite_data = False + + @pytest.mark.parametrize( + "mpnn_type", + [ + "SAGE", + "GIN", + "GAT", + "MFC", + "PNA", + "PNAPlus", + "CGCNN", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + "PAINN", + "MACE", + ], + ) + @pytest.mark.parametrize("ci_input", ["ci.json", "ci_multihead.json"]) + def pytest_train_model(mpnn_type, ci_input, overwrite_data=False): +> unittest_train_model(mpnn_type, None, None, ci_input, False, overwrite_data) + +tests/test_graphs.py:223: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_graphs.py:134: in unittest_train_model + hydragnn.run_training(config, use_deepspeed) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:80: in _ + world_size, world_rank = setup_ddp(use_deepspeed=use_deepspeed) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/utils/distributed/distributed.py:206: in setup_ddp + dist.init_process_group( +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:81: in wrapper + return func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:95: in wrapper + func_return = func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/distributed_c10d.py:1757: in init_process_group + store, rank, world_size = next(rendezvous_iterator) + ^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:278: in _env_rendezvous_handler + store = _create_c10d_store( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +hostname = '127.0.0.1', port = 8889, rank = 0, world_size = 1 +timeout = datetime.timedelta(seconds=1800), use_libuv = True + + def _create_c10d_store( + hostname, port, rank, world_size, timeout, use_libuv=True + ) -> Store: + """ + Smartly creates a c10d Store object on ``rank`` based on whether we need to reuse agent store. + + The TCPStore server is assumed to be hosted + on ``hostname:port``. + + By default, the TCPStore server uses the asynchronous implementation + ``LibUVStoreDaemon`` which utilizes libuv. + + If ``torchelastic_use_agent_store()`` is ``True``, then it is assumed that + the agent leader (node rank 0) hosts the TCPStore server (for which the + endpoint is specified by the given ``hostname:port``). Hence + ALL ranks will create and return a TCPStore client (e.g. ``start_daemon=False``). + + If ``torchelastic_use_agent_store()`` is ``False``, then rank 0 will host + the TCPStore (with multi-tenancy) and it is assumed that rank 0's hostname + and port are correctly passed via ``hostname`` and ``port``. All + non-zero ranks will create and return a TCPStore client. + """ + # check if port is uint16_t + if not 0 <= port < 2**16: + raise ValueError(f"port must have value from 0 to 65535 but was {port}.") + + if _torchelastic_use_agent_store(): + # We create a new TCPStore for every retry so no need to add prefix for each attempt. + return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=False, + timeout=timeout, + ) + else: + start_daemon = rank == 0 +> return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=start_daemon, + timeout=timeout, + multi_tenant=True, + use_libuv=use_libuv, + ) +E torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 8889, useIpv6: false, code: -48, name: EADDRINUSE, message: address already in use + +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:198: DistNetworkError +----------------------------- Captured stdout call ----------------------------- +Distributed data parallel: gloo master at 127.0.0.1:8889 +__________________ pytest_train_model[ci_multihead.json-SAGE] __________________ + +mpnn_type = 'SAGE', ci_input = 'ci_multihead.json', overwrite_data = False + + @pytest.mark.parametrize( + "mpnn_type", + [ + "SAGE", + "GIN", + "GAT", + "MFC", + "PNA", + "PNAPlus", + "CGCNN", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + "PAINN", + "MACE", + ], + ) + @pytest.mark.parametrize("ci_input", ["ci.json", "ci_multihead.json"]) + def pytest_train_model(mpnn_type, ci_input, overwrite_data=False): +> unittest_train_model(mpnn_type, None, None, ci_input, False, overwrite_data) + +tests/test_graphs.py:223: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_graphs.py:134: in unittest_train_model + hydragnn.run_training(config, use_deepspeed) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:80: in _ + world_size, world_rank = setup_ddp(use_deepspeed=use_deepspeed) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/utils/distributed/distributed.py:206: in setup_ddp + dist.init_process_group( +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:81: in wrapper + return func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:95: in wrapper + func_return = func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/distributed_c10d.py:1757: in init_process_group + store, rank, world_size = next(rendezvous_iterator) + ^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:278: in _env_rendezvous_handler + store = _create_c10d_store( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +hostname = '127.0.0.1', port = 8889, rank = 0, world_size = 1 +timeout = datetime.timedelta(seconds=1800), use_libuv = True + + def _create_c10d_store( + hostname, port, rank, world_size, timeout, use_libuv=True + ) -> Store: + """ + Smartly creates a c10d Store object on ``rank`` based on whether we need to reuse agent store. + + The TCPStore server is assumed to be hosted + on ``hostname:port``. + + By default, the TCPStore server uses the asynchronous implementation + ``LibUVStoreDaemon`` which utilizes libuv. + + If ``torchelastic_use_agent_store()`` is ``True``, then it is assumed that + the agent leader (node rank 0) hosts the TCPStore server (for which the + endpoint is specified by the given ``hostname:port``). Hence + ALL ranks will create and return a TCPStore client (e.g. ``start_daemon=False``). + + If ``torchelastic_use_agent_store()`` is ``False``, then rank 0 will host + the TCPStore (with multi-tenancy) and it is assumed that rank 0's hostname + and port are correctly passed via ``hostname`` and ``port``. All + non-zero ranks will create and return a TCPStore client. + """ + # check if port is uint16_t + if not 0 <= port < 2**16: + raise ValueError(f"port must have value from 0 to 65535 but was {port}.") + + if _torchelastic_use_agent_store(): + # We create a new TCPStore for every retry so no need to add prefix for each attempt. + return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=False, + timeout=timeout, + ) + else: + start_daemon = rank == 0 +> return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=start_daemon, + timeout=timeout, + multi_tenant=True, + use_libuv=use_libuv, + ) +E torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 8889, useIpv6: false, code: -48, name: EADDRINUSE, message: address already in use + +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:198: DistNetworkError +----------------------------- Captured stdout call ----------------------------- +Distributed data parallel: gloo master at 127.0.0.1:8889 +__________________ pytest_train_model[ci_multihead.json-GIN] ___________________ + +mpnn_type = 'GIN', ci_input = 'ci_multihead.json', overwrite_data = False + + @pytest.mark.parametrize( + "mpnn_type", + [ + "SAGE", + "GIN", + "GAT", + "MFC", + "PNA", + "PNAPlus", + "CGCNN", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + "PAINN", + "MACE", + ], + ) + @pytest.mark.parametrize("ci_input", ["ci.json", "ci_multihead.json"]) + def pytest_train_model(mpnn_type, ci_input, overwrite_data=False): +> unittest_train_model(mpnn_type, None, None, ci_input, False, overwrite_data) + +tests/test_graphs.py:223: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_graphs.py:134: in unittest_train_model + hydragnn.run_training(config, use_deepspeed) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:80: in _ + world_size, world_rank = setup_ddp(use_deepspeed=use_deepspeed) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/utils/distributed/distributed.py:206: in setup_ddp + dist.init_process_group( +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:81: in wrapper + return func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:95: in wrapper + func_return = func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/distributed_c10d.py:1757: in init_process_group + store, rank, world_size = next(rendezvous_iterator) + ^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:278: in _env_rendezvous_handler + store = _create_c10d_store( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +hostname = '127.0.0.1', port = 8889, rank = 0, world_size = 1 +timeout = datetime.timedelta(seconds=1800), use_libuv = True + + def _create_c10d_store( + hostname, port, rank, world_size, timeout, use_libuv=True + ) -> Store: + """ + Smartly creates a c10d Store object on ``rank`` based on whether we need to reuse agent store. + + The TCPStore server is assumed to be hosted + on ``hostname:port``. + + By default, the TCPStore server uses the asynchronous implementation + ``LibUVStoreDaemon`` which utilizes libuv. + + If ``torchelastic_use_agent_store()`` is ``True``, then it is assumed that + the agent leader (node rank 0) hosts the TCPStore server (for which the + endpoint is specified by the given ``hostname:port``). Hence + ALL ranks will create and return a TCPStore client (e.g. ``start_daemon=False``). + + If ``torchelastic_use_agent_store()`` is ``False``, then rank 0 will host + the TCPStore (with multi-tenancy) and it is assumed that rank 0's hostname + and port are correctly passed via ``hostname`` and ``port``. All + non-zero ranks will create and return a TCPStore client. + """ + # check if port is uint16_t + if not 0 <= port < 2**16: + raise ValueError(f"port must have value from 0 to 65535 but was {port}.") + + if _torchelastic_use_agent_store(): + # We create a new TCPStore for every retry so no need to add prefix for each attempt. + return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=False, + timeout=timeout, + ) + else: + start_daemon = rank == 0 +> return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=start_daemon, + timeout=timeout, + multi_tenant=True, + use_libuv=use_libuv, + ) +E torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 8889, useIpv6: false, code: -48, name: EADDRINUSE, message: address already in use + +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:198: DistNetworkError +----------------------------- Captured stdout call ----------------------------- +Distributed data parallel: gloo master at 127.0.0.1:8889 +__________________ pytest_train_model[ci_multihead.json-GAT] ___________________ + +mpnn_type = 'GAT', ci_input = 'ci_multihead.json', overwrite_data = False + + @pytest.mark.parametrize( + "mpnn_type", + [ + "SAGE", + "GIN", + "GAT", + "MFC", + "PNA", + "PNAPlus", + "CGCNN", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + "PAINN", + "MACE", + ], + ) + @pytest.mark.parametrize("ci_input", ["ci.json", "ci_multihead.json"]) + def pytest_train_model(mpnn_type, ci_input, overwrite_data=False): +> unittest_train_model(mpnn_type, None, None, ci_input, False, overwrite_data) + +tests/test_graphs.py:223: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_graphs.py:134: in unittest_train_model + hydragnn.run_training(config, use_deepspeed) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:80: in _ + world_size, world_rank = setup_ddp(use_deepspeed=use_deepspeed) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/utils/distributed/distributed.py:206: in setup_ddp + dist.init_process_group( +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:81: in wrapper + return func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:95: in wrapper + func_return = func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/distributed_c10d.py:1757: in init_process_group + store, rank, world_size = next(rendezvous_iterator) + ^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:278: in _env_rendezvous_handler + store = _create_c10d_store( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +hostname = '127.0.0.1', port = 8889, rank = 0, world_size = 1 +timeout = datetime.timedelta(seconds=1800), use_libuv = True + + def _create_c10d_store( + hostname, port, rank, world_size, timeout, use_libuv=True + ) -> Store: + """ + Smartly creates a c10d Store object on ``rank`` based on whether we need to reuse agent store. + + The TCPStore server is assumed to be hosted + on ``hostname:port``. + + By default, the TCPStore server uses the asynchronous implementation + ``LibUVStoreDaemon`` which utilizes libuv. + + If ``torchelastic_use_agent_store()`` is ``True``, then it is assumed that + the agent leader (node rank 0) hosts the TCPStore server (for which the + endpoint is specified by the given ``hostname:port``). Hence + ALL ranks will create and return a TCPStore client (e.g. ``start_daemon=False``). + + If ``torchelastic_use_agent_store()`` is ``False``, then rank 0 will host + the TCPStore (with multi-tenancy) and it is assumed that rank 0's hostname + and port are correctly passed via ``hostname`` and ``port``. All + non-zero ranks will create and return a TCPStore client. + """ + # check if port is uint16_t + if not 0 <= port < 2**16: + raise ValueError(f"port must have value from 0 to 65535 but was {port}.") + + if _torchelastic_use_agent_store(): + # We create a new TCPStore for every retry so no need to add prefix for each attempt. + return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=False, + timeout=timeout, + ) + else: + start_daemon = rank == 0 +> return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=start_daemon, + timeout=timeout, + multi_tenant=True, + use_libuv=use_libuv, + ) +E torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 8889, useIpv6: false, code: -48, name: EADDRINUSE, message: address already in use + +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:198: DistNetworkError +----------------------------- Captured stdout call ----------------------------- +Distributed data parallel: gloo master at 127.0.0.1:8889 +__________________ pytest_train_model[ci_multihead.json-MFC] ___________________ + +mpnn_type = 'MFC', ci_input = 'ci_multihead.json', overwrite_data = False + + @pytest.mark.parametrize( + "mpnn_type", + [ + "SAGE", + "GIN", + "GAT", + "MFC", + "PNA", + "PNAPlus", + "CGCNN", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + "PAINN", + "MACE", + ], + ) + @pytest.mark.parametrize("ci_input", ["ci.json", "ci_multihead.json"]) + def pytest_train_model(mpnn_type, ci_input, overwrite_data=False): +> unittest_train_model(mpnn_type, None, None, ci_input, False, overwrite_data) + +tests/test_graphs.py:223: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_graphs.py:134: in unittest_train_model + hydragnn.run_training(config, use_deepspeed) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:80: in _ + world_size, world_rank = setup_ddp(use_deepspeed=use_deepspeed) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/utils/distributed/distributed.py:206: in setup_ddp + dist.init_process_group( +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:81: in wrapper + return func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:95: in wrapper + func_return = func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/distributed_c10d.py:1757: in init_process_group + store, rank, world_size = next(rendezvous_iterator) + ^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:278: in _env_rendezvous_handler + store = _create_c10d_store( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +hostname = '127.0.0.1', port = 8889, rank = 0, world_size = 1 +timeout = datetime.timedelta(seconds=1800), use_libuv = True + + def _create_c10d_store( + hostname, port, rank, world_size, timeout, use_libuv=True + ) -> Store: + """ + Smartly creates a c10d Store object on ``rank`` based on whether we need to reuse agent store. + + The TCPStore server is assumed to be hosted + on ``hostname:port``. + + By default, the TCPStore server uses the asynchronous implementation + ``LibUVStoreDaemon`` which utilizes libuv. + + If ``torchelastic_use_agent_store()`` is ``True``, then it is assumed that + the agent leader (node rank 0) hosts the TCPStore server (for which the + endpoint is specified by the given ``hostname:port``). Hence + ALL ranks will create and return a TCPStore client (e.g. ``start_daemon=False``). + + If ``torchelastic_use_agent_store()`` is ``False``, then rank 0 will host + the TCPStore (with multi-tenancy) and it is assumed that rank 0's hostname + and port are correctly passed via ``hostname`` and ``port``. All + non-zero ranks will create and return a TCPStore client. + """ + # check if port is uint16_t + if not 0 <= port < 2**16: + raise ValueError(f"port must have value from 0 to 65535 but was {port}.") + + if _torchelastic_use_agent_store(): + # We create a new TCPStore for every retry so no need to add prefix for each attempt. + return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=False, + timeout=timeout, + ) + else: + start_daemon = rank == 0 +> return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=start_daemon, + timeout=timeout, + multi_tenant=True, + use_libuv=use_libuv, + ) +E torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 8889, useIpv6: false, code: -48, name: EADDRINUSE, message: address already in use + +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:198: DistNetworkError +----------------------------- Captured stdout call ----------------------------- +Distributed data parallel: gloo master at 127.0.0.1:8889 +__________________ pytest_train_model[ci_multihead.json-PNA] ___________________ + +mpnn_type = 'PNA', ci_input = 'ci_multihead.json', overwrite_data = False + + @pytest.mark.parametrize( + "mpnn_type", + [ + "SAGE", + "GIN", + "GAT", + "MFC", + "PNA", + "PNAPlus", + "CGCNN", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + "PAINN", + "MACE", + ], + ) + @pytest.mark.parametrize("ci_input", ["ci.json", "ci_multihead.json"]) + def pytest_train_model(mpnn_type, ci_input, overwrite_data=False): +> unittest_train_model(mpnn_type, None, None, ci_input, False, overwrite_data) + +tests/test_graphs.py:223: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_graphs.py:132: in unittest_train_model + hydragnn.run_training(config_file, use_deepspeed) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:62: in _ + run_training(config) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:80: in _ + world_size, world_rank = setup_ddp(use_deepspeed=use_deepspeed) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/utils/distributed/distributed.py:206: in setup_ddp + dist.init_process_group( +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:81: in wrapper + return func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:95: in wrapper + func_return = func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/distributed_c10d.py:1757: in init_process_group + store, rank, world_size = next(rendezvous_iterator) + ^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:278: in _env_rendezvous_handler + store = _create_c10d_store( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +hostname = '127.0.0.1', port = 8889, rank = 0, world_size = 1 +timeout = datetime.timedelta(seconds=1800), use_libuv = True + + def _create_c10d_store( + hostname, port, rank, world_size, timeout, use_libuv=True + ) -> Store: + """ + Smartly creates a c10d Store object on ``rank`` based on whether we need to reuse agent store. + + The TCPStore server is assumed to be hosted + on ``hostname:port``. + + By default, the TCPStore server uses the asynchronous implementation + ``LibUVStoreDaemon`` which utilizes libuv. + + If ``torchelastic_use_agent_store()`` is ``True``, then it is assumed that + the agent leader (node rank 0) hosts the TCPStore server (for which the + endpoint is specified by the given ``hostname:port``). Hence + ALL ranks will create and return a TCPStore client (e.g. ``start_daemon=False``). + + If ``torchelastic_use_agent_store()`` is ``False``, then rank 0 will host + the TCPStore (with multi-tenancy) and it is assumed that rank 0's hostname + and port are correctly passed via ``hostname`` and ``port``. All + non-zero ranks will create and return a TCPStore client. + """ + # check if port is uint16_t + if not 0 <= port < 2**16: + raise ValueError(f"port must have value from 0 to 65535 but was {port}.") + + if _torchelastic_use_agent_store(): + # We create a new TCPStore for every retry so no need to add prefix for each attempt. + return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=False, + timeout=timeout, + ) + else: + start_daemon = rank == 0 +> return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=start_daemon, + timeout=timeout, + multi_tenant=True, + use_libuv=use_libuv, + ) +E torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 8889, useIpv6: false, code: -48, name: EADDRINUSE, message: address already in use + +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:198: DistNetworkError +----------------------------- Captured stdout call ----------------------------- +Distributed data parallel: gloo master at 127.0.0.1:8889 +________________ pytest_train_model[ci_multihead.json-PNAPlus] _________________ + +mpnn_type = 'PNAPlus', ci_input = 'ci_multihead.json', overwrite_data = False + + @pytest.mark.parametrize( + "mpnn_type", + [ + "SAGE", + "GIN", + "GAT", + "MFC", + "PNA", + "PNAPlus", + "CGCNN", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + "PAINN", + "MACE", + ], + ) + @pytest.mark.parametrize("ci_input", ["ci.json", "ci_multihead.json"]) + def pytest_train_model(mpnn_type, ci_input, overwrite_data=False): +> unittest_train_model(mpnn_type, None, None, ci_input, False, overwrite_data) + +tests/test_graphs.py:223: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_graphs.py:134: in unittest_train_model + hydragnn.run_training(config, use_deepspeed) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:80: in _ + world_size, world_rank = setup_ddp(use_deepspeed=use_deepspeed) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/utils/distributed/distributed.py:206: in setup_ddp + dist.init_process_group( +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:81: in wrapper + return func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:95: in wrapper + func_return = func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/distributed_c10d.py:1757: in init_process_group + store, rank, world_size = next(rendezvous_iterator) + ^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:278: in _env_rendezvous_handler + store = _create_c10d_store( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +hostname = '127.0.0.1', port = 8889, rank = 0, world_size = 1 +timeout = datetime.timedelta(seconds=1800), use_libuv = True + + def _create_c10d_store( + hostname, port, rank, world_size, timeout, use_libuv=True + ) -> Store: + """ + Smartly creates a c10d Store object on ``rank`` based on whether we need to reuse agent store. + + The TCPStore server is assumed to be hosted + on ``hostname:port``. + + By default, the TCPStore server uses the asynchronous implementation + ``LibUVStoreDaemon`` which utilizes libuv. + + If ``torchelastic_use_agent_store()`` is ``True``, then it is assumed that + the agent leader (node rank 0) hosts the TCPStore server (for which the + endpoint is specified by the given ``hostname:port``). Hence + ALL ranks will create and return a TCPStore client (e.g. ``start_daemon=False``). + + If ``torchelastic_use_agent_store()`` is ``False``, then rank 0 will host + the TCPStore (with multi-tenancy) and it is assumed that rank 0's hostname + and port are correctly passed via ``hostname`` and ``port``. All + non-zero ranks will create and return a TCPStore client. + """ + # check if port is uint16_t + if not 0 <= port < 2**16: + raise ValueError(f"port must have value from 0 to 65535 but was {port}.") + + if _torchelastic_use_agent_store(): + # We create a new TCPStore for every retry so no need to add prefix for each attempt. + return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=False, + timeout=timeout, + ) + else: + start_daemon = rank == 0 +> return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=start_daemon, + timeout=timeout, + multi_tenant=True, + use_libuv=use_libuv, + ) +E torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 8889, useIpv6: false, code: -48, name: EADDRINUSE, message: address already in use + +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:198: DistNetworkError +----------------------------- Captured stdout call ----------------------------- +Distributed data parallel: gloo master at 127.0.0.1:8889 +_________________ pytest_train_model[ci_multihead.json-CGCNN] __________________ + +mpnn_type = 'CGCNN', ci_input = 'ci_multihead.json', overwrite_data = False + + @pytest.mark.parametrize( + "mpnn_type", + [ + "SAGE", + "GIN", + "GAT", + "MFC", + "PNA", + "PNAPlus", + "CGCNN", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + "PAINN", + "MACE", + ], + ) + @pytest.mark.parametrize("ci_input", ["ci.json", "ci_multihead.json"]) + def pytest_train_model(mpnn_type, ci_input, overwrite_data=False): +> unittest_train_model(mpnn_type, None, None, ci_input, False, overwrite_data) + +tests/test_graphs.py:223: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_graphs.py:134: in unittest_train_model + hydragnn.run_training(config, use_deepspeed) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:80: in _ + world_size, world_rank = setup_ddp(use_deepspeed=use_deepspeed) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/utils/distributed/distributed.py:206: in setup_ddp + dist.init_process_group( +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:81: in wrapper + return func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:95: in wrapper + func_return = func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/distributed_c10d.py:1757: in init_process_group + store, rank, world_size = next(rendezvous_iterator) + ^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:278: in _env_rendezvous_handler + store = _create_c10d_store( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +hostname = '127.0.0.1', port = 8889, rank = 0, world_size = 1 +timeout = datetime.timedelta(seconds=1800), use_libuv = True + + def _create_c10d_store( + hostname, port, rank, world_size, timeout, use_libuv=True + ) -> Store: + """ + Smartly creates a c10d Store object on ``rank`` based on whether we need to reuse agent store. + + The TCPStore server is assumed to be hosted + on ``hostname:port``. + + By default, the TCPStore server uses the asynchronous implementation + ``LibUVStoreDaemon`` which utilizes libuv. + + If ``torchelastic_use_agent_store()`` is ``True``, then it is assumed that + the agent leader (node rank 0) hosts the TCPStore server (for which the + endpoint is specified by the given ``hostname:port``). Hence + ALL ranks will create and return a TCPStore client (e.g. ``start_daemon=False``). + + If ``torchelastic_use_agent_store()`` is ``False``, then rank 0 will host + the TCPStore (with multi-tenancy) and it is assumed that rank 0's hostname + and port are correctly passed via ``hostname`` and ``port``. All + non-zero ranks will create and return a TCPStore client. + """ + # check if port is uint16_t + if not 0 <= port < 2**16: + raise ValueError(f"port must have value from 0 to 65535 but was {port}.") + + if _torchelastic_use_agent_store(): + # We create a new TCPStore for every retry so no need to add prefix for each attempt. + return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=False, + timeout=timeout, + ) + else: + start_daemon = rank == 0 +> return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=start_daemon, + timeout=timeout, + multi_tenant=True, + use_libuv=use_libuv, + ) +E torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 8889, useIpv6: false, code: -48, name: EADDRINUSE, message: address already in use + +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:198: DistNetworkError +----------------------------- Captured stdout call ----------------------------- +Distributed data parallel: gloo master at 127.0.0.1:8889 +_________________ pytest_train_model[ci_multihead.json-SchNet] _________________ + +mpnn_type = 'SchNet', ci_input = 'ci_multihead.json', overwrite_data = False + + @pytest.mark.parametrize( + "mpnn_type", + [ + "SAGE", + "GIN", + "GAT", + "MFC", + "PNA", + "PNAPlus", + "CGCNN", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + "PAINN", + "MACE", + ], + ) + @pytest.mark.parametrize("ci_input", ["ci.json", "ci_multihead.json"]) + def pytest_train_model(mpnn_type, ci_input, overwrite_data=False): +> unittest_train_model(mpnn_type, None, None, ci_input, False, overwrite_data) + +tests/test_graphs.py:223: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_graphs.py:134: in unittest_train_model + hydragnn.run_training(config, use_deepspeed) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:80: in _ + world_size, world_rank = setup_ddp(use_deepspeed=use_deepspeed) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/utils/distributed/distributed.py:206: in setup_ddp + dist.init_process_group( +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:81: in wrapper + return func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:95: in wrapper + func_return = func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/distributed_c10d.py:1757: in init_process_group + store, rank, world_size = next(rendezvous_iterator) + ^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:278: in _env_rendezvous_handler + store = _create_c10d_store( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +hostname = '127.0.0.1', port = 8889, rank = 0, world_size = 1 +timeout = datetime.timedelta(seconds=1800), use_libuv = True + + def _create_c10d_store( + hostname, port, rank, world_size, timeout, use_libuv=True + ) -> Store: + """ + Smartly creates a c10d Store object on ``rank`` based on whether we need to reuse agent store. + + The TCPStore server is assumed to be hosted + on ``hostname:port``. + + By default, the TCPStore server uses the asynchronous implementation + ``LibUVStoreDaemon`` which utilizes libuv. + + If ``torchelastic_use_agent_store()`` is ``True``, then it is assumed that + the agent leader (node rank 0) hosts the TCPStore server (for which the + endpoint is specified by the given ``hostname:port``). Hence + ALL ranks will create and return a TCPStore client (e.g. ``start_daemon=False``). + + If ``torchelastic_use_agent_store()`` is ``False``, then rank 0 will host + the TCPStore (with multi-tenancy) and it is assumed that rank 0's hostname + and port are correctly passed via ``hostname`` and ``port``. All + non-zero ranks will create and return a TCPStore client. + """ + # check if port is uint16_t + if not 0 <= port < 2**16: + raise ValueError(f"port must have value from 0 to 65535 but was {port}.") + + if _torchelastic_use_agent_store(): + # We create a new TCPStore for every retry so no need to add prefix for each attempt. + return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=False, + timeout=timeout, + ) + else: + start_daemon = rank == 0 +> return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=start_daemon, + timeout=timeout, + multi_tenant=True, + use_libuv=use_libuv, + ) +E torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 8889, useIpv6: false, code: -48, name: EADDRINUSE, message: address already in use + +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:198: DistNetworkError +----------------------------- Captured stdout call ----------------------------- +Distributed data parallel: gloo master at 127.0.0.1:8889 +________________ pytest_train_model[ci_multihead.json-DimeNet] _________________ + +mpnn_type = 'DimeNet', ci_input = 'ci_multihead.json', overwrite_data = False + + @pytest.mark.parametrize( + "mpnn_type", + [ + "SAGE", + "GIN", + "GAT", + "MFC", + "PNA", + "PNAPlus", + "CGCNN", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + "PAINN", + "MACE", + ], + ) + @pytest.mark.parametrize("ci_input", ["ci.json", "ci_multihead.json"]) + def pytest_train_model(mpnn_type, ci_input, overwrite_data=False): +> unittest_train_model(mpnn_type, None, None, ci_input, False, overwrite_data) + +tests/test_graphs.py:223: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_graphs.py:134: in unittest_train_model + hydragnn.run_training(config, use_deepspeed) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:80: in _ + world_size, world_rank = setup_ddp(use_deepspeed=use_deepspeed) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/utils/distributed/distributed.py:206: in setup_ddp + dist.init_process_group( +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:81: in wrapper + return func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:95: in wrapper + func_return = func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/distributed_c10d.py:1757: in init_process_group + store, rank, world_size = next(rendezvous_iterator) + ^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:278: in _env_rendezvous_handler + store = _create_c10d_store( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +hostname = '127.0.0.1', port = 8889, rank = 0, world_size = 1 +timeout = datetime.timedelta(seconds=1800), use_libuv = True + + def _create_c10d_store( + hostname, port, rank, world_size, timeout, use_libuv=True + ) -> Store: + """ + Smartly creates a c10d Store object on ``rank`` based on whether we need to reuse agent store. + + The TCPStore server is assumed to be hosted + on ``hostname:port``. + + By default, the TCPStore server uses the asynchronous implementation + ``LibUVStoreDaemon`` which utilizes libuv. + + If ``torchelastic_use_agent_store()`` is ``True``, then it is assumed that + the agent leader (node rank 0) hosts the TCPStore server (for which the + endpoint is specified by the given ``hostname:port``). Hence + ALL ranks will create and return a TCPStore client (e.g. ``start_daemon=False``). + + If ``torchelastic_use_agent_store()`` is ``False``, then rank 0 will host + the TCPStore (with multi-tenancy) and it is assumed that rank 0's hostname + and port are correctly passed via ``hostname`` and ``port``. All + non-zero ranks will create and return a TCPStore client. + """ + # check if port is uint16_t + if not 0 <= port < 2**16: + raise ValueError(f"port must have value from 0 to 65535 but was {port}.") + + if _torchelastic_use_agent_store(): + # We create a new TCPStore for every retry so no need to add prefix for each attempt. + return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=False, + timeout=timeout, + ) + else: + start_daemon = rank == 0 +> return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=start_daemon, + timeout=timeout, + multi_tenant=True, + use_libuv=use_libuv, + ) +E torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 8889, useIpv6: false, code: -48, name: EADDRINUSE, message: address already in use + +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:198: DistNetworkError +----------------------------- Captured stdout call ----------------------------- +Distributed data parallel: gloo master at 127.0.0.1:8889 +__________________ pytest_train_model[ci_multihead.json-EGNN] __________________ + +mpnn_type = 'EGNN', ci_input = 'ci_multihead.json', overwrite_data = False + + @pytest.mark.parametrize( + "mpnn_type", + [ + "SAGE", + "GIN", + "GAT", + "MFC", + "PNA", + "PNAPlus", + "CGCNN", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + "PAINN", + "MACE", + ], + ) + @pytest.mark.parametrize("ci_input", ["ci.json", "ci_multihead.json"]) + def pytest_train_model(mpnn_type, ci_input, overwrite_data=False): +> unittest_train_model(mpnn_type, None, None, ci_input, False, overwrite_data) + +tests/test_graphs.py:223: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_graphs.py:134: in unittest_train_model + hydragnn.run_training(config, use_deepspeed) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:80: in _ + world_size, world_rank = setup_ddp(use_deepspeed=use_deepspeed) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/utils/distributed/distributed.py:206: in setup_ddp + dist.init_process_group( +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:81: in wrapper + return func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:95: in wrapper + func_return = func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/distributed_c10d.py:1757: in init_process_group + store, rank, world_size = next(rendezvous_iterator) + ^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:278: in _env_rendezvous_handler + store = _create_c10d_store( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +hostname = '127.0.0.1', port = 8889, rank = 0, world_size = 1 +timeout = datetime.timedelta(seconds=1800), use_libuv = True + + def _create_c10d_store( + hostname, port, rank, world_size, timeout, use_libuv=True + ) -> Store: + """ + Smartly creates a c10d Store object on ``rank`` based on whether we need to reuse agent store. + + The TCPStore server is assumed to be hosted + on ``hostname:port``. + + By default, the TCPStore server uses the asynchronous implementation + ``LibUVStoreDaemon`` which utilizes libuv. + + If ``torchelastic_use_agent_store()`` is ``True``, then it is assumed that + the agent leader (node rank 0) hosts the TCPStore server (for which the + endpoint is specified by the given ``hostname:port``). Hence + ALL ranks will create and return a TCPStore client (e.g. ``start_daemon=False``). + + If ``torchelastic_use_agent_store()`` is ``False``, then rank 0 will host + the TCPStore (with multi-tenancy) and it is assumed that rank 0's hostname + and port are correctly passed via ``hostname`` and ``port``. All + non-zero ranks will create and return a TCPStore client. + """ + # check if port is uint16_t + if not 0 <= port < 2**16: + raise ValueError(f"port must have value from 0 to 65535 but was {port}.") + + if _torchelastic_use_agent_store(): + # We create a new TCPStore for every retry so no need to add prefix for each attempt. + return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=False, + timeout=timeout, + ) + else: + start_daemon = rank == 0 +> return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=start_daemon, + timeout=timeout, + multi_tenant=True, + use_libuv=use_libuv, + ) +E torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 8889, useIpv6: false, code: -48, name: EADDRINUSE, message: address already in use + +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:198: DistNetworkError +----------------------------- Captured stdout call ----------------------------- +Distributed data parallel: gloo master at 127.0.0.1:8889 +_________________ pytest_train_model[ci_multihead.json-PNAEq] __________________ + +mpnn_type = 'PNAEq', ci_input = 'ci_multihead.json', overwrite_data = False + + @pytest.mark.parametrize( + "mpnn_type", + [ + "SAGE", + "GIN", + "GAT", + "MFC", + "PNA", + "PNAPlus", + "CGCNN", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + "PAINN", + "MACE", + ], + ) + @pytest.mark.parametrize("ci_input", ["ci.json", "ci_multihead.json"]) + def pytest_train_model(mpnn_type, ci_input, overwrite_data=False): +> unittest_train_model(mpnn_type, None, None, ci_input, False, overwrite_data) + +tests/test_graphs.py:223: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_graphs.py:134: in unittest_train_model + hydragnn.run_training(config, use_deepspeed) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:80: in _ + world_size, world_rank = setup_ddp(use_deepspeed=use_deepspeed) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/utils/distributed/distributed.py:206: in setup_ddp + dist.init_process_group( +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:81: in wrapper + return func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:95: in wrapper + func_return = func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/distributed_c10d.py:1757: in init_process_group + store, rank, world_size = next(rendezvous_iterator) + ^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:278: in _env_rendezvous_handler + store = _create_c10d_store( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +hostname = '127.0.0.1', port = 8889, rank = 0, world_size = 1 +timeout = datetime.timedelta(seconds=1800), use_libuv = True + + def _create_c10d_store( + hostname, port, rank, world_size, timeout, use_libuv=True + ) -> Store: + """ + Smartly creates a c10d Store object on ``rank`` based on whether we need to reuse agent store. + + The TCPStore server is assumed to be hosted + on ``hostname:port``. + + By default, the TCPStore server uses the asynchronous implementation + ``LibUVStoreDaemon`` which utilizes libuv. + + If ``torchelastic_use_agent_store()`` is ``True``, then it is assumed that + the agent leader (node rank 0) hosts the TCPStore server (for which the + endpoint is specified by the given ``hostname:port``). Hence + ALL ranks will create and return a TCPStore client (e.g. ``start_daemon=False``). + + If ``torchelastic_use_agent_store()`` is ``False``, then rank 0 will host + the TCPStore (with multi-tenancy) and it is assumed that rank 0's hostname + and port are correctly passed via ``hostname`` and ``port``. All + non-zero ranks will create and return a TCPStore client. + """ + # check if port is uint16_t + if not 0 <= port < 2**16: + raise ValueError(f"port must have value from 0 to 65535 but was {port}.") + + if _torchelastic_use_agent_store(): + # We create a new TCPStore for every retry so no need to add prefix for each attempt. + return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=False, + timeout=timeout, + ) + else: + start_daemon = rank == 0 +> return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=start_daemon, + timeout=timeout, + multi_tenant=True, + use_libuv=use_libuv, + ) +E torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 8889, useIpv6: false, code: -48, name: EADDRINUSE, message: address already in use + +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:198: DistNetworkError +----------------------------- Captured stdout call ----------------------------- +Distributed data parallel: gloo master at 127.0.0.1:8889 +_________________ pytest_train_model[ci_multihead.json-PAINN] __________________ + +mpnn_type = 'PAINN', ci_input = 'ci_multihead.json', overwrite_data = False + + @pytest.mark.parametrize( + "mpnn_type", + [ + "SAGE", + "GIN", + "GAT", + "MFC", + "PNA", + "PNAPlus", + "CGCNN", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + "PAINN", + "MACE", + ], + ) + @pytest.mark.parametrize("ci_input", ["ci.json", "ci_multihead.json"]) + def pytest_train_model(mpnn_type, ci_input, overwrite_data=False): +> unittest_train_model(mpnn_type, None, None, ci_input, False, overwrite_data) + +tests/test_graphs.py:223: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_graphs.py:134: in unittest_train_model + hydragnn.run_training(config, use_deepspeed) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:80: in _ + world_size, world_rank = setup_ddp(use_deepspeed=use_deepspeed) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/utils/distributed/distributed.py:206: in setup_ddp + dist.init_process_group( +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:81: in wrapper + return func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:95: in wrapper + func_return = func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/distributed_c10d.py:1757: in init_process_group + store, rank, world_size = next(rendezvous_iterator) + ^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:278: in _env_rendezvous_handler + store = _create_c10d_store( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +hostname = '127.0.0.1', port = 8889, rank = 0, world_size = 1 +timeout = datetime.timedelta(seconds=1800), use_libuv = True + + def _create_c10d_store( + hostname, port, rank, world_size, timeout, use_libuv=True + ) -> Store: + """ + Smartly creates a c10d Store object on ``rank`` based on whether we need to reuse agent store. + + The TCPStore server is assumed to be hosted + on ``hostname:port``. + + By default, the TCPStore server uses the asynchronous implementation + ``LibUVStoreDaemon`` which utilizes libuv. + + If ``torchelastic_use_agent_store()`` is ``True``, then it is assumed that + the agent leader (node rank 0) hosts the TCPStore server (for which the + endpoint is specified by the given ``hostname:port``). Hence + ALL ranks will create and return a TCPStore client (e.g. ``start_daemon=False``). + + If ``torchelastic_use_agent_store()`` is ``False``, then rank 0 will host + the TCPStore (with multi-tenancy) and it is assumed that rank 0's hostname + and port are correctly passed via ``hostname`` and ``port``. All + non-zero ranks will create and return a TCPStore client. + """ + # check if port is uint16_t + if not 0 <= port < 2**16: + raise ValueError(f"port must have value from 0 to 65535 but was {port}.") + + if _torchelastic_use_agent_store(): + # We create a new TCPStore for every retry so no need to add prefix for each attempt. + return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=False, + timeout=timeout, + ) + else: + start_daemon = rank == 0 +> return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=start_daemon, + timeout=timeout, + multi_tenant=True, + use_libuv=use_libuv, + ) +E torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 8889, useIpv6: false, code: -48, name: EADDRINUSE, message: address already in use + +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:198: DistNetworkError +----------------------------- Captured stdout call ----------------------------- +Distributed data parallel: gloo master at 127.0.0.1:8889 +__________________ pytest_train_model[ci_multihead.json-MACE] __________________ + +mpnn_type = 'MACE', ci_input = 'ci_multihead.json', overwrite_data = False + + @pytest.mark.parametrize( + "mpnn_type", + [ + "SAGE", + "GIN", + "GAT", + "MFC", + "PNA", + "PNAPlus", + "CGCNN", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + "PAINN", + "MACE", + ], + ) + @pytest.mark.parametrize("ci_input", ["ci.json", "ci_multihead.json"]) + def pytest_train_model(mpnn_type, ci_input, overwrite_data=False): +> unittest_train_model(mpnn_type, None, None, ci_input, False, overwrite_data) + +tests/test_graphs.py:223: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_graphs.py:134: in unittest_train_model + hydragnn.run_training(config, use_deepspeed) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:80: in _ + world_size, world_rank = setup_ddp(use_deepspeed=use_deepspeed) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/utils/distributed/distributed.py:206: in setup_ddp + dist.init_process_group( +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:81: in wrapper + return func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:95: in wrapper + func_return = func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/distributed_c10d.py:1757: in init_process_group + store, rank, world_size = next(rendezvous_iterator) + ^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:278: in _env_rendezvous_handler + store = _create_c10d_store( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +hostname = '127.0.0.1', port = 8889, rank = 0, world_size = 1 +timeout = datetime.timedelta(seconds=1800), use_libuv = True + + def _create_c10d_store( + hostname, port, rank, world_size, timeout, use_libuv=True + ) -> Store: + """ + Smartly creates a c10d Store object on ``rank`` based on whether we need to reuse agent store. + + The TCPStore server is assumed to be hosted + on ``hostname:port``. + + By default, the TCPStore server uses the asynchronous implementation + ``LibUVStoreDaemon`` which utilizes libuv. + + If ``torchelastic_use_agent_store()`` is ``True``, then it is assumed that + the agent leader (node rank 0) hosts the TCPStore server (for which the + endpoint is specified by the given ``hostname:port``). Hence + ALL ranks will create and return a TCPStore client (e.g. ``start_daemon=False``). + + If ``torchelastic_use_agent_store()`` is ``False``, then rank 0 will host + the TCPStore (with multi-tenancy) and it is assumed that rank 0's hostname + and port are correctly passed via ``hostname`` and ``port``. All + non-zero ranks will create and return a TCPStore client. + """ + # check if port is uint16_t + if not 0 <= port < 2**16: + raise ValueError(f"port must have value from 0 to 65535 but was {port}.") + + if _torchelastic_use_agent_store(): + # We create a new TCPStore for every retry so no need to add prefix for each attempt. + return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=False, + timeout=timeout, + ) + else: + start_daemon = rank == 0 +> return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=start_daemon, + timeout=timeout, + multi_tenant=True, + use_libuv=use_libuv, + ) +E torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 8889, useIpv6: false, code: -48, name: EADDRINUSE, message: address already in use + +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:198: DistNetworkError +----------------------------- Captured stdout call ----------------------------- +Distributed data parallel: gloo master at 127.0.0.1:8889 +_______________________ pytest_train_model_lengths[GAT] ________________________ + +mpnn_type = 'GAT', overwrite_data = False + + @pytest.mark.parametrize( + "mpnn_type", + ["GAT", "PNA", "PNAPlus", "CGCNN", "SchNet", "DimeNet", "EGNN", "PNAEq", "PAINN"], + ) + def pytest_train_model_lengths(mpnn_type, overwrite_data=False): +> unittest_train_model(mpnn_type, None, None, "ci.json", True, overwrite_data) + +tests/test_graphs.py:232: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_graphs.py:134: in unittest_train_model + hydragnn.run_training(config, use_deepspeed) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:80: in _ + world_size, world_rank = setup_ddp(use_deepspeed=use_deepspeed) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/utils/distributed/distributed.py:206: in setup_ddp + dist.init_process_group( +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:81: in wrapper + return func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:95: in wrapper + func_return = func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/distributed_c10d.py:1757: in init_process_group + store, rank, world_size = next(rendezvous_iterator) + ^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:278: in _env_rendezvous_handler + store = _create_c10d_store( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +hostname = '127.0.0.1', port = 8889, rank = 0, world_size = 1 +timeout = datetime.timedelta(seconds=1800), use_libuv = True + + def _create_c10d_store( + hostname, port, rank, world_size, timeout, use_libuv=True + ) -> Store: + """ + Smartly creates a c10d Store object on ``rank`` based on whether we need to reuse agent store. + + The TCPStore server is assumed to be hosted + on ``hostname:port``. + + By default, the TCPStore server uses the asynchronous implementation + ``LibUVStoreDaemon`` which utilizes libuv. + + If ``torchelastic_use_agent_store()`` is ``True``, then it is assumed that + the agent leader (node rank 0) hosts the TCPStore server (for which the + endpoint is specified by the given ``hostname:port``). Hence + ALL ranks will create and return a TCPStore client (e.g. ``start_daemon=False``). + + If ``torchelastic_use_agent_store()`` is ``False``, then rank 0 will host + the TCPStore (with multi-tenancy) and it is assumed that rank 0's hostname + and port are correctly passed via ``hostname`` and ``port``. All + non-zero ranks will create and return a TCPStore client. + """ + # check if port is uint16_t + if not 0 <= port < 2**16: + raise ValueError(f"port must have value from 0 to 65535 but was {port}.") + + if _torchelastic_use_agent_store(): + # We create a new TCPStore for every retry so no need to add prefix for each attempt. + return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=False, + timeout=timeout, + ) + else: + start_daemon = rank == 0 +> return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=start_daemon, + timeout=timeout, + multi_tenant=True, + use_libuv=use_libuv, + ) +E torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 8889, useIpv6: false, code: -48, name: EADDRINUSE, message: address already in use + +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:198: DistNetworkError +----------------------------- Captured stdout call ----------------------------- +Distributed data parallel: gloo master at 127.0.0.1:8889 +_______________________ pytest_train_model_lengths[PNA] ________________________ + +mpnn_type = 'PNA', overwrite_data = False + + @pytest.mark.parametrize( + "mpnn_type", + ["GAT", "PNA", "PNAPlus", "CGCNN", "SchNet", "DimeNet", "EGNN", "PNAEq", "PAINN"], + ) + def pytest_train_model_lengths(mpnn_type, overwrite_data=False): +> unittest_train_model(mpnn_type, None, None, "ci.json", True, overwrite_data) + +tests/test_graphs.py:232: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_graphs.py:134: in unittest_train_model + hydragnn.run_training(config, use_deepspeed) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:80: in _ + world_size, world_rank = setup_ddp(use_deepspeed=use_deepspeed) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/utils/distributed/distributed.py:206: in setup_ddp + dist.init_process_group( +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:81: in wrapper + return func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:95: in wrapper + func_return = func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/distributed_c10d.py:1757: in init_process_group + store, rank, world_size = next(rendezvous_iterator) + ^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:278: in _env_rendezvous_handler + store = _create_c10d_store( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +hostname = '127.0.0.1', port = 8889, rank = 0, world_size = 1 +timeout = datetime.timedelta(seconds=1800), use_libuv = True + + def _create_c10d_store( + hostname, port, rank, world_size, timeout, use_libuv=True + ) -> Store: + """ + Smartly creates a c10d Store object on ``rank`` based on whether we need to reuse agent store. + + The TCPStore server is assumed to be hosted + on ``hostname:port``. + + By default, the TCPStore server uses the asynchronous implementation + ``LibUVStoreDaemon`` which utilizes libuv. + + If ``torchelastic_use_agent_store()`` is ``True``, then it is assumed that + the agent leader (node rank 0) hosts the TCPStore server (for which the + endpoint is specified by the given ``hostname:port``). Hence + ALL ranks will create and return a TCPStore client (e.g. ``start_daemon=False``). + + If ``torchelastic_use_agent_store()`` is ``False``, then rank 0 will host + the TCPStore (with multi-tenancy) and it is assumed that rank 0's hostname + and port are correctly passed via ``hostname`` and ``port``. All + non-zero ranks will create and return a TCPStore client. + """ + # check if port is uint16_t + if not 0 <= port < 2**16: + raise ValueError(f"port must have value from 0 to 65535 but was {port}.") + + if _torchelastic_use_agent_store(): + # We create a new TCPStore for every retry so no need to add prefix for each attempt. + return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=False, + timeout=timeout, + ) + else: + start_daemon = rank == 0 +> return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=start_daemon, + timeout=timeout, + multi_tenant=True, + use_libuv=use_libuv, + ) +E torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 8889, useIpv6: false, code: -48, name: EADDRINUSE, message: address already in use + +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:198: DistNetworkError +----------------------------- Captured stdout call ----------------------------- +Distributed data parallel: gloo master at 127.0.0.1:8889 +_____________________ pytest_train_model_lengths[PNAPlus] ______________________ + +mpnn_type = 'PNAPlus', overwrite_data = False + + @pytest.mark.parametrize( + "mpnn_type", + ["GAT", "PNA", "PNAPlus", "CGCNN", "SchNet", "DimeNet", "EGNN", "PNAEq", "PAINN"], + ) + def pytest_train_model_lengths(mpnn_type, overwrite_data=False): +> unittest_train_model(mpnn_type, None, None, "ci.json", True, overwrite_data) + +tests/test_graphs.py:232: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_graphs.py:134: in unittest_train_model + hydragnn.run_training(config, use_deepspeed) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:80: in _ + world_size, world_rank = setup_ddp(use_deepspeed=use_deepspeed) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/utils/distributed/distributed.py:206: in setup_ddp + dist.init_process_group( +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:81: in wrapper + return func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:95: in wrapper + func_return = func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/distributed_c10d.py:1757: in init_process_group + store, rank, world_size = next(rendezvous_iterator) + ^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:278: in _env_rendezvous_handler + store = _create_c10d_store( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +hostname = '127.0.0.1', port = 8889, rank = 0, world_size = 1 +timeout = datetime.timedelta(seconds=1800), use_libuv = True + + def _create_c10d_store( + hostname, port, rank, world_size, timeout, use_libuv=True + ) -> Store: + """ + Smartly creates a c10d Store object on ``rank`` based on whether we need to reuse agent store. + + The TCPStore server is assumed to be hosted + on ``hostname:port``. + + By default, the TCPStore server uses the asynchronous implementation + ``LibUVStoreDaemon`` which utilizes libuv. + + If ``torchelastic_use_agent_store()`` is ``True``, then it is assumed that + the agent leader (node rank 0) hosts the TCPStore server (for which the + endpoint is specified by the given ``hostname:port``). Hence + ALL ranks will create and return a TCPStore client (e.g. ``start_daemon=False``). + + If ``torchelastic_use_agent_store()`` is ``False``, then rank 0 will host + the TCPStore (with multi-tenancy) and it is assumed that rank 0's hostname + and port are correctly passed via ``hostname`` and ``port``. All + non-zero ranks will create and return a TCPStore client. + """ + # check if port is uint16_t + if not 0 <= port < 2**16: + raise ValueError(f"port must have value from 0 to 65535 but was {port}.") + + if _torchelastic_use_agent_store(): + # We create a new TCPStore for every retry so no need to add prefix for each attempt. + return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=False, + timeout=timeout, + ) + else: + start_daemon = rank == 0 +> return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=start_daemon, + timeout=timeout, + multi_tenant=True, + use_libuv=use_libuv, + ) +E torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 8889, useIpv6: false, code: -48, name: EADDRINUSE, message: address already in use + +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:198: DistNetworkError +----------------------------- Captured stdout call ----------------------------- +Distributed data parallel: gloo master at 127.0.0.1:8889 +______________________ pytest_train_model_lengths[CGCNN] _______________________ + +mpnn_type = 'CGCNN', overwrite_data = False + + @pytest.mark.parametrize( + "mpnn_type", + ["GAT", "PNA", "PNAPlus", "CGCNN", "SchNet", "DimeNet", "EGNN", "PNAEq", "PAINN"], + ) + def pytest_train_model_lengths(mpnn_type, overwrite_data=False): +> unittest_train_model(mpnn_type, None, None, "ci.json", True, overwrite_data) + +tests/test_graphs.py:232: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_graphs.py:134: in unittest_train_model + hydragnn.run_training(config, use_deepspeed) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:80: in _ + world_size, world_rank = setup_ddp(use_deepspeed=use_deepspeed) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/utils/distributed/distributed.py:206: in setup_ddp + dist.init_process_group( +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:81: in wrapper + return func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:95: in wrapper + func_return = func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/distributed_c10d.py:1757: in init_process_group + store, rank, world_size = next(rendezvous_iterator) + ^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:278: in _env_rendezvous_handler + store = _create_c10d_store( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +hostname = '127.0.0.1', port = 8889, rank = 0, world_size = 1 +timeout = datetime.timedelta(seconds=1800), use_libuv = True + + def _create_c10d_store( + hostname, port, rank, world_size, timeout, use_libuv=True + ) -> Store: + """ + Smartly creates a c10d Store object on ``rank`` based on whether we need to reuse agent store. + + The TCPStore server is assumed to be hosted + on ``hostname:port``. + + By default, the TCPStore server uses the asynchronous implementation + ``LibUVStoreDaemon`` which utilizes libuv. + + If ``torchelastic_use_agent_store()`` is ``True``, then it is assumed that + the agent leader (node rank 0) hosts the TCPStore server (for which the + endpoint is specified by the given ``hostname:port``). Hence + ALL ranks will create and return a TCPStore client (e.g. ``start_daemon=False``). + + If ``torchelastic_use_agent_store()`` is ``False``, then rank 0 will host + the TCPStore (with multi-tenancy) and it is assumed that rank 0's hostname + and port are correctly passed via ``hostname`` and ``port``. All + non-zero ranks will create and return a TCPStore client. + """ + # check if port is uint16_t + if not 0 <= port < 2**16: + raise ValueError(f"port must have value from 0 to 65535 but was {port}.") + + if _torchelastic_use_agent_store(): + # We create a new TCPStore for every retry so no need to add prefix for each attempt. + return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=False, + timeout=timeout, + ) + else: + start_daemon = rank == 0 +> return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=start_daemon, + timeout=timeout, + multi_tenant=True, + use_libuv=use_libuv, + ) +E torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 8889, useIpv6: false, code: -48, name: EADDRINUSE, message: address already in use + +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:198: DistNetworkError +----------------------------- Captured stdout call ----------------------------- +Distributed data parallel: gloo master at 127.0.0.1:8889 +______________________ pytest_train_model_lengths[SchNet] ______________________ + +mpnn_type = 'SchNet', overwrite_data = False + + @pytest.mark.parametrize( + "mpnn_type", + ["GAT", "PNA", "PNAPlus", "CGCNN", "SchNet", "DimeNet", "EGNN", "PNAEq", "PAINN"], + ) + def pytest_train_model_lengths(mpnn_type, overwrite_data=False): +> unittest_train_model(mpnn_type, None, None, "ci.json", True, overwrite_data) + +tests/test_graphs.py:232: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_graphs.py:134: in unittest_train_model + hydragnn.run_training(config, use_deepspeed) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:80: in _ + world_size, world_rank = setup_ddp(use_deepspeed=use_deepspeed) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/utils/distributed/distributed.py:206: in setup_ddp + dist.init_process_group( +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:81: in wrapper + return func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:95: in wrapper + func_return = func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/distributed_c10d.py:1757: in init_process_group + store, rank, world_size = next(rendezvous_iterator) + ^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:278: in _env_rendezvous_handler + store = _create_c10d_store( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +hostname = '127.0.0.1', port = 8889, rank = 0, world_size = 1 +timeout = datetime.timedelta(seconds=1800), use_libuv = True + + def _create_c10d_store( + hostname, port, rank, world_size, timeout, use_libuv=True + ) -> Store: + """ + Smartly creates a c10d Store object on ``rank`` based on whether we need to reuse agent store. + + The TCPStore server is assumed to be hosted + on ``hostname:port``. + + By default, the TCPStore server uses the asynchronous implementation + ``LibUVStoreDaemon`` which utilizes libuv. + + If ``torchelastic_use_agent_store()`` is ``True``, then it is assumed that + the agent leader (node rank 0) hosts the TCPStore server (for which the + endpoint is specified by the given ``hostname:port``). Hence + ALL ranks will create and return a TCPStore client (e.g. ``start_daemon=False``). + + If ``torchelastic_use_agent_store()`` is ``False``, then rank 0 will host + the TCPStore (with multi-tenancy) and it is assumed that rank 0's hostname + and port are correctly passed via ``hostname`` and ``port``. All + non-zero ranks will create and return a TCPStore client. + """ + # check if port is uint16_t + if not 0 <= port < 2**16: + raise ValueError(f"port must have value from 0 to 65535 but was {port}.") + + if _torchelastic_use_agent_store(): + # We create a new TCPStore for every retry so no need to add prefix for each attempt. + return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=False, + timeout=timeout, + ) + else: + start_daemon = rank == 0 +> return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=start_daemon, + timeout=timeout, + multi_tenant=True, + use_libuv=use_libuv, + ) +E torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 8889, useIpv6: false, code: -48, name: EADDRINUSE, message: address already in use + +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:198: DistNetworkError +----------------------------- Captured stdout call ----------------------------- +Distributed data parallel: gloo master at 127.0.0.1:8889 +_____________________ pytest_train_model_lengths[DimeNet] ______________________ + +mpnn_type = 'DimeNet', overwrite_data = False + + @pytest.mark.parametrize( + "mpnn_type", + ["GAT", "PNA", "PNAPlus", "CGCNN", "SchNet", "DimeNet", "EGNN", "PNAEq", "PAINN"], + ) + def pytest_train_model_lengths(mpnn_type, overwrite_data=False): +> unittest_train_model(mpnn_type, None, None, "ci.json", True, overwrite_data) + +tests/test_graphs.py:232: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_graphs.py:134: in unittest_train_model + hydragnn.run_training(config, use_deepspeed) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:80: in _ + world_size, world_rank = setup_ddp(use_deepspeed=use_deepspeed) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/utils/distributed/distributed.py:206: in setup_ddp + dist.init_process_group( +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:81: in wrapper + return func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:95: in wrapper + func_return = func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/distributed_c10d.py:1757: in init_process_group + store, rank, world_size = next(rendezvous_iterator) + ^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:278: in _env_rendezvous_handler + store = _create_c10d_store( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +hostname = '127.0.0.1', port = 8889, rank = 0, world_size = 1 +timeout = datetime.timedelta(seconds=1800), use_libuv = True + + def _create_c10d_store( + hostname, port, rank, world_size, timeout, use_libuv=True + ) -> Store: + """ + Smartly creates a c10d Store object on ``rank`` based on whether we need to reuse agent store. + + The TCPStore server is assumed to be hosted + on ``hostname:port``. + + By default, the TCPStore server uses the asynchronous implementation + ``LibUVStoreDaemon`` which utilizes libuv. + + If ``torchelastic_use_agent_store()`` is ``True``, then it is assumed that + the agent leader (node rank 0) hosts the TCPStore server (for which the + endpoint is specified by the given ``hostname:port``). Hence + ALL ranks will create and return a TCPStore client (e.g. ``start_daemon=False``). + + If ``torchelastic_use_agent_store()`` is ``False``, then rank 0 will host + the TCPStore (with multi-tenancy) and it is assumed that rank 0's hostname + and port are correctly passed via ``hostname`` and ``port``. All + non-zero ranks will create and return a TCPStore client. + """ + # check if port is uint16_t + if not 0 <= port < 2**16: + raise ValueError(f"port must have value from 0 to 65535 but was {port}.") + + if _torchelastic_use_agent_store(): + # We create a new TCPStore for every retry so no need to add prefix for each attempt. + return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=False, + timeout=timeout, + ) + else: + start_daemon = rank == 0 +> return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=start_daemon, + timeout=timeout, + multi_tenant=True, + use_libuv=use_libuv, + ) +E torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 8889, useIpv6: false, code: -48, name: EADDRINUSE, message: address already in use + +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:198: DistNetworkError +----------------------------- Captured stdout call ----------------------------- +Distributed data parallel: gloo master at 127.0.0.1:8889 +_______________________ pytest_train_model_lengths[EGNN] _______________________ + +mpnn_type = 'EGNN', overwrite_data = False + + @pytest.mark.parametrize( + "mpnn_type", + ["GAT", "PNA", "PNAPlus", "CGCNN", "SchNet", "DimeNet", "EGNN", "PNAEq", "PAINN"], + ) + def pytest_train_model_lengths(mpnn_type, overwrite_data=False): +> unittest_train_model(mpnn_type, None, None, "ci.json", True, overwrite_data) + +tests/test_graphs.py:232: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_graphs.py:134: in unittest_train_model + hydragnn.run_training(config, use_deepspeed) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:80: in _ + world_size, world_rank = setup_ddp(use_deepspeed=use_deepspeed) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/utils/distributed/distributed.py:206: in setup_ddp + dist.init_process_group( +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:81: in wrapper + return func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:95: in wrapper + func_return = func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/distributed_c10d.py:1757: in init_process_group + store, rank, world_size = next(rendezvous_iterator) + ^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:278: in _env_rendezvous_handler + store = _create_c10d_store( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +hostname = '127.0.0.1', port = 8889, rank = 0, world_size = 1 +timeout = datetime.timedelta(seconds=1800), use_libuv = True + + def _create_c10d_store( + hostname, port, rank, world_size, timeout, use_libuv=True + ) -> Store: + """ + Smartly creates a c10d Store object on ``rank`` based on whether we need to reuse agent store. + + The TCPStore server is assumed to be hosted + on ``hostname:port``. + + By default, the TCPStore server uses the asynchronous implementation + ``LibUVStoreDaemon`` which utilizes libuv. + + If ``torchelastic_use_agent_store()`` is ``True``, then it is assumed that + the agent leader (node rank 0) hosts the TCPStore server (for which the + endpoint is specified by the given ``hostname:port``). Hence + ALL ranks will create and return a TCPStore client (e.g. ``start_daemon=False``). + + If ``torchelastic_use_agent_store()`` is ``False``, then rank 0 will host + the TCPStore (with multi-tenancy) and it is assumed that rank 0's hostname + and port are correctly passed via ``hostname`` and ``port``. All + non-zero ranks will create and return a TCPStore client. + """ + # check if port is uint16_t + if not 0 <= port < 2**16: + raise ValueError(f"port must have value from 0 to 65535 but was {port}.") + + if _torchelastic_use_agent_store(): + # We create a new TCPStore for every retry so no need to add prefix for each attempt. + return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=False, + timeout=timeout, + ) + else: + start_daemon = rank == 0 +> return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=start_daemon, + timeout=timeout, + multi_tenant=True, + use_libuv=use_libuv, + ) +E torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 8889, useIpv6: false, code: -48, name: EADDRINUSE, message: address already in use + +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:198: DistNetworkError +----------------------------- Captured stdout call ----------------------------- +Distributed data parallel: gloo master at 127.0.0.1:8889 +______________________ pytest_train_model_lengths[PNAEq] _______________________ + +mpnn_type = 'PNAEq', overwrite_data = False + + @pytest.mark.parametrize( + "mpnn_type", + ["GAT", "PNA", "PNAPlus", "CGCNN", "SchNet", "DimeNet", "EGNN", "PNAEq", "PAINN"], + ) + def pytest_train_model_lengths(mpnn_type, overwrite_data=False): +> unittest_train_model(mpnn_type, None, None, "ci.json", True, overwrite_data) + +tests/test_graphs.py:232: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_graphs.py:134: in unittest_train_model + hydragnn.run_training(config, use_deepspeed) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:80: in _ + world_size, world_rank = setup_ddp(use_deepspeed=use_deepspeed) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/utils/distributed/distributed.py:206: in setup_ddp + dist.init_process_group( +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:81: in wrapper + return func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:95: in wrapper + func_return = func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/distributed_c10d.py:1757: in init_process_group + store, rank, world_size = next(rendezvous_iterator) + ^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:278: in _env_rendezvous_handler + store = _create_c10d_store( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +hostname = '127.0.0.1', port = 8889, rank = 0, world_size = 1 +timeout = datetime.timedelta(seconds=1800), use_libuv = True + + def _create_c10d_store( + hostname, port, rank, world_size, timeout, use_libuv=True + ) -> Store: + """ + Smartly creates a c10d Store object on ``rank`` based on whether we need to reuse agent store. + + The TCPStore server is assumed to be hosted + on ``hostname:port``. + + By default, the TCPStore server uses the asynchronous implementation + ``LibUVStoreDaemon`` which utilizes libuv. + + If ``torchelastic_use_agent_store()`` is ``True``, then it is assumed that + the agent leader (node rank 0) hosts the TCPStore server (for which the + endpoint is specified by the given ``hostname:port``). Hence + ALL ranks will create and return a TCPStore client (e.g. ``start_daemon=False``). + + If ``torchelastic_use_agent_store()`` is ``False``, then rank 0 will host + the TCPStore (with multi-tenancy) and it is assumed that rank 0's hostname + and port are correctly passed via ``hostname`` and ``port``. All + non-zero ranks will create and return a TCPStore client. + """ + # check if port is uint16_t + if not 0 <= port < 2**16: + raise ValueError(f"port must have value from 0 to 65535 but was {port}.") + + if _torchelastic_use_agent_store(): + # We create a new TCPStore for every retry so no need to add prefix for each attempt. + return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=False, + timeout=timeout, + ) + else: + start_daemon = rank == 0 +> return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=start_daemon, + timeout=timeout, + multi_tenant=True, + use_libuv=use_libuv, + ) +E torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 8889, useIpv6: false, code: -48, name: EADDRINUSE, message: address already in use + +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:198: DistNetworkError +----------------------------- Captured stdout call ----------------------------- +Distributed data parallel: gloo master at 127.0.0.1:8889 +______________________ pytest_train_model_lengths[PAINN] _______________________ + +mpnn_type = 'PAINN', overwrite_data = False + + @pytest.mark.parametrize( + "mpnn_type", + ["GAT", "PNA", "PNAPlus", "CGCNN", "SchNet", "DimeNet", "EGNN", "PNAEq", "PAINN"], + ) + def pytest_train_model_lengths(mpnn_type, overwrite_data=False): +> unittest_train_model(mpnn_type, None, None, "ci.json", True, overwrite_data) + +tests/test_graphs.py:232: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_graphs.py:134: in unittest_train_model + hydragnn.run_training(config, use_deepspeed) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:80: in _ + world_size, world_rank = setup_ddp(use_deepspeed=use_deepspeed) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/utils/distributed/distributed.py:206: in setup_ddp + dist.init_process_group( +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:81: in wrapper + return func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:95: in wrapper + func_return = func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/distributed_c10d.py:1757: in init_process_group + store, rank, world_size = next(rendezvous_iterator) + ^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:278: in _env_rendezvous_handler + store = _create_c10d_store( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +hostname = '127.0.0.1', port = 8889, rank = 0, world_size = 1 +timeout = datetime.timedelta(seconds=1800), use_libuv = True + + def _create_c10d_store( + hostname, port, rank, world_size, timeout, use_libuv=True + ) -> Store: + """ + Smartly creates a c10d Store object on ``rank`` based on whether we need to reuse agent store. + + The TCPStore server is assumed to be hosted + on ``hostname:port``. + + By default, the TCPStore server uses the asynchronous implementation + ``LibUVStoreDaemon`` which utilizes libuv. + + If ``torchelastic_use_agent_store()`` is ``True``, then it is assumed that + the agent leader (node rank 0) hosts the TCPStore server (for which the + endpoint is specified by the given ``hostname:port``). Hence + ALL ranks will create and return a TCPStore client (e.g. ``start_daemon=False``). + + If ``torchelastic_use_agent_store()`` is ``False``, then rank 0 will host + the TCPStore (with multi-tenancy) and it is assumed that rank 0's hostname + and port are correctly passed via ``hostname`` and ``port``. All + non-zero ranks will create and return a TCPStore client. + """ + # check if port is uint16_t + if not 0 <= port < 2**16: + raise ValueError(f"port must have value from 0 to 65535 but was {port}.") + + if _torchelastic_use_agent_store(): + # We create a new TCPStore for every retry so no need to add prefix for each attempt. + return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=False, + timeout=timeout, + ) + else: + start_daemon = rank == 0 +> return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=start_daemon, + timeout=timeout, + multi_tenant=True, + use_libuv=use_libuv, + ) +E torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 8889, useIpv6: false, code: -48, name: EADDRINUSE, message: address already in use + +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:198: DistNetworkError +----------------------------- Captured stdout call ----------------------------- +Distributed data parallel: gloo master at 127.0.0.1:8889 +________ pytest_train_model_lengths_global_attention[GAT-multihead-GPS] ________ + +mpnn_type = 'GAT', global_attn_engine = 'GPS', global_attn_type = 'multihead' +overwrite_data = False + + @pytest.mark.parametrize( + "global_attn_engine", + ["GPS"], + ) + @pytest.mark.parametrize("global_attn_type", ["multihead"]) + @pytest.mark.parametrize( + "mpnn_type", + ["GAT", "PNA", "PNAPlus", "CGCNN", "SchNet", "DimeNet", "EGNN", "PNAEq", "PAINN"], + ) + def pytest_train_model_lengths_global_attention( + mpnn_type, global_attn_engine, global_attn_type, overwrite_data=False + ): +> unittest_train_model( + mpnn_type, global_attn_engine, global_attn_type, "ci.json", True, overwrite_data + ) + +tests/test_graphs.py:248: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_graphs.py:134: in unittest_train_model + hydragnn.run_training(config, use_deepspeed) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:80: in _ + world_size, world_rank = setup_ddp(use_deepspeed=use_deepspeed) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/utils/distributed/distributed.py:206: in setup_ddp + dist.init_process_group( +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:81: in wrapper + return func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:95: in wrapper + func_return = func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/distributed_c10d.py:1757: in init_process_group + store, rank, world_size = next(rendezvous_iterator) + ^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:278: in _env_rendezvous_handler + store = _create_c10d_store( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +hostname = '127.0.0.1', port = 8889, rank = 0, world_size = 1 +timeout = datetime.timedelta(seconds=1800), use_libuv = True + + def _create_c10d_store( + hostname, port, rank, world_size, timeout, use_libuv=True + ) -> Store: + """ + Smartly creates a c10d Store object on ``rank`` based on whether we need to reuse agent store. + + The TCPStore server is assumed to be hosted + on ``hostname:port``. + + By default, the TCPStore server uses the asynchronous implementation + ``LibUVStoreDaemon`` which utilizes libuv. + + If ``torchelastic_use_agent_store()`` is ``True``, then it is assumed that + the agent leader (node rank 0) hosts the TCPStore server (for which the + endpoint is specified by the given ``hostname:port``). Hence + ALL ranks will create and return a TCPStore client (e.g. ``start_daemon=False``). + + If ``torchelastic_use_agent_store()`` is ``False``, then rank 0 will host + the TCPStore (with multi-tenancy) and it is assumed that rank 0's hostname + and port are correctly passed via ``hostname`` and ``port``. All + non-zero ranks will create and return a TCPStore client. + """ + # check if port is uint16_t + if not 0 <= port < 2**16: + raise ValueError(f"port must have value from 0 to 65535 but was {port}.") + + if _torchelastic_use_agent_store(): + # We create a new TCPStore for every retry so no need to add prefix for each attempt. + return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=False, + timeout=timeout, + ) + else: + start_daemon = rank == 0 +> return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=start_daemon, + timeout=timeout, + multi_tenant=True, + use_libuv=use_libuv, + ) +E torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 8889, useIpv6: false, code: -48, name: EADDRINUSE, message: address already in use + +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:198: DistNetworkError +----------------------------- Captured stdout call ----------------------------- +Distributed data parallel: gloo master at 127.0.0.1:8889 +________ pytest_train_model_lengths_global_attention[PNA-multihead-GPS] ________ + +mpnn_type = 'PNA', global_attn_engine = 'GPS', global_attn_type = 'multihead' +overwrite_data = False + + @pytest.mark.parametrize( + "global_attn_engine", + ["GPS"], + ) + @pytest.mark.parametrize("global_attn_type", ["multihead"]) + @pytest.mark.parametrize( + "mpnn_type", + ["GAT", "PNA", "PNAPlus", "CGCNN", "SchNet", "DimeNet", "EGNN", "PNAEq", "PAINN"], + ) + def pytest_train_model_lengths_global_attention( + mpnn_type, global_attn_engine, global_attn_type, overwrite_data=False + ): +> unittest_train_model( + mpnn_type, global_attn_engine, global_attn_type, "ci.json", True, overwrite_data + ) + +tests/test_graphs.py:248: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_graphs.py:134: in unittest_train_model + hydragnn.run_training(config, use_deepspeed) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:80: in _ + world_size, world_rank = setup_ddp(use_deepspeed=use_deepspeed) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/utils/distributed/distributed.py:206: in setup_ddp + dist.init_process_group( +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:81: in wrapper + return func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:95: in wrapper + func_return = func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/distributed_c10d.py:1757: in init_process_group + store, rank, world_size = next(rendezvous_iterator) + ^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:278: in _env_rendezvous_handler + store = _create_c10d_store( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +hostname = '127.0.0.1', port = 8889, rank = 0, world_size = 1 +timeout = datetime.timedelta(seconds=1800), use_libuv = True + + def _create_c10d_store( + hostname, port, rank, world_size, timeout, use_libuv=True + ) -> Store: + """ + Smartly creates a c10d Store object on ``rank`` based on whether we need to reuse agent store. + + The TCPStore server is assumed to be hosted + on ``hostname:port``. + + By default, the TCPStore server uses the asynchronous implementation + ``LibUVStoreDaemon`` which utilizes libuv. + + If ``torchelastic_use_agent_store()`` is ``True``, then it is assumed that + the agent leader (node rank 0) hosts the TCPStore server (for which the + endpoint is specified by the given ``hostname:port``). Hence + ALL ranks will create and return a TCPStore client (e.g. ``start_daemon=False``). + + If ``torchelastic_use_agent_store()`` is ``False``, then rank 0 will host + the TCPStore (with multi-tenancy) and it is assumed that rank 0's hostname + and port are correctly passed via ``hostname`` and ``port``. All + non-zero ranks will create and return a TCPStore client. + """ + # check if port is uint16_t + if not 0 <= port < 2**16: + raise ValueError(f"port must have value from 0 to 65535 but was {port}.") + + if _torchelastic_use_agent_store(): + # We create a new TCPStore for every retry so no need to add prefix for each attempt. + return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=False, + timeout=timeout, + ) + else: + start_daemon = rank == 0 +> return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=start_daemon, + timeout=timeout, + multi_tenant=True, + use_libuv=use_libuv, + ) +E torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 8889, useIpv6: false, code: -48, name: EADDRINUSE, message: address already in use + +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:198: DistNetworkError +----------------------------- Captured stdout call ----------------------------- +Distributed data parallel: gloo master at 127.0.0.1:8889 +______ pytest_train_model_lengths_global_attention[PNAPlus-multihead-GPS] ______ + +mpnn_type = 'PNAPlus', global_attn_engine = 'GPS' +global_attn_type = 'multihead', overwrite_data = False + + @pytest.mark.parametrize( + "global_attn_engine", + ["GPS"], + ) + @pytest.mark.parametrize("global_attn_type", ["multihead"]) + @pytest.mark.parametrize( + "mpnn_type", + ["GAT", "PNA", "PNAPlus", "CGCNN", "SchNet", "DimeNet", "EGNN", "PNAEq", "PAINN"], + ) + def pytest_train_model_lengths_global_attention( + mpnn_type, global_attn_engine, global_attn_type, overwrite_data=False + ): +> unittest_train_model( + mpnn_type, global_attn_engine, global_attn_type, "ci.json", True, overwrite_data + ) + +tests/test_graphs.py:248: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_graphs.py:134: in unittest_train_model + hydragnn.run_training(config, use_deepspeed) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:80: in _ + world_size, world_rank = setup_ddp(use_deepspeed=use_deepspeed) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/utils/distributed/distributed.py:206: in setup_ddp + dist.init_process_group( +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:81: in wrapper + return func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:95: in wrapper + func_return = func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/distributed_c10d.py:1757: in init_process_group + store, rank, world_size = next(rendezvous_iterator) + ^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:278: in _env_rendezvous_handler + store = _create_c10d_store( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +hostname = '127.0.0.1', port = 8889, rank = 0, world_size = 1 +timeout = datetime.timedelta(seconds=1800), use_libuv = True + + def _create_c10d_store( + hostname, port, rank, world_size, timeout, use_libuv=True + ) -> Store: + """ + Smartly creates a c10d Store object on ``rank`` based on whether we need to reuse agent store. + + The TCPStore server is assumed to be hosted + on ``hostname:port``. + + By default, the TCPStore server uses the asynchronous implementation + ``LibUVStoreDaemon`` which utilizes libuv. + + If ``torchelastic_use_agent_store()`` is ``True``, then it is assumed that + the agent leader (node rank 0) hosts the TCPStore server (for which the + endpoint is specified by the given ``hostname:port``). Hence + ALL ranks will create and return a TCPStore client (e.g. ``start_daemon=False``). + + If ``torchelastic_use_agent_store()`` is ``False``, then rank 0 will host + the TCPStore (with multi-tenancy) and it is assumed that rank 0's hostname + and port are correctly passed via ``hostname`` and ``port``. All + non-zero ranks will create and return a TCPStore client. + """ + # check if port is uint16_t + if not 0 <= port < 2**16: + raise ValueError(f"port must have value from 0 to 65535 but was {port}.") + + if _torchelastic_use_agent_store(): + # We create a new TCPStore for every retry so no need to add prefix for each attempt. + return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=False, + timeout=timeout, + ) + else: + start_daemon = rank == 0 +> return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=start_daemon, + timeout=timeout, + multi_tenant=True, + use_libuv=use_libuv, + ) +E torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 8889, useIpv6: false, code: -48, name: EADDRINUSE, message: address already in use + +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:198: DistNetworkError +----------------------------- Captured stdout call ----------------------------- +Distributed data parallel: gloo master at 127.0.0.1:8889 +_______ pytest_train_model_lengths_global_attention[CGCNN-multihead-GPS] _______ + +mpnn_type = 'CGCNN', global_attn_engine = 'GPS', global_attn_type = 'multihead' +overwrite_data = False + + @pytest.mark.parametrize( + "global_attn_engine", + ["GPS"], + ) + @pytest.mark.parametrize("global_attn_type", ["multihead"]) + @pytest.mark.parametrize( + "mpnn_type", + ["GAT", "PNA", "PNAPlus", "CGCNN", "SchNet", "DimeNet", "EGNN", "PNAEq", "PAINN"], + ) + def pytest_train_model_lengths_global_attention( + mpnn_type, global_attn_engine, global_attn_type, overwrite_data=False + ): +> unittest_train_model( + mpnn_type, global_attn_engine, global_attn_type, "ci.json", True, overwrite_data + ) + +tests/test_graphs.py:248: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_graphs.py:134: in unittest_train_model + hydragnn.run_training(config, use_deepspeed) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:80: in _ + world_size, world_rank = setup_ddp(use_deepspeed=use_deepspeed) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/utils/distributed/distributed.py:206: in setup_ddp + dist.init_process_group( +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:81: in wrapper + return func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:95: in wrapper + func_return = func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/distributed_c10d.py:1757: in init_process_group + store, rank, world_size = next(rendezvous_iterator) + ^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:278: in _env_rendezvous_handler + store = _create_c10d_store( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +hostname = '127.0.0.1', port = 8889, rank = 0, world_size = 1 +timeout = datetime.timedelta(seconds=1800), use_libuv = True + + def _create_c10d_store( + hostname, port, rank, world_size, timeout, use_libuv=True + ) -> Store: + """ + Smartly creates a c10d Store object on ``rank`` based on whether we need to reuse agent store. + + The TCPStore server is assumed to be hosted + on ``hostname:port``. + + By default, the TCPStore server uses the asynchronous implementation + ``LibUVStoreDaemon`` which utilizes libuv. + + If ``torchelastic_use_agent_store()`` is ``True``, then it is assumed that + the agent leader (node rank 0) hosts the TCPStore server (for which the + endpoint is specified by the given ``hostname:port``). Hence + ALL ranks will create and return a TCPStore client (e.g. ``start_daemon=False``). + + If ``torchelastic_use_agent_store()`` is ``False``, then rank 0 will host + the TCPStore (with multi-tenancy) and it is assumed that rank 0's hostname + and port are correctly passed via ``hostname`` and ``port``. All + non-zero ranks will create and return a TCPStore client. + """ + # check if port is uint16_t + if not 0 <= port < 2**16: + raise ValueError(f"port must have value from 0 to 65535 but was {port}.") + + if _torchelastic_use_agent_store(): + # We create a new TCPStore for every retry so no need to add prefix for each attempt. + return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=False, + timeout=timeout, + ) + else: + start_daemon = rank == 0 +> return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=start_daemon, + timeout=timeout, + multi_tenant=True, + use_libuv=use_libuv, + ) +E torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 8889, useIpv6: false, code: -48, name: EADDRINUSE, message: address already in use + +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:198: DistNetworkError +----------------------------- Captured stdout call ----------------------------- +Distributed data parallel: gloo master at 127.0.0.1:8889 +______ pytest_train_model_lengths_global_attention[SchNet-multihead-GPS] _______ + +mpnn_type = 'SchNet', global_attn_engine = 'GPS', global_attn_type = 'multihead' +overwrite_data = False + + @pytest.mark.parametrize( + "global_attn_engine", + ["GPS"], + ) + @pytest.mark.parametrize("global_attn_type", ["multihead"]) + @pytest.mark.parametrize( + "mpnn_type", + ["GAT", "PNA", "PNAPlus", "CGCNN", "SchNet", "DimeNet", "EGNN", "PNAEq", "PAINN"], + ) + def pytest_train_model_lengths_global_attention( + mpnn_type, global_attn_engine, global_attn_type, overwrite_data=False + ): +> unittest_train_model( + mpnn_type, global_attn_engine, global_attn_type, "ci.json", True, overwrite_data + ) + +tests/test_graphs.py:248: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_graphs.py:134: in unittest_train_model + hydragnn.run_training(config, use_deepspeed) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:80: in _ + world_size, world_rank = setup_ddp(use_deepspeed=use_deepspeed) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/utils/distributed/distributed.py:206: in setup_ddp + dist.init_process_group( +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:81: in wrapper + return func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:95: in wrapper + func_return = func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/distributed_c10d.py:1757: in init_process_group + store, rank, world_size = next(rendezvous_iterator) + ^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:278: in _env_rendezvous_handler + store = _create_c10d_store( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +hostname = '127.0.0.1', port = 8889, rank = 0, world_size = 1 +timeout = datetime.timedelta(seconds=1800), use_libuv = True + + def _create_c10d_store( + hostname, port, rank, world_size, timeout, use_libuv=True + ) -> Store: + """ + Smartly creates a c10d Store object on ``rank`` based on whether we need to reuse agent store. + + The TCPStore server is assumed to be hosted + on ``hostname:port``. + + By default, the TCPStore server uses the asynchronous implementation + ``LibUVStoreDaemon`` which utilizes libuv. + + If ``torchelastic_use_agent_store()`` is ``True``, then it is assumed that + the agent leader (node rank 0) hosts the TCPStore server (for which the + endpoint is specified by the given ``hostname:port``). Hence + ALL ranks will create and return a TCPStore client (e.g. ``start_daemon=False``). + + If ``torchelastic_use_agent_store()`` is ``False``, then rank 0 will host + the TCPStore (with multi-tenancy) and it is assumed that rank 0's hostname + and port are correctly passed via ``hostname`` and ``port``. All + non-zero ranks will create and return a TCPStore client. + """ + # check if port is uint16_t + if not 0 <= port < 2**16: + raise ValueError(f"port must have value from 0 to 65535 but was {port}.") + + if _torchelastic_use_agent_store(): + # We create a new TCPStore for every retry so no need to add prefix for each attempt. + return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=False, + timeout=timeout, + ) + else: + start_daemon = rank == 0 +> return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=start_daemon, + timeout=timeout, + multi_tenant=True, + use_libuv=use_libuv, + ) +E torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 8889, useIpv6: false, code: -48, name: EADDRINUSE, message: address already in use + +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:198: DistNetworkError +----------------------------- Captured stdout call ----------------------------- +Distributed data parallel: gloo master at 127.0.0.1:8889 +______ pytest_train_model_lengths_global_attention[DimeNet-multihead-GPS] ______ + +mpnn_type = 'DimeNet', global_attn_engine = 'GPS' +global_attn_type = 'multihead', overwrite_data = False + + @pytest.mark.parametrize( + "global_attn_engine", + ["GPS"], + ) + @pytest.mark.parametrize("global_attn_type", ["multihead"]) + @pytest.mark.parametrize( + "mpnn_type", + ["GAT", "PNA", "PNAPlus", "CGCNN", "SchNet", "DimeNet", "EGNN", "PNAEq", "PAINN"], + ) + def pytest_train_model_lengths_global_attention( + mpnn_type, global_attn_engine, global_attn_type, overwrite_data=False + ): +> unittest_train_model( + mpnn_type, global_attn_engine, global_attn_type, "ci.json", True, overwrite_data + ) + +tests/test_graphs.py:248: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_graphs.py:134: in unittest_train_model + hydragnn.run_training(config, use_deepspeed) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:80: in _ + world_size, world_rank = setup_ddp(use_deepspeed=use_deepspeed) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/utils/distributed/distributed.py:206: in setup_ddp + dist.init_process_group( +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:81: in wrapper + return func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:95: in wrapper + func_return = func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/distributed_c10d.py:1757: in init_process_group + store, rank, world_size = next(rendezvous_iterator) + ^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:278: in _env_rendezvous_handler + store = _create_c10d_store( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +hostname = '127.0.0.1', port = 8889, rank = 0, world_size = 1 +timeout = datetime.timedelta(seconds=1800), use_libuv = True + + def _create_c10d_store( + hostname, port, rank, world_size, timeout, use_libuv=True + ) -> Store: + """ + Smartly creates a c10d Store object on ``rank`` based on whether we need to reuse agent store. + + The TCPStore server is assumed to be hosted + on ``hostname:port``. + + By default, the TCPStore server uses the asynchronous implementation + ``LibUVStoreDaemon`` which utilizes libuv. + + If ``torchelastic_use_agent_store()`` is ``True``, then it is assumed that + the agent leader (node rank 0) hosts the TCPStore server (for which the + endpoint is specified by the given ``hostname:port``). Hence + ALL ranks will create and return a TCPStore client (e.g. ``start_daemon=False``). + + If ``torchelastic_use_agent_store()`` is ``False``, then rank 0 will host + the TCPStore (with multi-tenancy) and it is assumed that rank 0's hostname + and port are correctly passed via ``hostname`` and ``port``. All + non-zero ranks will create and return a TCPStore client. + """ + # check if port is uint16_t + if not 0 <= port < 2**16: + raise ValueError(f"port must have value from 0 to 65535 but was {port}.") + + if _torchelastic_use_agent_store(): + # We create a new TCPStore for every retry so no need to add prefix for each attempt. + return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=False, + timeout=timeout, + ) + else: + start_daemon = rank == 0 +> return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=start_daemon, + timeout=timeout, + multi_tenant=True, + use_libuv=use_libuv, + ) +E torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 8889, useIpv6: false, code: -48, name: EADDRINUSE, message: address already in use + +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:198: DistNetworkError +----------------------------- Captured stdout call ----------------------------- +Distributed data parallel: gloo master at 127.0.0.1:8889 +_______ pytest_train_model_lengths_global_attention[EGNN-multihead-GPS] ________ + +mpnn_type = 'EGNN', global_attn_engine = 'GPS', global_attn_type = 'multihead' +overwrite_data = False + + @pytest.mark.parametrize( + "global_attn_engine", + ["GPS"], + ) + @pytest.mark.parametrize("global_attn_type", ["multihead"]) + @pytest.mark.parametrize( + "mpnn_type", + ["GAT", "PNA", "PNAPlus", "CGCNN", "SchNet", "DimeNet", "EGNN", "PNAEq", "PAINN"], + ) + def pytest_train_model_lengths_global_attention( + mpnn_type, global_attn_engine, global_attn_type, overwrite_data=False + ): +> unittest_train_model( + mpnn_type, global_attn_engine, global_attn_type, "ci.json", True, overwrite_data + ) + +tests/test_graphs.py:248: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_graphs.py:134: in unittest_train_model + hydragnn.run_training(config, use_deepspeed) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:80: in _ + world_size, world_rank = setup_ddp(use_deepspeed=use_deepspeed) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/utils/distributed/distributed.py:206: in setup_ddp + dist.init_process_group( +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:81: in wrapper + return func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:95: in wrapper + func_return = func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/distributed_c10d.py:1757: in init_process_group + store, rank, world_size = next(rendezvous_iterator) + ^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:278: in _env_rendezvous_handler + store = _create_c10d_store( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +hostname = '127.0.0.1', port = 8889, rank = 0, world_size = 1 +timeout = datetime.timedelta(seconds=1800), use_libuv = True + + def _create_c10d_store( + hostname, port, rank, world_size, timeout, use_libuv=True + ) -> Store: + """ + Smartly creates a c10d Store object on ``rank`` based on whether we need to reuse agent store. + + The TCPStore server is assumed to be hosted + on ``hostname:port``. + + By default, the TCPStore server uses the asynchronous implementation + ``LibUVStoreDaemon`` which utilizes libuv. + + If ``torchelastic_use_agent_store()`` is ``True``, then it is assumed that + the agent leader (node rank 0) hosts the TCPStore server (for which the + endpoint is specified by the given ``hostname:port``). Hence + ALL ranks will create and return a TCPStore client (e.g. ``start_daemon=False``). + + If ``torchelastic_use_agent_store()`` is ``False``, then rank 0 will host + the TCPStore (with multi-tenancy) and it is assumed that rank 0's hostname + and port are correctly passed via ``hostname`` and ``port``. All + non-zero ranks will create and return a TCPStore client. + """ + # check if port is uint16_t + if not 0 <= port < 2**16: + raise ValueError(f"port must have value from 0 to 65535 but was {port}.") + + if _torchelastic_use_agent_store(): + # We create a new TCPStore for every retry so no need to add prefix for each attempt. + return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=False, + timeout=timeout, + ) + else: + start_daemon = rank == 0 +> return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=start_daemon, + timeout=timeout, + multi_tenant=True, + use_libuv=use_libuv, + ) +E torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 8889, useIpv6: false, code: -48, name: EADDRINUSE, message: address already in use + +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:198: DistNetworkError +----------------------------- Captured stdout call ----------------------------- +Distributed data parallel: gloo master at 127.0.0.1:8889 +_______ pytest_train_model_lengths_global_attention[PNAEq-multihead-GPS] _______ + +mpnn_type = 'PNAEq', global_attn_engine = 'GPS', global_attn_type = 'multihead' +overwrite_data = False + + @pytest.mark.parametrize( + "global_attn_engine", + ["GPS"], + ) + @pytest.mark.parametrize("global_attn_type", ["multihead"]) + @pytest.mark.parametrize( + "mpnn_type", + ["GAT", "PNA", "PNAPlus", "CGCNN", "SchNet", "DimeNet", "EGNN", "PNAEq", "PAINN"], + ) + def pytest_train_model_lengths_global_attention( + mpnn_type, global_attn_engine, global_attn_type, overwrite_data=False + ): +> unittest_train_model( + mpnn_type, global_attn_engine, global_attn_type, "ci.json", True, overwrite_data + ) + +tests/test_graphs.py:248: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_graphs.py:134: in unittest_train_model + hydragnn.run_training(config, use_deepspeed) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:80: in _ + world_size, world_rank = setup_ddp(use_deepspeed=use_deepspeed) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/utils/distributed/distributed.py:206: in setup_ddp + dist.init_process_group( +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:81: in wrapper + return func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:95: in wrapper + func_return = func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/distributed_c10d.py:1757: in init_process_group + store, rank, world_size = next(rendezvous_iterator) + ^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:278: in _env_rendezvous_handler + store = _create_c10d_store( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +hostname = '127.0.0.1', port = 8889, rank = 0, world_size = 1 +timeout = datetime.timedelta(seconds=1800), use_libuv = True + + def _create_c10d_store( + hostname, port, rank, world_size, timeout, use_libuv=True + ) -> Store: + """ + Smartly creates a c10d Store object on ``rank`` based on whether we need to reuse agent store. + + The TCPStore server is assumed to be hosted + on ``hostname:port``. + + By default, the TCPStore server uses the asynchronous implementation + ``LibUVStoreDaemon`` which utilizes libuv. + + If ``torchelastic_use_agent_store()`` is ``True``, then it is assumed that + the agent leader (node rank 0) hosts the TCPStore server (for which the + endpoint is specified by the given ``hostname:port``). Hence + ALL ranks will create and return a TCPStore client (e.g. ``start_daemon=False``). + + If ``torchelastic_use_agent_store()`` is ``False``, then rank 0 will host + the TCPStore (with multi-tenancy) and it is assumed that rank 0's hostname + and port are correctly passed via ``hostname`` and ``port``. All + non-zero ranks will create and return a TCPStore client. + """ + # check if port is uint16_t + if not 0 <= port < 2**16: + raise ValueError(f"port must have value from 0 to 65535 but was {port}.") + + if _torchelastic_use_agent_store(): + # We create a new TCPStore for every retry so no need to add prefix for each attempt. + return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=False, + timeout=timeout, + ) + else: + start_daemon = rank == 0 +> return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=start_daemon, + timeout=timeout, + multi_tenant=True, + use_libuv=use_libuv, + ) +E torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 8889, useIpv6: false, code: -48, name: EADDRINUSE, message: address already in use + +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:198: DistNetworkError +----------------------------- Captured stdout call ----------------------------- +Distributed data parallel: gloo master at 127.0.0.1:8889 +_______ pytest_train_model_lengths_global_attention[PAINN-multihead-GPS] _______ + +mpnn_type = 'PAINN', global_attn_engine = 'GPS', global_attn_type = 'multihead' +overwrite_data = False + + @pytest.mark.parametrize( + "global_attn_engine", + ["GPS"], + ) + @pytest.mark.parametrize("global_attn_type", ["multihead"]) + @pytest.mark.parametrize( + "mpnn_type", + ["GAT", "PNA", "PNAPlus", "CGCNN", "SchNet", "DimeNet", "EGNN", "PNAEq", "PAINN"], + ) + def pytest_train_model_lengths_global_attention( + mpnn_type, global_attn_engine, global_attn_type, overwrite_data=False + ): +> unittest_train_model( + mpnn_type, global_attn_engine, global_attn_type, "ci.json", True, overwrite_data + ) + +tests/test_graphs.py:248: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_graphs.py:134: in unittest_train_model + hydragnn.run_training(config, use_deepspeed) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:80: in _ + world_size, world_rank = setup_ddp(use_deepspeed=use_deepspeed) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/utils/distributed/distributed.py:206: in setup_ddp + dist.init_process_group( +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:81: in wrapper + return func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:95: in wrapper + func_return = func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/distributed_c10d.py:1757: in init_process_group + store, rank, world_size = next(rendezvous_iterator) + ^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:278: in _env_rendezvous_handler + store = _create_c10d_store( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +hostname = '127.0.0.1', port = 8889, rank = 0, world_size = 1 +timeout = datetime.timedelta(seconds=1800), use_libuv = True + + def _create_c10d_store( + hostname, port, rank, world_size, timeout, use_libuv=True + ) -> Store: + """ + Smartly creates a c10d Store object on ``rank`` based on whether we need to reuse agent store. + + The TCPStore server is assumed to be hosted + on ``hostname:port``. + + By default, the TCPStore server uses the asynchronous implementation + ``LibUVStoreDaemon`` which utilizes libuv. + + If ``torchelastic_use_agent_store()`` is ``True``, then it is assumed that + the agent leader (node rank 0) hosts the TCPStore server (for which the + endpoint is specified by the given ``hostname:port``). Hence + ALL ranks will create and return a TCPStore client (e.g. ``start_daemon=False``). + + If ``torchelastic_use_agent_store()`` is ``False``, then rank 0 will host + the TCPStore (with multi-tenancy) and it is assumed that rank 0's hostname + and port are correctly passed via ``hostname`` and ``port``. All + non-zero ranks will create and return a TCPStore client. + """ + # check if port is uint16_t + if not 0 <= port < 2**16: + raise ValueError(f"port must have value from 0 to 65535 but was {port}.") + + if _torchelastic_use_agent_store(): + # We create a new TCPStore for every retry so no need to add prefix for each attempt. + return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=False, + timeout=timeout, + ) + else: + start_daemon = rank == 0 +> return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=start_daemon, + timeout=timeout, + multi_tenant=True, + use_libuv=use_libuv, + ) +E torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 8889, useIpv6: false, code: -48, name: EADDRINUSE, message: address already in use + +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:198: DistNetworkError +----------------------------- Captured stdout call ----------------------------- +Distributed data parallel: gloo master at 127.0.0.1:8889 +____________________ pytest_train_mace_model_lengths[MACE] _____________________ + +mpnn_type = 'MACE', overwrite_data = False + + @pytest.mark.parametrize( + "mpnn_type", + ["MACE"], + ) + def pytest_train_mace_model_lengths(mpnn_type, overwrite_data=False): +> unittest_train_model(mpnn_type, None, None, "ci.json", True, overwrite_data) + +tests/test_graphs.py:259: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_graphs.py:134: in unittest_train_model + hydragnn.run_training(config, use_deepspeed) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:80: in _ + world_size, world_rank = setup_ddp(use_deepspeed=use_deepspeed) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/utils/distributed/distributed.py:206: in setup_ddp + dist.init_process_group( +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:81: in wrapper + return func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:95: in wrapper + func_return = func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/distributed_c10d.py:1757: in init_process_group + store, rank, world_size = next(rendezvous_iterator) + ^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:278: in _env_rendezvous_handler + store = _create_c10d_store( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +hostname = '127.0.0.1', port = 8889, rank = 0, world_size = 1 +timeout = datetime.timedelta(seconds=1800), use_libuv = True + + def _create_c10d_store( + hostname, port, rank, world_size, timeout, use_libuv=True + ) -> Store: + """ + Smartly creates a c10d Store object on ``rank`` based on whether we need to reuse agent store. + + The TCPStore server is assumed to be hosted + on ``hostname:port``. + + By default, the TCPStore server uses the asynchronous implementation + ``LibUVStoreDaemon`` which utilizes libuv. + + If ``torchelastic_use_agent_store()`` is ``True``, then it is assumed that + the agent leader (node rank 0) hosts the TCPStore server (for which the + endpoint is specified by the given ``hostname:port``). Hence + ALL ranks will create and return a TCPStore client (e.g. ``start_daemon=False``). + + If ``torchelastic_use_agent_store()`` is ``False``, then rank 0 will host + the TCPStore (with multi-tenancy) and it is assumed that rank 0's hostname + and port are correctly passed via ``hostname`` and ``port``. All + non-zero ranks will create and return a TCPStore client. + """ + # check if port is uint16_t + if not 0 <= port < 2**16: + raise ValueError(f"port must have value from 0 to 65535 but was {port}.") + + if _torchelastic_use_agent_store(): + # We create a new TCPStore for every retry so no need to add prefix for each attempt. + return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=False, + timeout=timeout, + ) + else: + start_daemon = rank == 0 +> return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=start_daemon, + timeout=timeout, + multi_tenant=True, + use_libuv=use_libuv, + ) +E torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 8889, useIpv6: false, code: -48, name: EADDRINUSE, message: address already in use + +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:198: DistNetworkError +----------------------------- Captured stdout call ----------------------------- +Distributed data parallel: gloo master at 127.0.0.1:8889 +_____________________ pytest_train_equivariant_model[EGNN] _____________________ + +mpnn_type = 'EGNN', overwrite_data = False + + @pytest.mark.parametrize("mpnn_type", ["EGNN", "SchNet", "PNAEq", "PAINN", "MACE"]) + def pytest_train_equivariant_model(mpnn_type, overwrite_data=False): +> unittest_train_model( + mpnn_type, None, None, "ci_equivariant.json", False, overwrite_data + ) + +tests/test_graphs.py:265: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_graphs.py:134: in unittest_train_model + hydragnn.run_training(config, use_deepspeed) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:80: in _ + world_size, world_rank = setup_ddp(use_deepspeed=use_deepspeed) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/utils/distributed/distributed.py:206: in setup_ddp + dist.init_process_group( +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:81: in wrapper + return func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:95: in wrapper + func_return = func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/distributed_c10d.py:1757: in init_process_group + store, rank, world_size = next(rendezvous_iterator) + ^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:278: in _env_rendezvous_handler + store = _create_c10d_store( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +hostname = '127.0.0.1', port = 8889, rank = 0, world_size = 1 +timeout = datetime.timedelta(seconds=1800), use_libuv = True + + def _create_c10d_store( + hostname, port, rank, world_size, timeout, use_libuv=True + ) -> Store: + """ + Smartly creates a c10d Store object on ``rank`` based on whether we need to reuse agent store. + + The TCPStore server is assumed to be hosted + on ``hostname:port``. + + By default, the TCPStore server uses the asynchronous implementation + ``LibUVStoreDaemon`` which utilizes libuv. + + If ``torchelastic_use_agent_store()`` is ``True``, then it is assumed that + the agent leader (node rank 0) hosts the TCPStore server (for which the + endpoint is specified by the given ``hostname:port``). Hence + ALL ranks will create and return a TCPStore client (e.g. ``start_daemon=False``). + + If ``torchelastic_use_agent_store()`` is ``False``, then rank 0 will host + the TCPStore (with multi-tenancy) and it is assumed that rank 0's hostname + and port are correctly passed via ``hostname`` and ``port``. All + non-zero ranks will create and return a TCPStore client. + """ + # check if port is uint16_t + if not 0 <= port < 2**16: + raise ValueError(f"port must have value from 0 to 65535 but was {port}.") + + if _torchelastic_use_agent_store(): + # We create a new TCPStore for every retry so no need to add prefix for each attempt. + return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=False, + timeout=timeout, + ) + else: + start_daemon = rank == 0 +> return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=start_daemon, + timeout=timeout, + multi_tenant=True, + use_libuv=use_libuv, + ) +E torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 8889, useIpv6: false, code: -48, name: EADDRINUSE, message: address already in use + +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:198: DistNetworkError +----------------------------- Captured stdout call ----------------------------- +Distributed data parallel: gloo master at 127.0.0.1:8889 +____________________ pytest_train_equivariant_model[SchNet] ____________________ + +mpnn_type = 'SchNet', overwrite_data = False + + @pytest.mark.parametrize("mpnn_type", ["EGNN", "SchNet", "PNAEq", "PAINN", "MACE"]) + def pytest_train_equivariant_model(mpnn_type, overwrite_data=False): +> unittest_train_model( + mpnn_type, None, None, "ci_equivariant.json", False, overwrite_data + ) + +tests/test_graphs.py:265: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_graphs.py:134: in unittest_train_model + hydragnn.run_training(config, use_deepspeed) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:80: in _ + world_size, world_rank = setup_ddp(use_deepspeed=use_deepspeed) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/utils/distributed/distributed.py:206: in setup_ddp + dist.init_process_group( +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:81: in wrapper + return func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:95: in wrapper + func_return = func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/distributed_c10d.py:1757: in init_process_group + store, rank, world_size = next(rendezvous_iterator) + ^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:278: in _env_rendezvous_handler + store = _create_c10d_store( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +hostname = '127.0.0.1', port = 8889, rank = 0, world_size = 1 +timeout = datetime.timedelta(seconds=1800), use_libuv = True + + def _create_c10d_store( + hostname, port, rank, world_size, timeout, use_libuv=True + ) -> Store: + """ + Smartly creates a c10d Store object on ``rank`` based on whether we need to reuse agent store. + + The TCPStore server is assumed to be hosted + on ``hostname:port``. + + By default, the TCPStore server uses the asynchronous implementation + ``LibUVStoreDaemon`` which utilizes libuv. + + If ``torchelastic_use_agent_store()`` is ``True``, then it is assumed that + the agent leader (node rank 0) hosts the TCPStore server (for which the + endpoint is specified by the given ``hostname:port``). Hence + ALL ranks will create and return a TCPStore client (e.g. ``start_daemon=False``). + + If ``torchelastic_use_agent_store()`` is ``False``, then rank 0 will host + the TCPStore (with multi-tenancy) and it is assumed that rank 0's hostname + and port are correctly passed via ``hostname`` and ``port``. All + non-zero ranks will create and return a TCPStore client. + """ + # check if port is uint16_t + if not 0 <= port < 2**16: + raise ValueError(f"port must have value from 0 to 65535 but was {port}.") + + if _torchelastic_use_agent_store(): + # We create a new TCPStore for every retry so no need to add prefix for each attempt. + return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=False, + timeout=timeout, + ) + else: + start_daemon = rank == 0 +> return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=start_daemon, + timeout=timeout, + multi_tenant=True, + use_libuv=use_libuv, + ) +E torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 8889, useIpv6: false, code: -48, name: EADDRINUSE, message: address already in use + +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:198: DistNetworkError +----------------------------- Captured stdout call ----------------------------- +Distributed data parallel: gloo master at 127.0.0.1:8889 +____________________ pytest_train_equivariant_model[PNAEq] _____________________ + +mpnn_type = 'PNAEq', overwrite_data = False + + @pytest.mark.parametrize("mpnn_type", ["EGNN", "SchNet", "PNAEq", "PAINN", "MACE"]) + def pytest_train_equivariant_model(mpnn_type, overwrite_data=False): +> unittest_train_model( + mpnn_type, None, None, "ci_equivariant.json", False, overwrite_data + ) + +tests/test_graphs.py:265: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_graphs.py:134: in unittest_train_model + hydragnn.run_training(config, use_deepspeed) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:80: in _ + world_size, world_rank = setup_ddp(use_deepspeed=use_deepspeed) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/utils/distributed/distributed.py:206: in setup_ddp + dist.init_process_group( +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:81: in wrapper + return func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:95: in wrapper + func_return = func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/distributed_c10d.py:1757: in init_process_group + store, rank, world_size = next(rendezvous_iterator) + ^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:278: in _env_rendezvous_handler + store = _create_c10d_store( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +hostname = '127.0.0.1', port = 8889, rank = 0, world_size = 1 +timeout = datetime.timedelta(seconds=1800), use_libuv = True + + def _create_c10d_store( + hostname, port, rank, world_size, timeout, use_libuv=True + ) -> Store: + """ + Smartly creates a c10d Store object on ``rank`` based on whether we need to reuse agent store. + + The TCPStore server is assumed to be hosted + on ``hostname:port``. + + By default, the TCPStore server uses the asynchronous implementation + ``LibUVStoreDaemon`` which utilizes libuv. + + If ``torchelastic_use_agent_store()`` is ``True``, then it is assumed that + the agent leader (node rank 0) hosts the TCPStore server (for which the + endpoint is specified by the given ``hostname:port``). Hence + ALL ranks will create and return a TCPStore client (e.g. ``start_daemon=False``). + + If ``torchelastic_use_agent_store()`` is ``False``, then rank 0 will host + the TCPStore (with multi-tenancy) and it is assumed that rank 0's hostname + and port are correctly passed via ``hostname`` and ``port``. All + non-zero ranks will create and return a TCPStore client. + """ + # check if port is uint16_t + if not 0 <= port < 2**16: + raise ValueError(f"port must have value from 0 to 65535 but was {port}.") + + if _torchelastic_use_agent_store(): + # We create a new TCPStore for every retry so no need to add prefix for each attempt. + return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=False, + timeout=timeout, + ) + else: + start_daemon = rank == 0 +> return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=start_daemon, + timeout=timeout, + multi_tenant=True, + use_libuv=use_libuv, + ) +E torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 8889, useIpv6: false, code: -48, name: EADDRINUSE, message: address already in use + +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:198: DistNetworkError +----------------------------- Captured stdout call ----------------------------- +Distributed data parallel: gloo master at 127.0.0.1:8889 +____________________ pytest_train_equivariant_model[PAINN] _____________________ + +mpnn_type = 'PAINN', overwrite_data = False + + @pytest.mark.parametrize("mpnn_type", ["EGNN", "SchNet", "PNAEq", "PAINN", "MACE"]) + def pytest_train_equivariant_model(mpnn_type, overwrite_data=False): +> unittest_train_model( + mpnn_type, None, None, "ci_equivariant.json", False, overwrite_data + ) + +tests/test_graphs.py:265: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_graphs.py:134: in unittest_train_model + hydragnn.run_training(config, use_deepspeed) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:80: in _ + world_size, world_rank = setup_ddp(use_deepspeed=use_deepspeed) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/utils/distributed/distributed.py:206: in setup_ddp + dist.init_process_group( +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:81: in wrapper + return func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:95: in wrapper + func_return = func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/distributed_c10d.py:1757: in init_process_group + store, rank, world_size = next(rendezvous_iterator) + ^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:278: in _env_rendezvous_handler + store = _create_c10d_store( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +hostname = '127.0.0.1', port = 8889, rank = 0, world_size = 1 +timeout = datetime.timedelta(seconds=1800), use_libuv = True + + def _create_c10d_store( + hostname, port, rank, world_size, timeout, use_libuv=True + ) -> Store: + """ + Smartly creates a c10d Store object on ``rank`` based on whether we need to reuse agent store. + + The TCPStore server is assumed to be hosted + on ``hostname:port``. + + By default, the TCPStore server uses the asynchronous implementation + ``LibUVStoreDaemon`` which utilizes libuv. + + If ``torchelastic_use_agent_store()`` is ``True``, then it is assumed that + the agent leader (node rank 0) hosts the TCPStore server (for which the + endpoint is specified by the given ``hostname:port``). Hence + ALL ranks will create and return a TCPStore client (e.g. ``start_daemon=False``). + + If ``torchelastic_use_agent_store()`` is ``False``, then rank 0 will host + the TCPStore (with multi-tenancy) and it is assumed that rank 0's hostname + and port are correctly passed via ``hostname`` and ``port``. All + non-zero ranks will create and return a TCPStore client. + """ + # check if port is uint16_t + if not 0 <= port < 2**16: + raise ValueError(f"port must have value from 0 to 65535 but was {port}.") + + if _torchelastic_use_agent_store(): + # We create a new TCPStore for every retry so no need to add prefix for each attempt. + return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=False, + timeout=timeout, + ) + else: + start_daemon = rank == 0 +> return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=start_daemon, + timeout=timeout, + multi_tenant=True, + use_libuv=use_libuv, + ) +E torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 8889, useIpv6: false, code: -48, name: EADDRINUSE, message: address already in use + +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:198: DistNetworkError +----------------------------- Captured stdout call ----------------------------- +Distributed data parallel: gloo master at 127.0.0.1:8889 +_____________________ pytest_train_equivariant_model[MACE] _____________________ + +mpnn_type = 'MACE', overwrite_data = False + + @pytest.mark.parametrize("mpnn_type", ["EGNN", "SchNet", "PNAEq", "PAINN", "MACE"]) + def pytest_train_equivariant_model(mpnn_type, overwrite_data=False): +> unittest_train_model( + mpnn_type, None, None, "ci_equivariant.json", False, overwrite_data + ) + +tests/test_graphs.py:265: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_graphs.py:134: in unittest_train_model + hydragnn.run_training(config, use_deepspeed) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:80: in _ + world_size, world_rank = setup_ddp(use_deepspeed=use_deepspeed) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/utils/distributed/distributed.py:206: in setup_ddp + dist.init_process_group( +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:81: in wrapper + return func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:95: in wrapper + func_return = func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/distributed_c10d.py:1757: in init_process_group + store, rank, world_size = next(rendezvous_iterator) + ^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:278: in _env_rendezvous_handler + store = _create_c10d_store( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +hostname = '127.0.0.1', port = 8889, rank = 0, world_size = 1 +timeout = datetime.timedelta(seconds=1800), use_libuv = True + + def _create_c10d_store( + hostname, port, rank, world_size, timeout, use_libuv=True + ) -> Store: + """ + Smartly creates a c10d Store object on ``rank`` based on whether we need to reuse agent store. + + The TCPStore server is assumed to be hosted + on ``hostname:port``. + + By default, the TCPStore server uses the asynchronous implementation + ``LibUVStoreDaemon`` which utilizes libuv. + + If ``torchelastic_use_agent_store()`` is ``True``, then it is assumed that + the agent leader (node rank 0) hosts the TCPStore server (for which the + endpoint is specified by the given ``hostname:port``). Hence + ALL ranks will create and return a TCPStore client (e.g. ``start_daemon=False``). + + If ``torchelastic_use_agent_store()`` is ``False``, then rank 0 will host + the TCPStore (with multi-tenancy) and it is assumed that rank 0's hostname + and port are correctly passed via ``hostname`` and ``port``. All + non-zero ranks will create and return a TCPStore client. + """ + # check if port is uint16_t + if not 0 <= port < 2**16: + raise ValueError(f"port must have value from 0 to 65535 but was {port}.") + + if _torchelastic_use_agent_store(): + # We create a new TCPStore for every retry so no need to add prefix for each attempt. + return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=False, + timeout=timeout, + ) + else: + start_daemon = rank == 0 +> return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=start_daemon, + timeout=timeout, + multi_tenant=True, + use_libuv=use_libuv, + ) +E torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 8889, useIpv6: false, code: -48, name: EADDRINUSE, message: address already in use + +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:198: DistNetworkError +----------------------------- Captured stdout call ----------------------------- +Distributed data parallel: gloo master at 127.0.0.1:8889 +_____________________ pytest_train_model_vectoroutput[GAT] _____________________ + +mpnn_type = 'GAT', overwrite_data = False + + @pytest.mark.parametrize( + "mpnn_type", + [ + "GAT", + "PNA", + "PNAPlus", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + ], + ) + def pytest_train_model_vectoroutput(mpnn_type, overwrite_data=False): +> unittest_train_model( + mpnn_type, None, None, "ci_vectoroutput.json", True, overwrite_data + ) + +tests/test_graphs.py:284: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_graphs.py:134: in unittest_train_model + hydragnn.run_training(config, use_deepspeed) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:80: in _ + world_size, world_rank = setup_ddp(use_deepspeed=use_deepspeed) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/utils/distributed/distributed.py:206: in setup_ddp + dist.init_process_group( +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:81: in wrapper + return func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:95: in wrapper + func_return = func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/distributed_c10d.py:1757: in init_process_group + store, rank, world_size = next(rendezvous_iterator) + ^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:278: in _env_rendezvous_handler + store = _create_c10d_store( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +hostname = '127.0.0.1', port = 8889, rank = 0, world_size = 1 +timeout = datetime.timedelta(seconds=1800), use_libuv = True + + def _create_c10d_store( + hostname, port, rank, world_size, timeout, use_libuv=True + ) -> Store: + """ + Smartly creates a c10d Store object on ``rank`` based on whether we need to reuse agent store. + + The TCPStore server is assumed to be hosted + on ``hostname:port``. + + By default, the TCPStore server uses the asynchronous implementation + ``LibUVStoreDaemon`` which utilizes libuv. + + If ``torchelastic_use_agent_store()`` is ``True``, then it is assumed that + the agent leader (node rank 0) hosts the TCPStore server (for which the + endpoint is specified by the given ``hostname:port``). Hence + ALL ranks will create and return a TCPStore client (e.g. ``start_daemon=False``). + + If ``torchelastic_use_agent_store()`` is ``False``, then rank 0 will host + the TCPStore (with multi-tenancy) and it is assumed that rank 0's hostname + and port are correctly passed via ``hostname`` and ``port``. All + non-zero ranks will create and return a TCPStore client. + """ + # check if port is uint16_t + if not 0 <= port < 2**16: + raise ValueError(f"port must have value from 0 to 65535 but was {port}.") + + if _torchelastic_use_agent_store(): + # We create a new TCPStore for every retry so no need to add prefix for each attempt. + return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=False, + timeout=timeout, + ) + else: + start_daemon = rank == 0 +> return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=start_daemon, + timeout=timeout, + multi_tenant=True, + use_libuv=use_libuv, + ) +E torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 8889, useIpv6: false, code: -48, name: EADDRINUSE, message: address already in use + +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:198: DistNetworkError +----------------------------- Captured stdout call ----------------------------- +Distributed data parallel: gloo master at 127.0.0.1:8889 +_____________________ pytest_train_model_vectoroutput[PNA] _____________________ + +mpnn_type = 'PNA', overwrite_data = False + + @pytest.mark.parametrize( + "mpnn_type", + [ + "GAT", + "PNA", + "PNAPlus", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + ], + ) + def pytest_train_model_vectoroutput(mpnn_type, overwrite_data=False): +> unittest_train_model( + mpnn_type, None, None, "ci_vectoroutput.json", True, overwrite_data + ) + +tests/test_graphs.py:284: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_graphs.py:134: in unittest_train_model + hydragnn.run_training(config, use_deepspeed) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:80: in _ + world_size, world_rank = setup_ddp(use_deepspeed=use_deepspeed) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/utils/distributed/distributed.py:206: in setup_ddp + dist.init_process_group( +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:81: in wrapper + return func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:95: in wrapper + func_return = func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/distributed_c10d.py:1757: in init_process_group + store, rank, world_size = next(rendezvous_iterator) + ^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:278: in _env_rendezvous_handler + store = _create_c10d_store( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +hostname = '127.0.0.1', port = 8889, rank = 0, world_size = 1 +timeout = datetime.timedelta(seconds=1800), use_libuv = True + + def _create_c10d_store( + hostname, port, rank, world_size, timeout, use_libuv=True + ) -> Store: + """ + Smartly creates a c10d Store object on ``rank`` based on whether we need to reuse agent store. + + The TCPStore server is assumed to be hosted + on ``hostname:port``. + + By default, the TCPStore server uses the asynchronous implementation + ``LibUVStoreDaemon`` which utilizes libuv. + + If ``torchelastic_use_agent_store()`` is ``True``, then it is assumed that + the agent leader (node rank 0) hosts the TCPStore server (for which the + endpoint is specified by the given ``hostname:port``). Hence + ALL ranks will create and return a TCPStore client (e.g. ``start_daemon=False``). + + If ``torchelastic_use_agent_store()`` is ``False``, then rank 0 will host + the TCPStore (with multi-tenancy) and it is assumed that rank 0's hostname + and port are correctly passed via ``hostname`` and ``port``. All + non-zero ranks will create and return a TCPStore client. + """ + # check if port is uint16_t + if not 0 <= port < 2**16: + raise ValueError(f"port must have value from 0 to 65535 but was {port}.") + + if _torchelastic_use_agent_store(): + # We create a new TCPStore for every retry so no need to add prefix for each attempt. + return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=False, + timeout=timeout, + ) + else: + start_daemon = rank == 0 +> return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=start_daemon, + timeout=timeout, + multi_tenant=True, + use_libuv=use_libuv, + ) +E torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 8889, useIpv6: false, code: -48, name: EADDRINUSE, message: address already in use + +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:198: DistNetworkError +----------------------------- Captured stdout call ----------------------------- +Distributed data parallel: gloo master at 127.0.0.1:8889 +___________________ pytest_train_model_vectoroutput[PNAPlus] ___________________ + +mpnn_type = 'PNAPlus', overwrite_data = False + + @pytest.mark.parametrize( + "mpnn_type", + [ + "GAT", + "PNA", + "PNAPlus", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + ], + ) + def pytest_train_model_vectoroutput(mpnn_type, overwrite_data=False): +> unittest_train_model( + mpnn_type, None, None, "ci_vectoroutput.json", True, overwrite_data + ) + +tests/test_graphs.py:284: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_graphs.py:134: in unittest_train_model + hydragnn.run_training(config, use_deepspeed) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:80: in _ + world_size, world_rank = setup_ddp(use_deepspeed=use_deepspeed) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/utils/distributed/distributed.py:206: in setup_ddp + dist.init_process_group( +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:81: in wrapper + return func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:95: in wrapper + func_return = func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/distributed_c10d.py:1757: in init_process_group + store, rank, world_size = next(rendezvous_iterator) + ^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:278: in _env_rendezvous_handler + store = _create_c10d_store( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +hostname = '127.0.0.1', port = 8889, rank = 0, world_size = 1 +timeout = datetime.timedelta(seconds=1800), use_libuv = True + + def _create_c10d_store( + hostname, port, rank, world_size, timeout, use_libuv=True + ) -> Store: + """ + Smartly creates a c10d Store object on ``rank`` based on whether we need to reuse agent store. + + The TCPStore server is assumed to be hosted + on ``hostname:port``. + + By default, the TCPStore server uses the asynchronous implementation + ``LibUVStoreDaemon`` which utilizes libuv. + + If ``torchelastic_use_agent_store()`` is ``True``, then it is assumed that + the agent leader (node rank 0) hosts the TCPStore server (for which the + endpoint is specified by the given ``hostname:port``). Hence + ALL ranks will create and return a TCPStore client (e.g. ``start_daemon=False``). + + If ``torchelastic_use_agent_store()`` is ``False``, then rank 0 will host + the TCPStore (with multi-tenancy) and it is assumed that rank 0's hostname + and port are correctly passed via ``hostname`` and ``port``. All + non-zero ranks will create and return a TCPStore client. + """ + # check if port is uint16_t + if not 0 <= port < 2**16: + raise ValueError(f"port must have value from 0 to 65535 but was {port}.") + + if _torchelastic_use_agent_store(): + # We create a new TCPStore for every retry so no need to add prefix for each attempt. + return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=False, + timeout=timeout, + ) + else: + start_daemon = rank == 0 +> return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=start_daemon, + timeout=timeout, + multi_tenant=True, + use_libuv=use_libuv, + ) +E torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 8889, useIpv6: false, code: -48, name: EADDRINUSE, message: address already in use + +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:198: DistNetworkError +----------------------------- Captured stdout call ----------------------------- +Distributed data parallel: gloo master at 127.0.0.1:8889 +___________________ pytest_train_model_vectoroutput[SchNet] ____________________ + +mpnn_type = 'SchNet', overwrite_data = False + + @pytest.mark.parametrize( + "mpnn_type", + [ + "GAT", + "PNA", + "PNAPlus", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + ], + ) + def pytest_train_model_vectoroutput(mpnn_type, overwrite_data=False): +> unittest_train_model( + mpnn_type, None, None, "ci_vectoroutput.json", True, overwrite_data + ) + +tests/test_graphs.py:284: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_graphs.py:134: in unittest_train_model + hydragnn.run_training(config, use_deepspeed) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:80: in _ + world_size, world_rank = setup_ddp(use_deepspeed=use_deepspeed) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/utils/distributed/distributed.py:206: in setup_ddp + dist.init_process_group( +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:81: in wrapper + return func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:95: in wrapper + func_return = func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/distributed_c10d.py:1757: in init_process_group + store, rank, world_size = next(rendezvous_iterator) + ^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:278: in _env_rendezvous_handler + store = _create_c10d_store( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +hostname = '127.0.0.1', port = 8889, rank = 0, world_size = 1 +timeout = datetime.timedelta(seconds=1800), use_libuv = True + + def _create_c10d_store( + hostname, port, rank, world_size, timeout, use_libuv=True + ) -> Store: + """ + Smartly creates a c10d Store object on ``rank`` based on whether we need to reuse agent store. + + The TCPStore server is assumed to be hosted + on ``hostname:port``. + + By default, the TCPStore server uses the asynchronous implementation + ``LibUVStoreDaemon`` which utilizes libuv. + + If ``torchelastic_use_agent_store()`` is ``True``, then it is assumed that + the agent leader (node rank 0) hosts the TCPStore server (for which the + endpoint is specified by the given ``hostname:port``). Hence + ALL ranks will create and return a TCPStore client (e.g. ``start_daemon=False``). + + If ``torchelastic_use_agent_store()`` is ``False``, then rank 0 will host + the TCPStore (with multi-tenancy) and it is assumed that rank 0's hostname + and port are correctly passed via ``hostname`` and ``port``. All + non-zero ranks will create and return a TCPStore client. + """ + # check if port is uint16_t + if not 0 <= port < 2**16: + raise ValueError(f"port must have value from 0 to 65535 but was {port}.") + + if _torchelastic_use_agent_store(): + # We create a new TCPStore for every retry so no need to add prefix for each attempt. + return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=False, + timeout=timeout, + ) + else: + start_daemon = rank == 0 +> return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=start_daemon, + timeout=timeout, + multi_tenant=True, + use_libuv=use_libuv, + ) +E torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 8889, useIpv6: false, code: -48, name: EADDRINUSE, message: address already in use + +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:198: DistNetworkError +----------------------------- Captured stdout call ----------------------------- +Distributed data parallel: gloo master at 127.0.0.1:8889 +___________________ pytest_train_model_vectoroutput[DimeNet] ___________________ + +mpnn_type = 'DimeNet', overwrite_data = False + + @pytest.mark.parametrize( + "mpnn_type", + [ + "GAT", + "PNA", + "PNAPlus", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + ], + ) + def pytest_train_model_vectoroutput(mpnn_type, overwrite_data=False): +> unittest_train_model( + mpnn_type, None, None, "ci_vectoroutput.json", True, overwrite_data + ) + +tests/test_graphs.py:284: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_graphs.py:134: in unittest_train_model + hydragnn.run_training(config, use_deepspeed) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:80: in _ + world_size, world_rank = setup_ddp(use_deepspeed=use_deepspeed) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/utils/distributed/distributed.py:206: in setup_ddp + dist.init_process_group( +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:81: in wrapper + return func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:95: in wrapper + func_return = func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/distributed_c10d.py:1757: in init_process_group + store, rank, world_size = next(rendezvous_iterator) + ^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:278: in _env_rendezvous_handler + store = _create_c10d_store( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +hostname = '127.0.0.1', port = 8889, rank = 0, world_size = 1 +timeout = datetime.timedelta(seconds=1800), use_libuv = True + + def _create_c10d_store( + hostname, port, rank, world_size, timeout, use_libuv=True + ) -> Store: + """ + Smartly creates a c10d Store object on ``rank`` based on whether we need to reuse agent store. + + The TCPStore server is assumed to be hosted + on ``hostname:port``. + + By default, the TCPStore server uses the asynchronous implementation + ``LibUVStoreDaemon`` which utilizes libuv. + + If ``torchelastic_use_agent_store()`` is ``True``, then it is assumed that + the agent leader (node rank 0) hosts the TCPStore server (for which the + endpoint is specified by the given ``hostname:port``). Hence + ALL ranks will create and return a TCPStore client (e.g. ``start_daemon=False``). + + If ``torchelastic_use_agent_store()`` is ``False``, then rank 0 will host + the TCPStore (with multi-tenancy) and it is assumed that rank 0's hostname + and port are correctly passed via ``hostname`` and ``port``. All + non-zero ranks will create and return a TCPStore client. + """ + # check if port is uint16_t + if not 0 <= port < 2**16: + raise ValueError(f"port must have value from 0 to 65535 but was {port}.") + + if _torchelastic_use_agent_store(): + # We create a new TCPStore for every retry so no need to add prefix for each attempt. + return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=False, + timeout=timeout, + ) + else: + start_daemon = rank == 0 +> return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=start_daemon, + timeout=timeout, + multi_tenant=True, + use_libuv=use_libuv, + ) +E torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 8889, useIpv6: false, code: -48, name: EADDRINUSE, message: address already in use + +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:198: DistNetworkError +----------------------------- Captured stdout call ----------------------------- +Distributed data parallel: gloo master at 127.0.0.1:8889 +____________________ pytest_train_model_vectoroutput[EGNN] _____________________ + +mpnn_type = 'EGNN', overwrite_data = False + + @pytest.mark.parametrize( + "mpnn_type", + [ + "GAT", + "PNA", + "PNAPlus", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + ], + ) + def pytest_train_model_vectoroutput(mpnn_type, overwrite_data=False): +> unittest_train_model( + mpnn_type, None, None, "ci_vectoroutput.json", True, overwrite_data + ) + +tests/test_graphs.py:284: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_graphs.py:134: in unittest_train_model + hydragnn.run_training(config, use_deepspeed) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:80: in _ + world_size, world_rank = setup_ddp(use_deepspeed=use_deepspeed) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/utils/distributed/distributed.py:206: in setup_ddp + dist.init_process_group( +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:81: in wrapper + return func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:95: in wrapper + func_return = func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/distributed_c10d.py:1757: in init_process_group + store, rank, world_size = next(rendezvous_iterator) + ^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:278: in _env_rendezvous_handler + store = _create_c10d_store( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +hostname = '127.0.0.1', port = 8889, rank = 0, world_size = 1 +timeout = datetime.timedelta(seconds=1800), use_libuv = True + + def _create_c10d_store( + hostname, port, rank, world_size, timeout, use_libuv=True + ) -> Store: + """ + Smartly creates a c10d Store object on ``rank`` based on whether we need to reuse agent store. + + The TCPStore server is assumed to be hosted + on ``hostname:port``. + + By default, the TCPStore server uses the asynchronous implementation + ``LibUVStoreDaemon`` which utilizes libuv. + + If ``torchelastic_use_agent_store()`` is ``True``, then it is assumed that + the agent leader (node rank 0) hosts the TCPStore server (for which the + endpoint is specified by the given ``hostname:port``). Hence + ALL ranks will create and return a TCPStore client (e.g. ``start_daemon=False``). + + If ``torchelastic_use_agent_store()`` is ``False``, then rank 0 will host + the TCPStore (with multi-tenancy) and it is assumed that rank 0's hostname + and port are correctly passed via ``hostname`` and ``port``. All + non-zero ranks will create and return a TCPStore client. + """ + # check if port is uint16_t + if not 0 <= port < 2**16: + raise ValueError(f"port must have value from 0 to 65535 but was {port}.") + + if _torchelastic_use_agent_store(): + # We create a new TCPStore for every retry so no need to add prefix for each attempt. + return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=False, + timeout=timeout, + ) + else: + start_daemon = rank == 0 +> return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=start_daemon, + timeout=timeout, + multi_tenant=True, + use_libuv=use_libuv, + ) +E torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 8889, useIpv6: false, code: -48, name: EADDRINUSE, message: address already in use + +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:198: DistNetworkError +----------------------------- Captured stdout call ----------------------------- +Distributed data parallel: gloo master at 127.0.0.1:8889 +____________________ pytest_train_model_vectoroutput[PNAEq] ____________________ + +mpnn_type = 'PNAEq', overwrite_data = False + + @pytest.mark.parametrize( + "mpnn_type", + [ + "GAT", + "PNA", + "PNAPlus", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + ], + ) + def pytest_train_model_vectoroutput(mpnn_type, overwrite_data=False): +> unittest_train_model( + mpnn_type, None, None, "ci_vectoroutput.json", True, overwrite_data + ) + +tests/test_graphs.py:284: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_graphs.py:134: in unittest_train_model + hydragnn.run_training(config, use_deepspeed) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:80: in _ + world_size, world_rank = setup_ddp(use_deepspeed=use_deepspeed) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/utils/distributed/distributed.py:206: in setup_ddp + dist.init_process_group( +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:81: in wrapper + return func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:95: in wrapper + func_return = func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/distributed_c10d.py:1757: in init_process_group + store, rank, world_size = next(rendezvous_iterator) + ^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:278: in _env_rendezvous_handler + store = _create_c10d_store( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +hostname = '127.0.0.1', port = 8889, rank = 0, world_size = 1 +timeout = datetime.timedelta(seconds=1800), use_libuv = True + + def _create_c10d_store( + hostname, port, rank, world_size, timeout, use_libuv=True + ) -> Store: + """ + Smartly creates a c10d Store object on ``rank`` based on whether we need to reuse agent store. + + The TCPStore server is assumed to be hosted + on ``hostname:port``. + + By default, the TCPStore server uses the asynchronous implementation + ``LibUVStoreDaemon`` which utilizes libuv. + + If ``torchelastic_use_agent_store()`` is ``True``, then it is assumed that + the agent leader (node rank 0) hosts the TCPStore server (for which the + endpoint is specified by the given ``hostname:port``). Hence + ALL ranks will create and return a TCPStore client (e.g. ``start_daemon=False``). + + If ``torchelastic_use_agent_store()`` is ``False``, then rank 0 will host + the TCPStore (with multi-tenancy) and it is assumed that rank 0's hostname + and port are correctly passed via ``hostname`` and ``port``. All + non-zero ranks will create and return a TCPStore client. + """ + # check if port is uint16_t + if not 0 <= port < 2**16: + raise ValueError(f"port must have value from 0 to 65535 but was {port}.") + + if _torchelastic_use_agent_store(): + # We create a new TCPStore for every retry so no need to add prefix for each attempt. + return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=False, + timeout=timeout, + ) + else: + start_daemon = rank == 0 +> return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=start_daemon, + timeout=timeout, + multi_tenant=True, + use_libuv=use_libuv, + ) +E torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 8889, useIpv6: false, code: -48, name: EADDRINUSE, message: address already in use + +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:198: DistNetworkError +----------------------------- Captured stdout call ----------------------------- +Distributed data parallel: gloo master at 127.0.0.1:8889 +______________________ pytest_train_model_conv_head[SAGE] ______________________ + +mpnn_type = 'SAGE', overwrite_data = False + + @pytest.mark.parametrize( + "mpnn_type", + [ + "SAGE", + "GIN", + "GAT", + "MFC", + "PNA", + "PNAPlus", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + "PAINN", + ], + ) + def pytest_train_model_conv_head(mpnn_type, overwrite_data=False): +> unittest_train_model( + mpnn_type, None, None, "ci_conv_head.json", False, overwrite_data + ) + +tests/test_graphs.py:306: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_graphs.py:134: in unittest_train_model + hydragnn.run_training(config, use_deepspeed) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:80: in _ + world_size, world_rank = setup_ddp(use_deepspeed=use_deepspeed) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/utils/distributed/distributed.py:206: in setup_ddp + dist.init_process_group( +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:81: in wrapper + return func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:95: in wrapper + func_return = func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/distributed_c10d.py:1757: in init_process_group + store, rank, world_size = next(rendezvous_iterator) + ^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:278: in _env_rendezvous_handler + store = _create_c10d_store( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +hostname = '127.0.0.1', port = 8889, rank = 0, world_size = 1 +timeout = datetime.timedelta(seconds=1800), use_libuv = True + + def _create_c10d_store( + hostname, port, rank, world_size, timeout, use_libuv=True + ) -> Store: + """ + Smartly creates a c10d Store object on ``rank`` based on whether we need to reuse agent store. + + The TCPStore server is assumed to be hosted + on ``hostname:port``. + + By default, the TCPStore server uses the asynchronous implementation + ``LibUVStoreDaemon`` which utilizes libuv. + + If ``torchelastic_use_agent_store()`` is ``True``, then it is assumed that + the agent leader (node rank 0) hosts the TCPStore server (for which the + endpoint is specified by the given ``hostname:port``). Hence + ALL ranks will create and return a TCPStore client (e.g. ``start_daemon=False``). + + If ``torchelastic_use_agent_store()`` is ``False``, then rank 0 will host + the TCPStore (with multi-tenancy) and it is assumed that rank 0's hostname + and port are correctly passed via ``hostname`` and ``port``. All + non-zero ranks will create and return a TCPStore client. + """ + # check if port is uint16_t + if not 0 <= port < 2**16: + raise ValueError(f"port must have value from 0 to 65535 but was {port}.") + + if _torchelastic_use_agent_store(): + # We create a new TCPStore for every retry so no need to add prefix for each attempt. + return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=False, + timeout=timeout, + ) + else: + start_daemon = rank == 0 +> return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=start_daemon, + timeout=timeout, + multi_tenant=True, + use_libuv=use_libuv, + ) +E torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 8889, useIpv6: false, code: -48, name: EADDRINUSE, message: address already in use + +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:198: DistNetworkError +----------------------------- Captured stdout call ----------------------------- +Distributed data parallel: gloo master at 127.0.0.1:8889 +______________________ pytest_train_model_conv_head[GIN] _______________________ + +mpnn_type = 'GIN', overwrite_data = False + + @pytest.mark.parametrize( + "mpnn_type", + [ + "SAGE", + "GIN", + "GAT", + "MFC", + "PNA", + "PNAPlus", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + "PAINN", + ], + ) + def pytest_train_model_conv_head(mpnn_type, overwrite_data=False): +> unittest_train_model( + mpnn_type, None, None, "ci_conv_head.json", False, overwrite_data + ) + +tests/test_graphs.py:306: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_graphs.py:134: in unittest_train_model + hydragnn.run_training(config, use_deepspeed) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:80: in _ + world_size, world_rank = setup_ddp(use_deepspeed=use_deepspeed) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/utils/distributed/distributed.py:206: in setup_ddp + dist.init_process_group( +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:81: in wrapper + return func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:95: in wrapper + func_return = func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/distributed_c10d.py:1757: in init_process_group + store, rank, world_size = next(rendezvous_iterator) + ^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:278: in _env_rendezvous_handler + store = _create_c10d_store( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +hostname = '127.0.0.1', port = 8889, rank = 0, world_size = 1 +timeout = datetime.timedelta(seconds=1800), use_libuv = True + + def _create_c10d_store( + hostname, port, rank, world_size, timeout, use_libuv=True + ) -> Store: + """ + Smartly creates a c10d Store object on ``rank`` based on whether we need to reuse agent store. + + The TCPStore server is assumed to be hosted + on ``hostname:port``. + + By default, the TCPStore server uses the asynchronous implementation + ``LibUVStoreDaemon`` which utilizes libuv. + + If ``torchelastic_use_agent_store()`` is ``True``, then it is assumed that + the agent leader (node rank 0) hosts the TCPStore server (for which the + endpoint is specified by the given ``hostname:port``). Hence + ALL ranks will create and return a TCPStore client (e.g. ``start_daemon=False``). + + If ``torchelastic_use_agent_store()`` is ``False``, then rank 0 will host + the TCPStore (with multi-tenancy) and it is assumed that rank 0's hostname + and port are correctly passed via ``hostname`` and ``port``. All + non-zero ranks will create and return a TCPStore client. + """ + # check if port is uint16_t + if not 0 <= port < 2**16: + raise ValueError(f"port must have value from 0 to 65535 but was {port}.") + + if _torchelastic_use_agent_store(): + # We create a new TCPStore for every retry so no need to add prefix for each attempt. + return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=False, + timeout=timeout, + ) + else: + start_daemon = rank == 0 +> return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=start_daemon, + timeout=timeout, + multi_tenant=True, + use_libuv=use_libuv, + ) +E torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 8889, useIpv6: false, code: -48, name: EADDRINUSE, message: address already in use + +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:198: DistNetworkError +----------------------------- Captured stdout call ----------------------------- +Distributed data parallel: gloo master at 127.0.0.1:8889 +______________________ pytest_train_model_conv_head[GAT] _______________________ + +mpnn_type = 'GAT', overwrite_data = False + + @pytest.mark.parametrize( + "mpnn_type", + [ + "SAGE", + "GIN", + "GAT", + "MFC", + "PNA", + "PNAPlus", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + "PAINN", + ], + ) + def pytest_train_model_conv_head(mpnn_type, overwrite_data=False): +> unittest_train_model( + mpnn_type, None, None, "ci_conv_head.json", False, overwrite_data + ) + +tests/test_graphs.py:306: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_graphs.py:134: in unittest_train_model + hydragnn.run_training(config, use_deepspeed) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:80: in _ + world_size, world_rank = setup_ddp(use_deepspeed=use_deepspeed) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/utils/distributed/distributed.py:206: in setup_ddp + dist.init_process_group( +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:81: in wrapper + return func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:95: in wrapper + func_return = func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/distributed_c10d.py:1757: in init_process_group + store, rank, world_size = next(rendezvous_iterator) + ^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:278: in _env_rendezvous_handler + store = _create_c10d_store( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +hostname = '127.0.0.1', port = 8889, rank = 0, world_size = 1 +timeout = datetime.timedelta(seconds=1800), use_libuv = True + + def _create_c10d_store( + hostname, port, rank, world_size, timeout, use_libuv=True + ) -> Store: + """ + Smartly creates a c10d Store object on ``rank`` based on whether we need to reuse agent store. + + The TCPStore server is assumed to be hosted + on ``hostname:port``. + + By default, the TCPStore server uses the asynchronous implementation + ``LibUVStoreDaemon`` which utilizes libuv. + + If ``torchelastic_use_agent_store()`` is ``True``, then it is assumed that + the agent leader (node rank 0) hosts the TCPStore server (for which the + endpoint is specified by the given ``hostname:port``). Hence + ALL ranks will create and return a TCPStore client (e.g. ``start_daemon=False``). + + If ``torchelastic_use_agent_store()`` is ``False``, then rank 0 will host + the TCPStore (with multi-tenancy) and it is assumed that rank 0's hostname + and port are correctly passed via ``hostname`` and ``port``. All + non-zero ranks will create and return a TCPStore client. + """ + # check if port is uint16_t + if not 0 <= port < 2**16: + raise ValueError(f"port must have value from 0 to 65535 but was {port}.") + + if _torchelastic_use_agent_store(): + # We create a new TCPStore for every retry so no need to add prefix for each attempt. + return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=False, + timeout=timeout, + ) + else: + start_daemon = rank == 0 +> return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=start_daemon, + timeout=timeout, + multi_tenant=True, + use_libuv=use_libuv, + ) +E torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 8889, useIpv6: false, code: -48, name: EADDRINUSE, message: address already in use + +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:198: DistNetworkError +----------------------------- Captured stdout call ----------------------------- +Distributed data parallel: gloo master at 127.0.0.1:8889 +______________________ pytest_train_model_conv_head[MFC] _______________________ + +mpnn_type = 'MFC', overwrite_data = False + + @pytest.mark.parametrize( + "mpnn_type", + [ + "SAGE", + "GIN", + "GAT", + "MFC", + "PNA", + "PNAPlus", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + "PAINN", + ], + ) + def pytest_train_model_conv_head(mpnn_type, overwrite_data=False): +> unittest_train_model( + mpnn_type, None, None, "ci_conv_head.json", False, overwrite_data + ) + +tests/test_graphs.py:306: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_graphs.py:134: in unittest_train_model + hydragnn.run_training(config, use_deepspeed) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:80: in _ + world_size, world_rank = setup_ddp(use_deepspeed=use_deepspeed) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/utils/distributed/distributed.py:206: in setup_ddp + dist.init_process_group( +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:81: in wrapper + return func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:95: in wrapper + func_return = func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/distributed_c10d.py:1757: in init_process_group + store, rank, world_size = next(rendezvous_iterator) + ^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:278: in _env_rendezvous_handler + store = _create_c10d_store( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +hostname = '127.0.0.1', port = 8889, rank = 0, world_size = 1 +timeout = datetime.timedelta(seconds=1800), use_libuv = True + + def _create_c10d_store( + hostname, port, rank, world_size, timeout, use_libuv=True + ) -> Store: + """ + Smartly creates a c10d Store object on ``rank`` based on whether we need to reuse agent store. + + The TCPStore server is assumed to be hosted + on ``hostname:port``. + + By default, the TCPStore server uses the asynchronous implementation + ``LibUVStoreDaemon`` which utilizes libuv. + + If ``torchelastic_use_agent_store()`` is ``True``, then it is assumed that + the agent leader (node rank 0) hosts the TCPStore server (for which the + endpoint is specified by the given ``hostname:port``). Hence + ALL ranks will create and return a TCPStore client (e.g. ``start_daemon=False``). + + If ``torchelastic_use_agent_store()`` is ``False``, then rank 0 will host + the TCPStore (with multi-tenancy) and it is assumed that rank 0's hostname + and port are correctly passed via ``hostname`` and ``port``. All + non-zero ranks will create and return a TCPStore client. + """ + # check if port is uint16_t + if not 0 <= port < 2**16: + raise ValueError(f"port must have value from 0 to 65535 but was {port}.") + + if _torchelastic_use_agent_store(): + # We create a new TCPStore for every retry so no need to add prefix for each attempt. + return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=False, + timeout=timeout, + ) + else: + start_daemon = rank == 0 +> return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=start_daemon, + timeout=timeout, + multi_tenant=True, + use_libuv=use_libuv, + ) +E torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 8889, useIpv6: false, code: -48, name: EADDRINUSE, message: address already in use + +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:198: DistNetworkError +----------------------------- Captured stdout call ----------------------------- +Distributed data parallel: gloo master at 127.0.0.1:8889 +______________________ pytest_train_model_conv_head[PNA] _______________________ + +mpnn_type = 'PNA', overwrite_data = False + + @pytest.mark.parametrize( + "mpnn_type", + [ + "SAGE", + "GIN", + "GAT", + "MFC", + "PNA", + "PNAPlus", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + "PAINN", + ], + ) + def pytest_train_model_conv_head(mpnn_type, overwrite_data=False): +> unittest_train_model( + mpnn_type, None, None, "ci_conv_head.json", False, overwrite_data + ) + +tests/test_graphs.py:306: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_graphs.py:132: in unittest_train_model + hydragnn.run_training(config_file, use_deepspeed) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:62: in _ + run_training(config) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:80: in _ + world_size, world_rank = setup_ddp(use_deepspeed=use_deepspeed) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/utils/distributed/distributed.py:206: in setup_ddp + dist.init_process_group( +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:81: in wrapper + return func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:95: in wrapper + func_return = func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/distributed_c10d.py:1757: in init_process_group + store, rank, world_size = next(rendezvous_iterator) + ^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:278: in _env_rendezvous_handler + store = _create_c10d_store( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +hostname = '127.0.0.1', port = 8889, rank = 0, world_size = 1 +timeout = datetime.timedelta(seconds=1800), use_libuv = True + + def _create_c10d_store( + hostname, port, rank, world_size, timeout, use_libuv=True + ) -> Store: + """ + Smartly creates a c10d Store object on ``rank`` based on whether we need to reuse agent store. + + The TCPStore server is assumed to be hosted + on ``hostname:port``. + + By default, the TCPStore server uses the asynchronous implementation + ``LibUVStoreDaemon`` which utilizes libuv. + + If ``torchelastic_use_agent_store()`` is ``True``, then it is assumed that + the agent leader (node rank 0) hosts the TCPStore server (for which the + endpoint is specified by the given ``hostname:port``). Hence + ALL ranks will create and return a TCPStore client (e.g. ``start_daemon=False``). + + If ``torchelastic_use_agent_store()`` is ``False``, then rank 0 will host + the TCPStore (with multi-tenancy) and it is assumed that rank 0's hostname + and port are correctly passed via ``hostname`` and ``port``. All + non-zero ranks will create and return a TCPStore client. + """ + # check if port is uint16_t + if not 0 <= port < 2**16: + raise ValueError(f"port must have value from 0 to 65535 but was {port}.") + + if _torchelastic_use_agent_store(): + # We create a new TCPStore for every retry so no need to add prefix for each attempt. + return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=False, + timeout=timeout, + ) + else: + start_daemon = rank == 0 +> return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=start_daemon, + timeout=timeout, + multi_tenant=True, + use_libuv=use_libuv, + ) +E torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 8889, useIpv6: false, code: -48, name: EADDRINUSE, message: address already in use + +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:198: DistNetworkError +----------------------------- Captured stdout call ----------------------------- +Distributed data parallel: gloo master at 127.0.0.1:8889 +____________________ pytest_train_model_conv_head[PNAPlus] _____________________ + +mpnn_type = 'PNAPlus', overwrite_data = False + + @pytest.mark.parametrize( + "mpnn_type", + [ + "SAGE", + "GIN", + "GAT", + "MFC", + "PNA", + "PNAPlus", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + "PAINN", + ], + ) + def pytest_train_model_conv_head(mpnn_type, overwrite_data=False): +> unittest_train_model( + mpnn_type, None, None, "ci_conv_head.json", False, overwrite_data + ) + +tests/test_graphs.py:306: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_graphs.py:134: in unittest_train_model + hydragnn.run_training(config, use_deepspeed) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:80: in _ + world_size, world_rank = setup_ddp(use_deepspeed=use_deepspeed) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/utils/distributed/distributed.py:206: in setup_ddp + dist.init_process_group( +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:81: in wrapper + return func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:95: in wrapper + func_return = func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/distributed_c10d.py:1757: in init_process_group + store, rank, world_size = next(rendezvous_iterator) + ^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:278: in _env_rendezvous_handler + store = _create_c10d_store( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +hostname = '127.0.0.1', port = 8889, rank = 0, world_size = 1 +timeout = datetime.timedelta(seconds=1800), use_libuv = True + + def _create_c10d_store( + hostname, port, rank, world_size, timeout, use_libuv=True + ) -> Store: + """ + Smartly creates a c10d Store object on ``rank`` based on whether we need to reuse agent store. + + The TCPStore server is assumed to be hosted + on ``hostname:port``. + + By default, the TCPStore server uses the asynchronous implementation + ``LibUVStoreDaemon`` which utilizes libuv. + + If ``torchelastic_use_agent_store()`` is ``True``, then it is assumed that + the agent leader (node rank 0) hosts the TCPStore server (for which the + endpoint is specified by the given ``hostname:port``). Hence + ALL ranks will create and return a TCPStore client (e.g. ``start_daemon=False``). + + If ``torchelastic_use_agent_store()`` is ``False``, then rank 0 will host + the TCPStore (with multi-tenancy) and it is assumed that rank 0's hostname + and port are correctly passed via ``hostname`` and ``port``. All + non-zero ranks will create and return a TCPStore client. + """ + # check if port is uint16_t + if not 0 <= port < 2**16: + raise ValueError(f"port must have value from 0 to 65535 but was {port}.") + + if _torchelastic_use_agent_store(): + # We create a new TCPStore for every retry so no need to add prefix for each attempt. + return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=False, + timeout=timeout, + ) + else: + start_daemon = rank == 0 +> return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=start_daemon, + timeout=timeout, + multi_tenant=True, + use_libuv=use_libuv, + ) +E torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 8889, useIpv6: false, code: -48, name: EADDRINUSE, message: address already in use + +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:198: DistNetworkError +----------------------------- Captured stdout call ----------------------------- +Distributed data parallel: gloo master at 127.0.0.1:8889 +_____________________ pytest_train_model_conv_head[SchNet] _____________________ + +mpnn_type = 'SchNet', overwrite_data = False + + @pytest.mark.parametrize( + "mpnn_type", + [ + "SAGE", + "GIN", + "GAT", + "MFC", + "PNA", + "PNAPlus", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + "PAINN", + ], + ) + def pytest_train_model_conv_head(mpnn_type, overwrite_data=False): +> unittest_train_model( + mpnn_type, None, None, "ci_conv_head.json", False, overwrite_data + ) + +tests/test_graphs.py:306: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_graphs.py:134: in unittest_train_model + hydragnn.run_training(config, use_deepspeed) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:80: in _ + world_size, world_rank = setup_ddp(use_deepspeed=use_deepspeed) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/utils/distributed/distributed.py:206: in setup_ddp + dist.init_process_group( +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:81: in wrapper + return func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:95: in wrapper + func_return = func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/distributed_c10d.py:1757: in init_process_group + store, rank, world_size = next(rendezvous_iterator) + ^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:278: in _env_rendezvous_handler + store = _create_c10d_store( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +hostname = '127.0.0.1', port = 8889, rank = 0, world_size = 1 +timeout = datetime.timedelta(seconds=1800), use_libuv = True + + def _create_c10d_store( + hostname, port, rank, world_size, timeout, use_libuv=True + ) -> Store: + """ + Smartly creates a c10d Store object on ``rank`` based on whether we need to reuse agent store. + + The TCPStore server is assumed to be hosted + on ``hostname:port``. + + By default, the TCPStore server uses the asynchronous implementation + ``LibUVStoreDaemon`` which utilizes libuv. + + If ``torchelastic_use_agent_store()`` is ``True``, then it is assumed that + the agent leader (node rank 0) hosts the TCPStore server (for which the + endpoint is specified by the given ``hostname:port``). Hence + ALL ranks will create and return a TCPStore client (e.g. ``start_daemon=False``). + + If ``torchelastic_use_agent_store()`` is ``False``, then rank 0 will host + the TCPStore (with multi-tenancy) and it is assumed that rank 0's hostname + and port are correctly passed via ``hostname`` and ``port``. All + non-zero ranks will create and return a TCPStore client. + """ + # check if port is uint16_t + if not 0 <= port < 2**16: + raise ValueError(f"port must have value from 0 to 65535 but was {port}.") + + if _torchelastic_use_agent_store(): + # We create a new TCPStore for every retry so no need to add prefix for each attempt. + return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=False, + timeout=timeout, + ) + else: + start_daemon = rank == 0 +> return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=start_daemon, + timeout=timeout, + multi_tenant=True, + use_libuv=use_libuv, + ) +E torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 8889, useIpv6: false, code: -48, name: EADDRINUSE, message: address already in use + +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:198: DistNetworkError +----------------------------- Captured stdout call ----------------------------- +Distributed data parallel: gloo master at 127.0.0.1:8889 +____________________ pytest_train_model_conv_head[DimeNet] _____________________ + +mpnn_type = 'DimeNet', overwrite_data = False + + @pytest.mark.parametrize( + "mpnn_type", + [ + "SAGE", + "GIN", + "GAT", + "MFC", + "PNA", + "PNAPlus", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + "PAINN", + ], + ) + def pytest_train_model_conv_head(mpnn_type, overwrite_data=False): +> unittest_train_model( + mpnn_type, None, None, "ci_conv_head.json", False, overwrite_data + ) + +tests/test_graphs.py:306: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_graphs.py:134: in unittest_train_model + hydragnn.run_training(config, use_deepspeed) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:80: in _ + world_size, world_rank = setup_ddp(use_deepspeed=use_deepspeed) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/utils/distributed/distributed.py:206: in setup_ddp + dist.init_process_group( +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:81: in wrapper + return func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:95: in wrapper + func_return = func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/distributed_c10d.py:1757: in init_process_group + store, rank, world_size = next(rendezvous_iterator) + ^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:278: in _env_rendezvous_handler + store = _create_c10d_store( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +hostname = '127.0.0.1', port = 8889, rank = 0, world_size = 1 +timeout = datetime.timedelta(seconds=1800), use_libuv = True + + def _create_c10d_store( + hostname, port, rank, world_size, timeout, use_libuv=True + ) -> Store: + """ + Smartly creates a c10d Store object on ``rank`` based on whether we need to reuse agent store. + + The TCPStore server is assumed to be hosted + on ``hostname:port``. + + By default, the TCPStore server uses the asynchronous implementation + ``LibUVStoreDaemon`` which utilizes libuv. + + If ``torchelastic_use_agent_store()`` is ``True``, then it is assumed that + the agent leader (node rank 0) hosts the TCPStore server (for which the + endpoint is specified by the given ``hostname:port``). Hence + ALL ranks will create and return a TCPStore client (e.g. ``start_daemon=False``). + + If ``torchelastic_use_agent_store()`` is ``False``, then rank 0 will host + the TCPStore (with multi-tenancy) and it is assumed that rank 0's hostname + and port are correctly passed via ``hostname`` and ``port``. All + non-zero ranks will create and return a TCPStore client. + """ + # check if port is uint16_t + if not 0 <= port < 2**16: + raise ValueError(f"port must have value from 0 to 65535 but was {port}.") + + if _torchelastic_use_agent_store(): + # We create a new TCPStore for every retry so no need to add prefix for each attempt. + return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=False, + timeout=timeout, + ) + else: + start_daemon = rank == 0 +> return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=start_daemon, + timeout=timeout, + multi_tenant=True, + use_libuv=use_libuv, + ) +E torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 8889, useIpv6: false, code: -48, name: EADDRINUSE, message: address already in use + +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:198: DistNetworkError +----------------------------- Captured stdout call ----------------------------- +Distributed data parallel: gloo master at 127.0.0.1:8889 +______________________ pytest_train_model_conv_head[EGNN] ______________________ + +mpnn_type = 'EGNN', overwrite_data = False + + @pytest.mark.parametrize( + "mpnn_type", + [ + "SAGE", + "GIN", + "GAT", + "MFC", + "PNA", + "PNAPlus", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + "PAINN", + ], + ) + def pytest_train_model_conv_head(mpnn_type, overwrite_data=False): +> unittest_train_model( + mpnn_type, None, None, "ci_conv_head.json", False, overwrite_data + ) + +tests/test_graphs.py:306: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_graphs.py:134: in unittest_train_model + hydragnn.run_training(config, use_deepspeed) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:80: in _ + world_size, world_rank = setup_ddp(use_deepspeed=use_deepspeed) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/utils/distributed/distributed.py:206: in setup_ddp + dist.init_process_group( +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:81: in wrapper + return func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:95: in wrapper + func_return = func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/distributed_c10d.py:1757: in init_process_group + store, rank, world_size = next(rendezvous_iterator) + ^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:278: in _env_rendezvous_handler + store = _create_c10d_store( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +hostname = '127.0.0.1', port = 8889, rank = 0, world_size = 1 +timeout = datetime.timedelta(seconds=1800), use_libuv = True + + def _create_c10d_store( + hostname, port, rank, world_size, timeout, use_libuv=True + ) -> Store: + """ + Smartly creates a c10d Store object on ``rank`` based on whether we need to reuse agent store. + + The TCPStore server is assumed to be hosted + on ``hostname:port``. + + By default, the TCPStore server uses the asynchronous implementation + ``LibUVStoreDaemon`` which utilizes libuv. + + If ``torchelastic_use_agent_store()`` is ``True``, then it is assumed that + the agent leader (node rank 0) hosts the TCPStore server (for which the + endpoint is specified by the given ``hostname:port``). Hence + ALL ranks will create and return a TCPStore client (e.g. ``start_daemon=False``). + + If ``torchelastic_use_agent_store()`` is ``False``, then rank 0 will host + the TCPStore (with multi-tenancy) and it is assumed that rank 0's hostname + and port are correctly passed via ``hostname`` and ``port``. All + non-zero ranks will create and return a TCPStore client. + """ + # check if port is uint16_t + if not 0 <= port < 2**16: + raise ValueError(f"port must have value from 0 to 65535 but was {port}.") + + if _torchelastic_use_agent_store(): + # We create a new TCPStore for every retry so no need to add prefix for each attempt. + return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=False, + timeout=timeout, + ) + else: + start_daemon = rank == 0 +> return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=start_daemon, + timeout=timeout, + multi_tenant=True, + use_libuv=use_libuv, + ) +E torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 8889, useIpv6: false, code: -48, name: EADDRINUSE, message: address already in use + +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:198: DistNetworkError +----------------------------- Captured stdout call ----------------------------- +Distributed data parallel: gloo master at 127.0.0.1:8889 +_____________________ pytest_train_model_conv_head[PNAEq] ______________________ + +mpnn_type = 'PNAEq', overwrite_data = False + + @pytest.mark.parametrize( + "mpnn_type", + [ + "SAGE", + "GIN", + "GAT", + "MFC", + "PNA", + "PNAPlus", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + "PAINN", + ], + ) + def pytest_train_model_conv_head(mpnn_type, overwrite_data=False): +> unittest_train_model( + mpnn_type, None, None, "ci_conv_head.json", False, overwrite_data + ) + +tests/test_graphs.py:306: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_graphs.py:134: in unittest_train_model + hydragnn.run_training(config, use_deepspeed) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:80: in _ + world_size, world_rank = setup_ddp(use_deepspeed=use_deepspeed) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/utils/distributed/distributed.py:206: in setup_ddp + dist.init_process_group( +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:81: in wrapper + return func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:95: in wrapper + func_return = func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/distributed_c10d.py:1757: in init_process_group + store, rank, world_size = next(rendezvous_iterator) + ^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:278: in _env_rendezvous_handler + store = _create_c10d_store( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +hostname = '127.0.0.1', port = 8889, rank = 0, world_size = 1 +timeout = datetime.timedelta(seconds=1800), use_libuv = True + + def _create_c10d_store( + hostname, port, rank, world_size, timeout, use_libuv=True + ) -> Store: + """ + Smartly creates a c10d Store object on ``rank`` based on whether we need to reuse agent store. + + The TCPStore server is assumed to be hosted + on ``hostname:port``. + + By default, the TCPStore server uses the asynchronous implementation + ``LibUVStoreDaemon`` which utilizes libuv. + + If ``torchelastic_use_agent_store()`` is ``True``, then it is assumed that + the agent leader (node rank 0) hosts the TCPStore server (for which the + endpoint is specified by the given ``hostname:port``). Hence + ALL ranks will create and return a TCPStore client (e.g. ``start_daemon=False``). + + If ``torchelastic_use_agent_store()`` is ``False``, then rank 0 will host + the TCPStore (with multi-tenancy) and it is assumed that rank 0's hostname + and port are correctly passed via ``hostname`` and ``port``. All + non-zero ranks will create and return a TCPStore client. + """ + # check if port is uint16_t + if not 0 <= port < 2**16: + raise ValueError(f"port must have value from 0 to 65535 but was {port}.") + + if _torchelastic_use_agent_store(): + # We create a new TCPStore for every retry so no need to add prefix for each attempt. + return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=False, + timeout=timeout, + ) + else: + start_daemon = rank == 0 +> return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=start_daemon, + timeout=timeout, + multi_tenant=True, + use_libuv=use_libuv, + ) +E torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 8889, useIpv6: false, code: -48, name: EADDRINUSE, message: address already in use + +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:198: DistNetworkError +----------------------------- Captured stdout call ----------------------------- +Distributed data parallel: gloo master at 127.0.0.1:8889 +_____________________ pytest_train_model_conv_head[PAINN] ______________________ + +mpnn_type = 'PAINN', overwrite_data = False + + @pytest.mark.parametrize( + "mpnn_type", + [ + "SAGE", + "GIN", + "GAT", + "MFC", + "PNA", + "PNAPlus", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + "PAINN", + ], + ) + def pytest_train_model_conv_head(mpnn_type, overwrite_data=False): +> unittest_train_model( + mpnn_type, None, None, "ci_conv_head.json", False, overwrite_data + ) + +tests/test_graphs.py:306: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_graphs.py:134: in unittest_train_model + hydragnn.run_training(config, use_deepspeed) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:80: in _ + world_size, world_rank = setup_ddp(use_deepspeed=use_deepspeed) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/utils/distributed/distributed.py:206: in setup_ddp + dist.init_process_group( +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:81: in wrapper + return func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:95: in wrapper + func_return = func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/distributed_c10d.py:1757: in init_process_group + store, rank, world_size = next(rendezvous_iterator) + ^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:278: in _env_rendezvous_handler + store = _create_c10d_store( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +hostname = '127.0.0.1', port = 8889, rank = 0, world_size = 1 +timeout = datetime.timedelta(seconds=1800), use_libuv = True + + def _create_c10d_store( + hostname, port, rank, world_size, timeout, use_libuv=True + ) -> Store: + """ + Smartly creates a c10d Store object on ``rank`` based on whether we need to reuse agent store. + + The TCPStore server is assumed to be hosted + on ``hostname:port``. + + By default, the TCPStore server uses the asynchronous implementation + ``LibUVStoreDaemon`` which utilizes libuv. + + If ``torchelastic_use_agent_store()`` is ``True``, then it is assumed that + the agent leader (node rank 0) hosts the TCPStore server (for which the + endpoint is specified by the given ``hostname:port``). Hence + ALL ranks will create and return a TCPStore client (e.g. ``start_daemon=False``). + + If ``torchelastic_use_agent_store()`` is ``False``, then rank 0 will host + the TCPStore (with multi-tenancy) and it is assumed that rank 0's hostname + and port are correctly passed via ``hostname`` and ``port``. All + non-zero ranks will create and return a TCPStore client. + """ + # check if port is uint16_t + if not 0 <= port < 2**16: + raise ValueError(f"port must have value from 0 to 65535 but was {port}.") + + if _torchelastic_use_agent_store(): + # We create a new TCPStore for every retry so no need to add prefix for each attempt. + return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=False, + timeout=timeout, + ) + else: + start_daemon = rank == 0 +> return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=start_daemon, + timeout=timeout, + multi_tenant=True, + use_libuv=use_libuv, + ) +E torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 8889, useIpv6: false, code: -48, name: EADDRINUSE, message: address already in use + +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:198: DistNetworkError +----------------------------- Captured stdout call ----------------------------- +Distributed data parallel: gloo master at 127.0.0.1:8889 +__________________________ pytest_loss_functions[mse] __________________________ + +loss_function_type = 'mse', ci_input = 'ci.json', overwrite_data = False + + @pytest.mark.parametrize( + "loss_function_type", ["mse", "mae", "rmse", "GaussianNLLLoss"] + ) + def pytest_loss_functions(loss_function_type, ci_input="ci.json", overwrite_data=False): + if loss_function_type == "GaussianNLLLoss": + ci_input = "ci_multihead.json" +> unittest_loss_and_activation_functions( + "relu", loss_function_type, ci_input, overwrite_data + ) + +tests/test_loss_and_activation_functions.py:110: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_loss_and_activation_functions.py:100: in unittest_loss_and_activation_functions + hydragnn.run_training(config) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:80: in _ + world_size, world_rank = setup_ddp(use_deepspeed=use_deepspeed) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/utils/distributed/distributed.py:206: in setup_ddp + dist.init_process_group( +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:81: in wrapper + return func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:95: in wrapper + func_return = func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/distributed_c10d.py:1757: in init_process_group + store, rank, world_size = next(rendezvous_iterator) + ^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:278: in _env_rendezvous_handler + store = _create_c10d_store( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +hostname = '127.0.0.1', port = 8889, rank = 0, world_size = 1 +timeout = datetime.timedelta(seconds=1800), use_libuv = True + + def _create_c10d_store( + hostname, port, rank, world_size, timeout, use_libuv=True + ) -> Store: + """ + Smartly creates a c10d Store object on ``rank`` based on whether we need to reuse agent store. + + The TCPStore server is assumed to be hosted + on ``hostname:port``. + + By default, the TCPStore server uses the asynchronous implementation + ``LibUVStoreDaemon`` which utilizes libuv. + + If ``torchelastic_use_agent_store()`` is ``True``, then it is assumed that + the agent leader (node rank 0) hosts the TCPStore server (for which the + endpoint is specified by the given ``hostname:port``). Hence + ALL ranks will create and return a TCPStore client (e.g. ``start_daemon=False``). + + If ``torchelastic_use_agent_store()`` is ``False``, then rank 0 will host + the TCPStore (with multi-tenancy) and it is assumed that rank 0's hostname + and port are correctly passed via ``hostname`` and ``port``. All + non-zero ranks will create and return a TCPStore client. + """ + # check if port is uint16_t + if not 0 <= port < 2**16: + raise ValueError(f"port must have value from 0 to 65535 but was {port}.") + + if _torchelastic_use_agent_store(): + # We create a new TCPStore for every retry so no need to add prefix for each attempt. + return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=False, + timeout=timeout, + ) + else: + start_daemon = rank == 0 +> return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=start_daemon, + timeout=timeout, + multi_tenant=True, + use_libuv=use_libuv, + ) +E torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 8889, useIpv6: false, code: -48, name: EADDRINUSE, message: address already in use + +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:198: DistNetworkError +----------------------------- Captured stdout call ----------------------------- +Distributed data parallel: gloo master at 127.0.0.1:8889 +__________________________ pytest_loss_functions[mae] __________________________ + +loss_function_type = 'mae', ci_input = 'ci.json', overwrite_data = False + + @pytest.mark.parametrize( + "loss_function_type", ["mse", "mae", "rmse", "GaussianNLLLoss"] + ) + def pytest_loss_functions(loss_function_type, ci_input="ci.json", overwrite_data=False): + if loss_function_type == "GaussianNLLLoss": + ci_input = "ci_multihead.json" +> unittest_loss_and_activation_functions( + "relu", loss_function_type, ci_input, overwrite_data + ) + +tests/test_loss_and_activation_functions.py:110: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_loss_and_activation_functions.py:100: in unittest_loss_and_activation_functions + hydragnn.run_training(config) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:80: in _ + world_size, world_rank = setup_ddp(use_deepspeed=use_deepspeed) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/utils/distributed/distributed.py:206: in setup_ddp + dist.init_process_group( +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:81: in wrapper + return func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:95: in wrapper + func_return = func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/distributed_c10d.py:1757: in init_process_group + store, rank, world_size = next(rendezvous_iterator) + ^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:278: in _env_rendezvous_handler + store = _create_c10d_store( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +hostname = '127.0.0.1', port = 8889, rank = 0, world_size = 1 +timeout = datetime.timedelta(seconds=1800), use_libuv = True + + def _create_c10d_store( + hostname, port, rank, world_size, timeout, use_libuv=True + ) -> Store: + """ + Smartly creates a c10d Store object on ``rank`` based on whether we need to reuse agent store. + + The TCPStore server is assumed to be hosted + on ``hostname:port``. + + By default, the TCPStore server uses the asynchronous implementation + ``LibUVStoreDaemon`` which utilizes libuv. + + If ``torchelastic_use_agent_store()`` is ``True``, then it is assumed that + the agent leader (node rank 0) hosts the TCPStore server (for which the + endpoint is specified by the given ``hostname:port``). Hence + ALL ranks will create and return a TCPStore client (e.g. ``start_daemon=False``). + + If ``torchelastic_use_agent_store()`` is ``False``, then rank 0 will host + the TCPStore (with multi-tenancy) and it is assumed that rank 0's hostname + and port are correctly passed via ``hostname`` and ``port``. All + non-zero ranks will create and return a TCPStore client. + """ + # check if port is uint16_t + if not 0 <= port < 2**16: + raise ValueError(f"port must have value from 0 to 65535 but was {port}.") + + if _torchelastic_use_agent_store(): + # We create a new TCPStore for every retry so no need to add prefix for each attempt. + return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=False, + timeout=timeout, + ) + else: + start_daemon = rank == 0 +> return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=start_daemon, + timeout=timeout, + multi_tenant=True, + use_libuv=use_libuv, + ) +E torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 8889, useIpv6: false, code: -48, name: EADDRINUSE, message: address already in use + +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:198: DistNetworkError +----------------------------- Captured stdout call ----------------------------- +Distributed data parallel: gloo master at 127.0.0.1:8889 +_________________________ pytest_loss_functions[rmse] __________________________ + +loss_function_type = 'rmse', ci_input = 'ci.json', overwrite_data = False + + @pytest.mark.parametrize( + "loss_function_type", ["mse", "mae", "rmse", "GaussianNLLLoss"] + ) + def pytest_loss_functions(loss_function_type, ci_input="ci.json", overwrite_data=False): + if loss_function_type == "GaussianNLLLoss": + ci_input = "ci_multihead.json" +> unittest_loss_and_activation_functions( + "relu", loss_function_type, ci_input, overwrite_data + ) + +tests/test_loss_and_activation_functions.py:110: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_loss_and_activation_functions.py:100: in unittest_loss_and_activation_functions + hydragnn.run_training(config) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:80: in _ + world_size, world_rank = setup_ddp(use_deepspeed=use_deepspeed) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/utils/distributed/distributed.py:206: in setup_ddp + dist.init_process_group( +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:81: in wrapper + return func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:95: in wrapper + func_return = func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/distributed_c10d.py:1757: in init_process_group + store, rank, world_size = next(rendezvous_iterator) + ^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:278: in _env_rendezvous_handler + store = _create_c10d_store( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +hostname = '127.0.0.1', port = 8889, rank = 0, world_size = 1 +timeout = datetime.timedelta(seconds=1800), use_libuv = True + + def _create_c10d_store( + hostname, port, rank, world_size, timeout, use_libuv=True + ) -> Store: + """ + Smartly creates a c10d Store object on ``rank`` based on whether we need to reuse agent store. + + The TCPStore server is assumed to be hosted + on ``hostname:port``. + + By default, the TCPStore server uses the asynchronous implementation + ``LibUVStoreDaemon`` which utilizes libuv. + + If ``torchelastic_use_agent_store()`` is ``True``, then it is assumed that + the agent leader (node rank 0) hosts the TCPStore server (for which the + endpoint is specified by the given ``hostname:port``). Hence + ALL ranks will create and return a TCPStore client (e.g. ``start_daemon=False``). + + If ``torchelastic_use_agent_store()`` is ``False``, then rank 0 will host + the TCPStore (with multi-tenancy) and it is assumed that rank 0's hostname + and port are correctly passed via ``hostname`` and ``port``. All + non-zero ranks will create and return a TCPStore client. + """ + # check if port is uint16_t + if not 0 <= port < 2**16: + raise ValueError(f"port must have value from 0 to 65535 but was {port}.") + + if _torchelastic_use_agent_store(): + # We create a new TCPStore for every retry so no need to add prefix for each attempt. + return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=False, + timeout=timeout, + ) + else: + start_daemon = rank == 0 +> return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=start_daemon, + timeout=timeout, + multi_tenant=True, + use_libuv=use_libuv, + ) +E torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 8889, useIpv6: false, code: -48, name: EADDRINUSE, message: address already in use + +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:198: DistNetworkError +----------------------------- Captured stdout call ----------------------------- +Distributed data parallel: gloo master at 127.0.0.1:8889 +____________________ pytest_loss_functions[GaussianNLLLoss] ____________________ + +loss_function_type = 'GaussianNLLLoss', ci_input = 'ci_multihead.json' +overwrite_data = False + + @pytest.mark.parametrize( + "loss_function_type", ["mse", "mae", "rmse", "GaussianNLLLoss"] + ) + def pytest_loss_functions(loss_function_type, ci_input="ci.json", overwrite_data=False): + if loss_function_type == "GaussianNLLLoss": + ci_input = "ci_multihead.json" +> unittest_loss_and_activation_functions( + "relu", loss_function_type, ci_input, overwrite_data + ) + +tests/test_loss_and_activation_functions.py:110: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_loss_and_activation_functions.py:100: in unittest_loss_and_activation_functions + hydragnn.run_training(config) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:80: in _ + world_size, world_rank = setup_ddp(use_deepspeed=use_deepspeed) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/utils/distributed/distributed.py:206: in setup_ddp + dist.init_process_group( +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:81: in wrapper + return func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:95: in wrapper + func_return = func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/distributed_c10d.py:1757: in init_process_group + store, rank, world_size = next(rendezvous_iterator) + ^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:278: in _env_rendezvous_handler + store = _create_c10d_store( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +hostname = '127.0.0.1', port = 8889, rank = 0, world_size = 1 +timeout = datetime.timedelta(seconds=1800), use_libuv = True + + def _create_c10d_store( + hostname, port, rank, world_size, timeout, use_libuv=True + ) -> Store: + """ + Smartly creates a c10d Store object on ``rank`` based on whether we need to reuse agent store. + + The TCPStore server is assumed to be hosted + on ``hostname:port``. + + By default, the TCPStore server uses the asynchronous implementation + ``LibUVStoreDaemon`` which utilizes libuv. + + If ``torchelastic_use_agent_store()`` is ``True``, then it is assumed that + the agent leader (node rank 0) hosts the TCPStore server (for which the + endpoint is specified by the given ``hostname:port``). Hence + ALL ranks will create and return a TCPStore client (e.g. ``start_daemon=False``). + + If ``torchelastic_use_agent_store()`` is ``False``, then rank 0 will host + the TCPStore (with multi-tenancy) and it is assumed that rank 0's hostname + and port are correctly passed via ``hostname`` and ``port``. All + non-zero ranks will create and return a TCPStore client. + """ + # check if port is uint16_t + if not 0 <= port < 2**16: + raise ValueError(f"port must have value from 0 to 65535 but was {port}.") + + if _torchelastic_use_agent_store(): + # We create a new TCPStore for every retry so no need to add prefix for each attempt. + return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=False, + timeout=timeout, + ) + else: + start_daemon = rank == 0 +> return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=start_daemon, + timeout=timeout, + multi_tenant=True, + use_libuv=use_libuv, + ) +E torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 8889, useIpv6: false, code: -48, name: EADDRINUSE, message: address already in use + +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:198: DistNetworkError +----------------------------- Captured stdout call ----------------------------- +Distributed data parallel: gloo master at 127.0.0.1:8889 +_________________ pytest_activation_functions_multihead[relu] __________________ + +activation_function_type = 'relu', ci_input = 'ci_multihead.json' +overwrite_data = False + + @pytest.mark.parametrize( + "activation_function_type", + ["relu", "selu", "prelu", "elu", "lrelu_01", "lrelu_025", "lrelu_05"], + ) + def pytest_activation_functions_multihead( + activation_function_type, ci_input="ci_multihead.json", overwrite_data=False + ): +> unittest_loss_and_activation_functions( + activation_function_type, "mse", ci_input, overwrite_data + ) + +tests/test_loss_and_activation_functions.py:123: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_loss_and_activation_functions.py:100: in unittest_loss_and_activation_functions + hydragnn.run_training(config) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:80: in _ + world_size, world_rank = setup_ddp(use_deepspeed=use_deepspeed) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/utils/distributed/distributed.py:206: in setup_ddp + dist.init_process_group( +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:81: in wrapper + return func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:95: in wrapper + func_return = func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/distributed_c10d.py:1757: in init_process_group + store, rank, world_size = next(rendezvous_iterator) + ^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:278: in _env_rendezvous_handler + store = _create_c10d_store( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +hostname = '127.0.0.1', port = 8889, rank = 0, world_size = 1 +timeout = datetime.timedelta(seconds=1800), use_libuv = True + + def _create_c10d_store( + hostname, port, rank, world_size, timeout, use_libuv=True + ) -> Store: + """ + Smartly creates a c10d Store object on ``rank`` based on whether we need to reuse agent store. + + The TCPStore server is assumed to be hosted + on ``hostname:port``. + + By default, the TCPStore server uses the asynchronous implementation + ``LibUVStoreDaemon`` which utilizes libuv. + + If ``torchelastic_use_agent_store()`` is ``True``, then it is assumed that + the agent leader (node rank 0) hosts the TCPStore server (for which the + endpoint is specified by the given ``hostname:port``). Hence + ALL ranks will create and return a TCPStore client (e.g. ``start_daemon=False``). + + If ``torchelastic_use_agent_store()`` is ``False``, then rank 0 will host + the TCPStore (with multi-tenancy) and it is assumed that rank 0's hostname + and port are correctly passed via ``hostname`` and ``port``. All + non-zero ranks will create and return a TCPStore client. + """ + # check if port is uint16_t + if not 0 <= port < 2**16: + raise ValueError(f"port must have value from 0 to 65535 but was {port}.") + + if _torchelastic_use_agent_store(): + # We create a new TCPStore for every retry so no need to add prefix for each attempt. + return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=False, + timeout=timeout, + ) + else: + start_daemon = rank == 0 +> return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=start_daemon, + timeout=timeout, + multi_tenant=True, + use_libuv=use_libuv, + ) +E torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 8889, useIpv6: false, code: -48, name: EADDRINUSE, message: address already in use + +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:198: DistNetworkError +----------------------------- Captured stdout call ----------------------------- +Distributed data parallel: gloo master at 127.0.0.1:8889 +_________________ pytest_activation_functions_multihead[selu] __________________ + +activation_function_type = 'selu', ci_input = 'ci_multihead.json' +overwrite_data = False + + @pytest.mark.parametrize( + "activation_function_type", + ["relu", "selu", "prelu", "elu", "lrelu_01", "lrelu_025", "lrelu_05"], + ) + def pytest_activation_functions_multihead( + activation_function_type, ci_input="ci_multihead.json", overwrite_data=False + ): +> unittest_loss_and_activation_functions( + activation_function_type, "mse", ci_input, overwrite_data + ) + +tests/test_loss_and_activation_functions.py:123: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_loss_and_activation_functions.py:100: in unittest_loss_and_activation_functions + hydragnn.run_training(config) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:80: in _ + world_size, world_rank = setup_ddp(use_deepspeed=use_deepspeed) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/utils/distributed/distributed.py:206: in setup_ddp + dist.init_process_group( +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:81: in wrapper + return func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:95: in wrapper + func_return = func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/distributed_c10d.py:1757: in init_process_group + store, rank, world_size = next(rendezvous_iterator) + ^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:278: in _env_rendezvous_handler + store = _create_c10d_store( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +hostname = '127.0.0.1', port = 8889, rank = 0, world_size = 1 +timeout = datetime.timedelta(seconds=1800), use_libuv = True + + def _create_c10d_store( + hostname, port, rank, world_size, timeout, use_libuv=True + ) -> Store: + """ + Smartly creates a c10d Store object on ``rank`` based on whether we need to reuse agent store. + + The TCPStore server is assumed to be hosted + on ``hostname:port``. + + By default, the TCPStore server uses the asynchronous implementation + ``LibUVStoreDaemon`` which utilizes libuv. + + If ``torchelastic_use_agent_store()`` is ``True``, then it is assumed that + the agent leader (node rank 0) hosts the TCPStore server (for which the + endpoint is specified by the given ``hostname:port``). Hence + ALL ranks will create and return a TCPStore client (e.g. ``start_daemon=False``). + + If ``torchelastic_use_agent_store()`` is ``False``, then rank 0 will host + the TCPStore (with multi-tenancy) and it is assumed that rank 0's hostname + and port are correctly passed via ``hostname`` and ``port``. All + non-zero ranks will create and return a TCPStore client. + """ + # check if port is uint16_t + if not 0 <= port < 2**16: + raise ValueError(f"port must have value from 0 to 65535 but was {port}.") + + if _torchelastic_use_agent_store(): + # We create a new TCPStore for every retry so no need to add prefix for each attempt. + return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=False, + timeout=timeout, + ) + else: + start_daemon = rank == 0 +> return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=start_daemon, + timeout=timeout, + multi_tenant=True, + use_libuv=use_libuv, + ) +E torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 8889, useIpv6: false, code: -48, name: EADDRINUSE, message: address already in use + +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:198: DistNetworkError +----------------------------- Captured stdout call ----------------------------- +Distributed data parallel: gloo master at 127.0.0.1:8889 +_________________ pytest_activation_functions_multihead[prelu] _________________ + +activation_function_type = 'prelu', ci_input = 'ci_multihead.json' +overwrite_data = False + + @pytest.mark.parametrize( + "activation_function_type", + ["relu", "selu", "prelu", "elu", "lrelu_01", "lrelu_025", "lrelu_05"], + ) + def pytest_activation_functions_multihead( + activation_function_type, ci_input="ci_multihead.json", overwrite_data=False + ): +> unittest_loss_and_activation_functions( + activation_function_type, "mse", ci_input, overwrite_data + ) + +tests/test_loss_and_activation_functions.py:123: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_loss_and_activation_functions.py:100: in unittest_loss_and_activation_functions + hydragnn.run_training(config) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:80: in _ + world_size, world_rank = setup_ddp(use_deepspeed=use_deepspeed) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/utils/distributed/distributed.py:206: in setup_ddp + dist.init_process_group( +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:81: in wrapper + return func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:95: in wrapper + func_return = func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/distributed_c10d.py:1757: in init_process_group + store, rank, world_size = next(rendezvous_iterator) + ^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:278: in _env_rendezvous_handler + store = _create_c10d_store( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +hostname = '127.0.0.1', port = 8889, rank = 0, world_size = 1 +timeout = datetime.timedelta(seconds=1800), use_libuv = True + + def _create_c10d_store( + hostname, port, rank, world_size, timeout, use_libuv=True + ) -> Store: + """ + Smartly creates a c10d Store object on ``rank`` based on whether we need to reuse agent store. + + The TCPStore server is assumed to be hosted + on ``hostname:port``. + + By default, the TCPStore server uses the asynchronous implementation + ``LibUVStoreDaemon`` which utilizes libuv. + + If ``torchelastic_use_agent_store()`` is ``True``, then it is assumed that + the agent leader (node rank 0) hosts the TCPStore server (for which the + endpoint is specified by the given ``hostname:port``). Hence + ALL ranks will create and return a TCPStore client (e.g. ``start_daemon=False``). + + If ``torchelastic_use_agent_store()`` is ``False``, then rank 0 will host + the TCPStore (with multi-tenancy) and it is assumed that rank 0's hostname + and port are correctly passed via ``hostname`` and ``port``. All + non-zero ranks will create and return a TCPStore client. + """ + # check if port is uint16_t + if not 0 <= port < 2**16: + raise ValueError(f"port must have value from 0 to 65535 but was {port}.") + + if _torchelastic_use_agent_store(): + # We create a new TCPStore for every retry so no need to add prefix for each attempt. + return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=False, + timeout=timeout, + ) + else: + start_daemon = rank == 0 +> return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=start_daemon, + timeout=timeout, + multi_tenant=True, + use_libuv=use_libuv, + ) +E torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 8889, useIpv6: false, code: -48, name: EADDRINUSE, message: address already in use + +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:198: DistNetworkError +----------------------------- Captured stdout call ----------------------------- +Distributed data parallel: gloo master at 127.0.0.1:8889 +__________________ pytest_activation_functions_multihead[elu] __________________ + +activation_function_type = 'elu', ci_input = 'ci_multihead.json' +overwrite_data = False + + @pytest.mark.parametrize( + "activation_function_type", + ["relu", "selu", "prelu", "elu", "lrelu_01", "lrelu_025", "lrelu_05"], + ) + def pytest_activation_functions_multihead( + activation_function_type, ci_input="ci_multihead.json", overwrite_data=False + ): +> unittest_loss_and_activation_functions( + activation_function_type, "mse", ci_input, overwrite_data + ) + +tests/test_loss_and_activation_functions.py:123: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_loss_and_activation_functions.py:100: in unittest_loss_and_activation_functions + hydragnn.run_training(config) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:80: in _ + world_size, world_rank = setup_ddp(use_deepspeed=use_deepspeed) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/utils/distributed/distributed.py:206: in setup_ddp + dist.init_process_group( +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:81: in wrapper + return func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:95: in wrapper + func_return = func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/distributed_c10d.py:1757: in init_process_group + store, rank, world_size = next(rendezvous_iterator) + ^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:278: in _env_rendezvous_handler + store = _create_c10d_store( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +hostname = '127.0.0.1', port = 8889, rank = 0, world_size = 1 +timeout = datetime.timedelta(seconds=1800), use_libuv = True + + def _create_c10d_store( + hostname, port, rank, world_size, timeout, use_libuv=True + ) -> Store: + """ + Smartly creates a c10d Store object on ``rank`` based on whether we need to reuse agent store. + + The TCPStore server is assumed to be hosted + on ``hostname:port``. + + By default, the TCPStore server uses the asynchronous implementation + ``LibUVStoreDaemon`` which utilizes libuv. + + If ``torchelastic_use_agent_store()`` is ``True``, then it is assumed that + the agent leader (node rank 0) hosts the TCPStore server (for which the + endpoint is specified by the given ``hostname:port``). Hence + ALL ranks will create and return a TCPStore client (e.g. ``start_daemon=False``). + + If ``torchelastic_use_agent_store()`` is ``False``, then rank 0 will host + the TCPStore (with multi-tenancy) and it is assumed that rank 0's hostname + and port are correctly passed via ``hostname`` and ``port``. All + non-zero ranks will create and return a TCPStore client. + """ + # check if port is uint16_t + if not 0 <= port < 2**16: + raise ValueError(f"port must have value from 0 to 65535 but was {port}.") + + if _torchelastic_use_agent_store(): + # We create a new TCPStore for every retry so no need to add prefix for each attempt. + return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=False, + timeout=timeout, + ) + else: + start_daemon = rank == 0 +> return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=start_daemon, + timeout=timeout, + multi_tenant=True, + use_libuv=use_libuv, + ) +E torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 8889, useIpv6: false, code: -48, name: EADDRINUSE, message: address already in use + +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:198: DistNetworkError +----------------------------- Captured stdout call ----------------------------- +Distributed data parallel: gloo master at 127.0.0.1:8889 +_______________ pytest_activation_functions_multihead[lrelu_01] ________________ + +activation_function_type = 'lrelu_01', ci_input = 'ci_multihead.json' +overwrite_data = False + + @pytest.mark.parametrize( + "activation_function_type", + ["relu", "selu", "prelu", "elu", "lrelu_01", "lrelu_025", "lrelu_05"], + ) + def pytest_activation_functions_multihead( + activation_function_type, ci_input="ci_multihead.json", overwrite_data=False + ): +> unittest_loss_and_activation_functions( + activation_function_type, "mse", ci_input, overwrite_data + ) + +tests/test_loss_and_activation_functions.py:123: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_loss_and_activation_functions.py:100: in unittest_loss_and_activation_functions + hydragnn.run_training(config) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:80: in _ + world_size, world_rank = setup_ddp(use_deepspeed=use_deepspeed) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/utils/distributed/distributed.py:206: in setup_ddp + dist.init_process_group( +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:81: in wrapper + return func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:95: in wrapper + func_return = func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/distributed_c10d.py:1757: in init_process_group + store, rank, world_size = next(rendezvous_iterator) + ^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:278: in _env_rendezvous_handler + store = _create_c10d_store( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +hostname = '127.0.0.1', port = 8889, rank = 0, world_size = 1 +timeout = datetime.timedelta(seconds=1800), use_libuv = True + + def _create_c10d_store( + hostname, port, rank, world_size, timeout, use_libuv=True + ) -> Store: + """ + Smartly creates a c10d Store object on ``rank`` based on whether we need to reuse agent store. + + The TCPStore server is assumed to be hosted + on ``hostname:port``. + + By default, the TCPStore server uses the asynchronous implementation + ``LibUVStoreDaemon`` which utilizes libuv. + + If ``torchelastic_use_agent_store()`` is ``True``, then it is assumed that + the agent leader (node rank 0) hosts the TCPStore server (for which the + endpoint is specified by the given ``hostname:port``). Hence + ALL ranks will create and return a TCPStore client (e.g. ``start_daemon=False``). + + If ``torchelastic_use_agent_store()`` is ``False``, then rank 0 will host + the TCPStore (with multi-tenancy) and it is assumed that rank 0's hostname + and port are correctly passed via ``hostname`` and ``port``. All + non-zero ranks will create and return a TCPStore client. + """ + # check if port is uint16_t + if not 0 <= port < 2**16: + raise ValueError(f"port must have value from 0 to 65535 but was {port}.") + + if _torchelastic_use_agent_store(): + # We create a new TCPStore for every retry so no need to add prefix for each attempt. + return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=False, + timeout=timeout, + ) + else: + start_daemon = rank == 0 +> return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=start_daemon, + timeout=timeout, + multi_tenant=True, + use_libuv=use_libuv, + ) +E torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 8889, useIpv6: false, code: -48, name: EADDRINUSE, message: address already in use + +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:198: DistNetworkError +----------------------------- Captured stdout call ----------------------------- +Distributed data parallel: gloo master at 127.0.0.1:8889 +_______________ pytest_activation_functions_multihead[lrelu_025] _______________ + +activation_function_type = 'lrelu_025', ci_input = 'ci_multihead.json' +overwrite_data = False + + @pytest.mark.parametrize( + "activation_function_type", + ["relu", "selu", "prelu", "elu", "lrelu_01", "lrelu_025", "lrelu_05"], + ) + def pytest_activation_functions_multihead( + activation_function_type, ci_input="ci_multihead.json", overwrite_data=False + ): +> unittest_loss_and_activation_functions( + activation_function_type, "mse", ci_input, overwrite_data + ) + +tests/test_loss_and_activation_functions.py:123: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_loss_and_activation_functions.py:100: in unittest_loss_and_activation_functions + hydragnn.run_training(config) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:80: in _ + world_size, world_rank = setup_ddp(use_deepspeed=use_deepspeed) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/utils/distributed/distributed.py:206: in setup_ddp + dist.init_process_group( +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:81: in wrapper + return func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:95: in wrapper + func_return = func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/distributed_c10d.py:1757: in init_process_group + store, rank, world_size = next(rendezvous_iterator) + ^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:278: in _env_rendezvous_handler + store = _create_c10d_store( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +hostname = '127.0.0.1', port = 8889, rank = 0, world_size = 1 +timeout = datetime.timedelta(seconds=1800), use_libuv = True + + def _create_c10d_store( + hostname, port, rank, world_size, timeout, use_libuv=True + ) -> Store: + """ + Smartly creates a c10d Store object on ``rank`` based on whether we need to reuse agent store. + + The TCPStore server is assumed to be hosted + on ``hostname:port``. + + By default, the TCPStore server uses the asynchronous implementation + ``LibUVStoreDaemon`` which utilizes libuv. + + If ``torchelastic_use_agent_store()`` is ``True``, then it is assumed that + the agent leader (node rank 0) hosts the TCPStore server (for which the + endpoint is specified by the given ``hostname:port``). Hence + ALL ranks will create and return a TCPStore client (e.g. ``start_daemon=False``). + + If ``torchelastic_use_agent_store()`` is ``False``, then rank 0 will host + the TCPStore (with multi-tenancy) and it is assumed that rank 0's hostname + and port are correctly passed via ``hostname`` and ``port``. All + non-zero ranks will create and return a TCPStore client. + """ + # check if port is uint16_t + if not 0 <= port < 2**16: + raise ValueError(f"port must have value from 0 to 65535 but was {port}.") + + if _torchelastic_use_agent_store(): + # We create a new TCPStore for every retry so no need to add prefix for each attempt. + return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=False, + timeout=timeout, + ) + else: + start_daemon = rank == 0 +> return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=start_daemon, + timeout=timeout, + multi_tenant=True, + use_libuv=use_libuv, + ) +E torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 8889, useIpv6: false, code: -48, name: EADDRINUSE, message: address already in use + +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:198: DistNetworkError +----------------------------- Captured stdout call ----------------------------- +Distributed data parallel: gloo master at 127.0.0.1:8889 +_______________ pytest_activation_functions_multihead[lrelu_05] ________________ + +activation_function_type = 'lrelu_05', ci_input = 'ci_multihead.json' +overwrite_data = False + + @pytest.mark.parametrize( + "activation_function_type", + ["relu", "selu", "prelu", "elu", "lrelu_01", "lrelu_025", "lrelu_05"], + ) + def pytest_activation_functions_multihead( + activation_function_type, ci_input="ci_multihead.json", overwrite_data=False + ): +> unittest_loss_and_activation_functions( + activation_function_type, "mse", ci_input, overwrite_data + ) + +tests/test_loss_and_activation_functions.py:123: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_loss_and_activation_functions.py:100: in unittest_loss_and_activation_functions + hydragnn.run_training(config) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:80: in _ + world_size, world_rank = setup_ddp(use_deepspeed=use_deepspeed) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/utils/distributed/distributed.py:206: in setup_ddp + dist.init_process_group( +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:81: in wrapper + return func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:95: in wrapper + func_return = func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/distributed_c10d.py:1757: in init_process_group + store, rank, world_size = next(rendezvous_iterator) + ^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:278: in _env_rendezvous_handler + store = _create_c10d_store( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +hostname = '127.0.0.1', port = 8889, rank = 0, world_size = 1 +timeout = datetime.timedelta(seconds=1800), use_libuv = True + + def _create_c10d_store( + hostname, port, rank, world_size, timeout, use_libuv=True + ) -> Store: + """ + Smartly creates a c10d Store object on ``rank`` based on whether we need to reuse agent store. + + The TCPStore server is assumed to be hosted + on ``hostname:port``. + + By default, the TCPStore server uses the asynchronous implementation + ``LibUVStoreDaemon`` which utilizes libuv. + + If ``torchelastic_use_agent_store()`` is ``True``, then it is assumed that + the agent leader (node rank 0) hosts the TCPStore server (for which the + endpoint is specified by the given ``hostname:port``). Hence + ALL ranks will create and return a TCPStore client (e.g. ``start_daemon=False``). + + If ``torchelastic_use_agent_store()`` is ``False``, then rank 0 will host + the TCPStore (with multi-tenancy) and it is assumed that rank 0's hostname + and port are correctly passed via ``hostname`` and ``port``. All + non-zero ranks will create and return a TCPStore client. + """ + # check if port is uint16_t + if not 0 <= port < 2**16: + raise ValueError(f"port must have value from 0 to 65535 but was {port}.") + + if _torchelastic_use_agent_store(): + # We create a new TCPStore for every retry so no need to add prefix for each attempt. + return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=False, + timeout=timeout, + ) + else: + start_daemon = rank == 0 +> return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=start_daemon, + timeout=timeout, + multi_tenant=True, + use_libuv=use_libuv, + ) +E torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 8889, useIpv6: false, code: -48, name: EADDRINUSE, message: address already in use + +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:198: DistNetworkError +----------------------------- Captured stdout call ----------------------------- +Distributed data parallel: gloo master at 127.0.0.1:8889 +________________ pytest_activation_functions_vectoroutput[relu] ________________ + +activation_function_type = 'relu', ci_input = 'ci_vectoroutput.json' +overwrite_data = False + + @pytest.mark.parametrize( + "activation_function_type", + ["relu", "selu", "prelu", "elu", "lrelu_01", "lrelu_025", "lrelu_05"], + ) + def pytest_activation_functions_vectoroutput( + activation_function_type, ci_input="ci_vectoroutput.json", overwrite_data=False + ): +> unittest_loss_and_activation_functions( + activation_function_type, "mse", ci_input, overwrite_data + ) + +tests/test_loss_and_activation_functions.py:136: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_loss_and_activation_functions.py:100: in unittest_loss_and_activation_functions + hydragnn.run_training(config) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:80: in _ + world_size, world_rank = setup_ddp(use_deepspeed=use_deepspeed) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/utils/distributed/distributed.py:206: in setup_ddp + dist.init_process_group( +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:81: in wrapper + return func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:95: in wrapper + func_return = func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/distributed_c10d.py:1757: in init_process_group + store, rank, world_size = next(rendezvous_iterator) + ^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:278: in _env_rendezvous_handler + store = _create_c10d_store( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +hostname = '127.0.0.1', port = 8889, rank = 0, world_size = 1 +timeout = datetime.timedelta(seconds=1800), use_libuv = True + + def _create_c10d_store( + hostname, port, rank, world_size, timeout, use_libuv=True + ) -> Store: + """ + Smartly creates a c10d Store object on ``rank`` based on whether we need to reuse agent store. + + The TCPStore server is assumed to be hosted + on ``hostname:port``. + + By default, the TCPStore server uses the asynchronous implementation + ``LibUVStoreDaemon`` which utilizes libuv. + + If ``torchelastic_use_agent_store()`` is ``True``, then it is assumed that + the agent leader (node rank 0) hosts the TCPStore server (for which the + endpoint is specified by the given ``hostname:port``). Hence + ALL ranks will create and return a TCPStore client (e.g. ``start_daemon=False``). + + If ``torchelastic_use_agent_store()`` is ``False``, then rank 0 will host + the TCPStore (with multi-tenancy) and it is assumed that rank 0's hostname + and port are correctly passed via ``hostname`` and ``port``. All + non-zero ranks will create and return a TCPStore client. + """ + # check if port is uint16_t + if not 0 <= port < 2**16: + raise ValueError(f"port must have value from 0 to 65535 but was {port}.") + + if _torchelastic_use_agent_store(): + # We create a new TCPStore for every retry so no need to add prefix for each attempt. + return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=False, + timeout=timeout, + ) + else: + start_daemon = rank == 0 +> return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=start_daemon, + timeout=timeout, + multi_tenant=True, + use_libuv=use_libuv, + ) +E torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 8889, useIpv6: false, code: -48, name: EADDRINUSE, message: address already in use + +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:198: DistNetworkError +----------------------------- Captured stdout call ----------------------------- +Distributed data parallel: gloo master at 127.0.0.1:8889 +________________ pytest_activation_functions_vectoroutput[selu] ________________ + +activation_function_type = 'selu', ci_input = 'ci_vectoroutput.json' +overwrite_data = False + + @pytest.mark.parametrize( + "activation_function_type", + ["relu", "selu", "prelu", "elu", "lrelu_01", "lrelu_025", "lrelu_05"], + ) + def pytest_activation_functions_vectoroutput( + activation_function_type, ci_input="ci_vectoroutput.json", overwrite_data=False + ): +> unittest_loss_and_activation_functions( + activation_function_type, "mse", ci_input, overwrite_data + ) + +tests/test_loss_and_activation_functions.py:136: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_loss_and_activation_functions.py:100: in unittest_loss_and_activation_functions + hydragnn.run_training(config) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:80: in _ + world_size, world_rank = setup_ddp(use_deepspeed=use_deepspeed) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/utils/distributed/distributed.py:206: in setup_ddp + dist.init_process_group( +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:81: in wrapper + return func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:95: in wrapper + func_return = func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/distributed_c10d.py:1757: in init_process_group + store, rank, world_size = next(rendezvous_iterator) + ^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:278: in _env_rendezvous_handler + store = _create_c10d_store( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +hostname = '127.0.0.1', port = 8889, rank = 0, world_size = 1 +timeout = datetime.timedelta(seconds=1800), use_libuv = True + + def _create_c10d_store( + hostname, port, rank, world_size, timeout, use_libuv=True + ) -> Store: + """ + Smartly creates a c10d Store object on ``rank`` based on whether we need to reuse agent store. + + The TCPStore server is assumed to be hosted + on ``hostname:port``. + + By default, the TCPStore server uses the asynchronous implementation + ``LibUVStoreDaemon`` which utilizes libuv. + + If ``torchelastic_use_agent_store()`` is ``True``, then it is assumed that + the agent leader (node rank 0) hosts the TCPStore server (for which the + endpoint is specified by the given ``hostname:port``). Hence + ALL ranks will create and return a TCPStore client (e.g. ``start_daemon=False``). + + If ``torchelastic_use_agent_store()`` is ``False``, then rank 0 will host + the TCPStore (with multi-tenancy) and it is assumed that rank 0's hostname + and port are correctly passed via ``hostname`` and ``port``. All + non-zero ranks will create and return a TCPStore client. + """ + # check if port is uint16_t + if not 0 <= port < 2**16: + raise ValueError(f"port must have value from 0 to 65535 but was {port}.") + + if _torchelastic_use_agent_store(): + # We create a new TCPStore for every retry so no need to add prefix for each attempt. + return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=False, + timeout=timeout, + ) + else: + start_daemon = rank == 0 +> return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=start_daemon, + timeout=timeout, + multi_tenant=True, + use_libuv=use_libuv, + ) +E torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 8889, useIpv6: false, code: -48, name: EADDRINUSE, message: address already in use + +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:198: DistNetworkError +----------------------------- Captured stdout call ----------------------------- +Distributed data parallel: gloo master at 127.0.0.1:8889 +_______________ pytest_activation_functions_vectoroutput[prelu] ________________ + +activation_function_type = 'prelu', ci_input = 'ci_vectoroutput.json' +overwrite_data = False + + @pytest.mark.parametrize( + "activation_function_type", + ["relu", "selu", "prelu", "elu", "lrelu_01", "lrelu_025", "lrelu_05"], + ) + def pytest_activation_functions_vectoroutput( + activation_function_type, ci_input="ci_vectoroutput.json", overwrite_data=False + ): +> unittest_loss_and_activation_functions( + activation_function_type, "mse", ci_input, overwrite_data + ) + +tests/test_loss_and_activation_functions.py:136: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_loss_and_activation_functions.py:100: in unittest_loss_and_activation_functions + hydragnn.run_training(config) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:80: in _ + world_size, world_rank = setup_ddp(use_deepspeed=use_deepspeed) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/utils/distributed/distributed.py:206: in setup_ddp + dist.init_process_group( +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:81: in wrapper + return func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:95: in wrapper + func_return = func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/distributed_c10d.py:1757: in init_process_group + store, rank, world_size = next(rendezvous_iterator) + ^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:278: in _env_rendezvous_handler + store = _create_c10d_store( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +hostname = '127.0.0.1', port = 8889, rank = 0, world_size = 1 +timeout = datetime.timedelta(seconds=1800), use_libuv = True + + def _create_c10d_store( + hostname, port, rank, world_size, timeout, use_libuv=True + ) -> Store: + """ + Smartly creates a c10d Store object on ``rank`` based on whether we need to reuse agent store. + + The TCPStore server is assumed to be hosted + on ``hostname:port``. + + By default, the TCPStore server uses the asynchronous implementation + ``LibUVStoreDaemon`` which utilizes libuv. + + If ``torchelastic_use_agent_store()`` is ``True``, then it is assumed that + the agent leader (node rank 0) hosts the TCPStore server (for which the + endpoint is specified by the given ``hostname:port``). Hence + ALL ranks will create and return a TCPStore client (e.g. ``start_daemon=False``). + + If ``torchelastic_use_agent_store()`` is ``False``, then rank 0 will host + the TCPStore (with multi-tenancy) and it is assumed that rank 0's hostname + and port are correctly passed via ``hostname`` and ``port``. All + non-zero ranks will create and return a TCPStore client. + """ + # check if port is uint16_t + if not 0 <= port < 2**16: + raise ValueError(f"port must have value from 0 to 65535 but was {port}.") + + if _torchelastic_use_agent_store(): + # We create a new TCPStore for every retry so no need to add prefix for each attempt. + return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=False, + timeout=timeout, + ) + else: + start_daemon = rank == 0 +> return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=start_daemon, + timeout=timeout, + multi_tenant=True, + use_libuv=use_libuv, + ) +E torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 8889, useIpv6: false, code: -48, name: EADDRINUSE, message: address already in use + +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:198: DistNetworkError +----------------------------- Captured stdout call ----------------------------- +Distributed data parallel: gloo master at 127.0.0.1:8889 +________________ pytest_activation_functions_vectoroutput[elu] _________________ + +activation_function_type = 'elu', ci_input = 'ci_vectoroutput.json' +overwrite_data = False + + @pytest.mark.parametrize( + "activation_function_type", + ["relu", "selu", "prelu", "elu", "lrelu_01", "lrelu_025", "lrelu_05"], + ) + def pytest_activation_functions_vectoroutput( + activation_function_type, ci_input="ci_vectoroutput.json", overwrite_data=False + ): +> unittest_loss_and_activation_functions( + activation_function_type, "mse", ci_input, overwrite_data + ) + +tests/test_loss_and_activation_functions.py:136: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_loss_and_activation_functions.py:100: in unittest_loss_and_activation_functions + hydragnn.run_training(config) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:80: in _ + world_size, world_rank = setup_ddp(use_deepspeed=use_deepspeed) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/utils/distributed/distributed.py:206: in setup_ddp + dist.init_process_group( +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:81: in wrapper + return func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:95: in wrapper + func_return = func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/distributed_c10d.py:1757: in init_process_group + store, rank, world_size = next(rendezvous_iterator) + ^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:278: in _env_rendezvous_handler + store = _create_c10d_store( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +hostname = '127.0.0.1', port = 8889, rank = 0, world_size = 1 +timeout = datetime.timedelta(seconds=1800), use_libuv = True + + def _create_c10d_store( + hostname, port, rank, world_size, timeout, use_libuv=True + ) -> Store: + """ + Smartly creates a c10d Store object on ``rank`` based on whether we need to reuse agent store. + + The TCPStore server is assumed to be hosted + on ``hostname:port``. + + By default, the TCPStore server uses the asynchronous implementation + ``LibUVStoreDaemon`` which utilizes libuv. + + If ``torchelastic_use_agent_store()`` is ``True``, then it is assumed that + the agent leader (node rank 0) hosts the TCPStore server (for which the + endpoint is specified by the given ``hostname:port``). Hence + ALL ranks will create and return a TCPStore client (e.g. ``start_daemon=False``). + + If ``torchelastic_use_agent_store()`` is ``False``, then rank 0 will host + the TCPStore (with multi-tenancy) and it is assumed that rank 0's hostname + and port are correctly passed via ``hostname`` and ``port``. All + non-zero ranks will create and return a TCPStore client. + """ + # check if port is uint16_t + if not 0 <= port < 2**16: + raise ValueError(f"port must have value from 0 to 65535 but was {port}.") + + if _torchelastic_use_agent_store(): + # We create a new TCPStore for every retry so no need to add prefix for each attempt. + return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=False, + timeout=timeout, + ) + else: + start_daemon = rank == 0 +> return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=start_daemon, + timeout=timeout, + multi_tenant=True, + use_libuv=use_libuv, + ) +E torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 8889, useIpv6: false, code: -48, name: EADDRINUSE, message: address already in use + +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:198: DistNetworkError +----------------------------- Captured stdout call ----------------------------- +Distributed data parallel: gloo master at 127.0.0.1:8889 +______________ pytest_activation_functions_vectoroutput[lrelu_01] ______________ + +activation_function_type = 'lrelu_01', ci_input = 'ci_vectoroutput.json' +overwrite_data = False + + @pytest.mark.parametrize( + "activation_function_type", + ["relu", "selu", "prelu", "elu", "lrelu_01", "lrelu_025", "lrelu_05"], + ) + def pytest_activation_functions_vectoroutput( + activation_function_type, ci_input="ci_vectoroutput.json", overwrite_data=False + ): +> unittest_loss_and_activation_functions( + activation_function_type, "mse", ci_input, overwrite_data + ) + +tests/test_loss_and_activation_functions.py:136: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_loss_and_activation_functions.py:100: in unittest_loss_and_activation_functions + hydragnn.run_training(config) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:80: in _ + world_size, world_rank = setup_ddp(use_deepspeed=use_deepspeed) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/utils/distributed/distributed.py:206: in setup_ddp + dist.init_process_group( +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:81: in wrapper + return func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:95: in wrapper + func_return = func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/distributed_c10d.py:1757: in init_process_group + store, rank, world_size = next(rendezvous_iterator) + ^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:278: in _env_rendezvous_handler + store = _create_c10d_store( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +hostname = '127.0.0.1', port = 8889, rank = 0, world_size = 1 +timeout = datetime.timedelta(seconds=1800), use_libuv = True + + def _create_c10d_store( + hostname, port, rank, world_size, timeout, use_libuv=True + ) -> Store: + """ + Smartly creates a c10d Store object on ``rank`` based on whether we need to reuse agent store. + + The TCPStore server is assumed to be hosted + on ``hostname:port``. + + By default, the TCPStore server uses the asynchronous implementation + ``LibUVStoreDaemon`` which utilizes libuv. + + If ``torchelastic_use_agent_store()`` is ``True``, then it is assumed that + the agent leader (node rank 0) hosts the TCPStore server (for which the + endpoint is specified by the given ``hostname:port``). Hence + ALL ranks will create and return a TCPStore client (e.g. ``start_daemon=False``). + + If ``torchelastic_use_agent_store()`` is ``False``, then rank 0 will host + the TCPStore (with multi-tenancy) and it is assumed that rank 0's hostname + and port are correctly passed via ``hostname`` and ``port``. All + non-zero ranks will create and return a TCPStore client. + """ + # check if port is uint16_t + if not 0 <= port < 2**16: + raise ValueError(f"port must have value from 0 to 65535 but was {port}.") + + if _torchelastic_use_agent_store(): + # We create a new TCPStore for every retry so no need to add prefix for each attempt. + return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=False, + timeout=timeout, + ) + else: + start_daemon = rank == 0 +> return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=start_daemon, + timeout=timeout, + multi_tenant=True, + use_libuv=use_libuv, + ) +E torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 8889, useIpv6: false, code: -48, name: EADDRINUSE, message: address already in use + +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:198: DistNetworkError +----------------------------- Captured stdout call ----------------------------- +Distributed data parallel: gloo master at 127.0.0.1:8889 +_____________ pytest_activation_functions_vectoroutput[lrelu_025] ______________ + +activation_function_type = 'lrelu_025', ci_input = 'ci_vectoroutput.json' +overwrite_data = False + + @pytest.mark.parametrize( + "activation_function_type", + ["relu", "selu", "prelu", "elu", "lrelu_01", "lrelu_025", "lrelu_05"], + ) + def pytest_activation_functions_vectoroutput( + activation_function_type, ci_input="ci_vectoroutput.json", overwrite_data=False + ): +> unittest_loss_and_activation_functions( + activation_function_type, "mse", ci_input, overwrite_data + ) + +tests/test_loss_and_activation_functions.py:136: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_loss_and_activation_functions.py:100: in unittest_loss_and_activation_functions + hydragnn.run_training(config) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:80: in _ + world_size, world_rank = setup_ddp(use_deepspeed=use_deepspeed) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/utils/distributed/distributed.py:206: in setup_ddp + dist.init_process_group( +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:81: in wrapper + return func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:95: in wrapper + func_return = func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/distributed_c10d.py:1757: in init_process_group + store, rank, world_size = next(rendezvous_iterator) + ^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:278: in _env_rendezvous_handler + store = _create_c10d_store( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +hostname = '127.0.0.1', port = 8889, rank = 0, world_size = 1 +timeout = datetime.timedelta(seconds=1800), use_libuv = True + + def _create_c10d_store( + hostname, port, rank, world_size, timeout, use_libuv=True + ) -> Store: + """ + Smartly creates a c10d Store object on ``rank`` based on whether we need to reuse agent store. + + The TCPStore server is assumed to be hosted + on ``hostname:port``. + + By default, the TCPStore server uses the asynchronous implementation + ``LibUVStoreDaemon`` which utilizes libuv. + + If ``torchelastic_use_agent_store()`` is ``True``, then it is assumed that + the agent leader (node rank 0) hosts the TCPStore server (for which the + endpoint is specified by the given ``hostname:port``). Hence + ALL ranks will create and return a TCPStore client (e.g. ``start_daemon=False``). + + If ``torchelastic_use_agent_store()`` is ``False``, then rank 0 will host + the TCPStore (with multi-tenancy) and it is assumed that rank 0's hostname + and port are correctly passed via ``hostname`` and ``port``. All + non-zero ranks will create and return a TCPStore client. + """ + # check if port is uint16_t + if not 0 <= port < 2**16: + raise ValueError(f"port must have value from 0 to 65535 but was {port}.") + + if _torchelastic_use_agent_store(): + # We create a new TCPStore for every retry so no need to add prefix for each attempt. + return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=False, + timeout=timeout, + ) + else: + start_daemon = rank == 0 +> return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=start_daemon, + timeout=timeout, + multi_tenant=True, + use_libuv=use_libuv, + ) +E torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 8889, useIpv6: false, code: -48, name: EADDRINUSE, message: address already in use + +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:198: DistNetworkError +----------------------------- Captured stdout call ----------------------------- +Distributed data parallel: gloo master at 127.0.0.1:8889 +______________ pytest_activation_functions_vectoroutput[lrelu_05] ______________ + +activation_function_type = 'lrelu_05', ci_input = 'ci_vectoroutput.json' +overwrite_data = False + + @pytest.mark.parametrize( + "activation_function_type", + ["relu", "selu", "prelu", "elu", "lrelu_01", "lrelu_025", "lrelu_05"], + ) + def pytest_activation_functions_vectoroutput( + activation_function_type, ci_input="ci_vectoroutput.json", overwrite_data=False + ): +> unittest_loss_and_activation_functions( + activation_function_type, "mse", ci_input, overwrite_data + ) + +tests/test_loss_and_activation_functions.py:136: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_loss_and_activation_functions.py:100: in unittest_loss_and_activation_functions + hydragnn.run_training(config) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:80: in _ + world_size, world_rank = setup_ddp(use_deepspeed=use_deepspeed) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/utils/distributed/distributed.py:206: in setup_ddp + dist.init_process_group( +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:81: in wrapper + return func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:95: in wrapper + func_return = func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/distributed_c10d.py:1757: in init_process_group + store, rank, world_size = next(rendezvous_iterator) + ^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:278: in _env_rendezvous_handler + store = _create_c10d_store( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +hostname = '127.0.0.1', port = 8889, rank = 0, world_size = 1 +timeout = datetime.timedelta(seconds=1800), use_libuv = True + + def _create_c10d_store( + hostname, port, rank, world_size, timeout, use_libuv=True + ) -> Store: + """ + Smartly creates a c10d Store object on ``rank`` based on whether we need to reuse agent store. + + The TCPStore server is assumed to be hosted + on ``hostname:port``. + + By default, the TCPStore server uses the asynchronous implementation + ``LibUVStoreDaemon`` which utilizes libuv. + + If ``torchelastic_use_agent_store()`` is ``True``, then it is assumed that + the agent leader (node rank 0) hosts the TCPStore server (for which the + endpoint is specified by the given ``hostname:port``). Hence + ALL ranks will create and return a TCPStore client (e.g. ``start_daemon=False``). + + If ``torchelastic_use_agent_store()`` is ``False``, then rank 0 will host + the TCPStore (with multi-tenancy) and it is assumed that rank 0's hostname + and port are correctly passed via ``hostname`` and ``port``. All + non-zero ranks will create and return a TCPStore client. + """ + # check if port is uint16_t + if not 0 <= port < 2**16: + raise ValueError(f"port must have value from 0 to 65535 but was {port}.") + + if _torchelastic_use_agent_store(): + # We create a new TCPStore for every retry so no need to add prefix for each attempt. + return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=False, + timeout=timeout, + ) + else: + start_daemon = rank == 0 +> return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=start_daemon, + timeout=timeout, + multi_tenant=True, + use_libuv=use_libuv, + ) +E torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 8889, useIpv6: false, code: -48, name: EADDRINUSE, message: address already in use + +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:198: DistNetworkError +----------------------------- Captured stdout call ----------------------------- +Distributed data parallel: gloo master at 127.0.0.1:8889 +____________________________ pytest_model_loadpred _____________________________ + + def pytest_model_loadpred(): + model_type = "PNA" + ci_input = "ci_multihead.json" + config_file = os.path.join(os.getcwd(), "tests/inputs", ci_input) + with open(config_file, "r") as f: + config = json.load(f) + config["NeuralNetwork"]["Architecture"]["model_type"] = model_type + # Update var_config with features before accessing input_node_features + var_config = config["NeuralNetwork"]["Variables_of_interest"] + var_config = update_var_config_with_features(var_config) + config["NeuralNetwork"]["Variables_of_interest"] = var_config + # get the directory of trained model + log_name = hydragnn.utils.input_config_parsing.config_utils.get_log_name_config( + config + ) + modelfile = os.path.join("./logs/", log_name, log_name + ".pk") + # check if pretrained model and pkl datasets files exists + case_exist = True + config_file = os.path.join("./logs/", log_name, "config.json") + if not (os.path.isfile(modelfile) and os.path.isfile(config_file)): + print("Model or configure file not found: ", modelfile, config_file) + case_exist = False + else: + with open(config_file, "r") as f: + config = json.load(f) + # Update var_config after loading saved config + var_config = config["NeuralNetwork"]["Variables_of_interest"] + var_config = update_var_config_with_features(var_config) + config["NeuralNetwork"]["Variables_of_interest"] = var_config + for dataset_name, raw_data_path in config["Dataset"]["path"].items(): + if not os.path.isfile(raw_data_path): + print(dataset_name, "datasets not found: ", raw_data_path) + case_exist = False + break + if not case_exist: + unittest_train_model( + config["NeuralNetwork"]["Architecture"]["model_type"], + "ci_multihead.json", + False, + False, + ) +> unittest_model_prediction(config) + +tests/test_model_loadpred.py:109: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_model_loadpred.py:23: in unittest_model_prediction + _, _ = hydragnn.utils.distributed.setup_ddp() + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/utils/distributed/distributed.py:206: in setup_ddp + dist.init_process_group( +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:81: in wrapper + return func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:95: in wrapper + func_return = func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/distributed_c10d.py:1757: in init_process_group + store, rank, world_size = next(rendezvous_iterator) + ^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:278: in _env_rendezvous_handler + store = _create_c10d_store( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +hostname = '127.0.0.1', port = 8889, rank = 0, world_size = 1 +timeout = datetime.timedelta(seconds=1800), use_libuv = True + + def _create_c10d_store( + hostname, port, rank, world_size, timeout, use_libuv=True + ) -> Store: + """ + Smartly creates a c10d Store object on ``rank`` based on whether we need to reuse agent store. + + The TCPStore server is assumed to be hosted + on ``hostname:port``. + + By default, the TCPStore server uses the asynchronous implementation + ``LibUVStoreDaemon`` which utilizes libuv. + + If ``torchelastic_use_agent_store()`` is ``True``, then it is assumed that + the agent leader (node rank 0) hosts the TCPStore server (for which the + endpoint is specified by the given ``hostname:port``). Hence + ALL ranks will create and return a TCPStore client (e.g. ``start_daemon=False``). + + If ``torchelastic_use_agent_store()`` is ``False``, then rank 0 will host + the TCPStore (with multi-tenancy) and it is assumed that rank 0's hostname + and port are correctly passed via ``hostname`` and ``port``. All + non-zero ranks will create and return a TCPStore client. + """ + # check if port is uint16_t + if not 0 <= port < 2**16: + raise ValueError(f"port must have value from 0 to 65535 but was {port}.") + + if _torchelastic_use_agent_store(): + # We create a new TCPStore for every retry so no need to add prefix for each attempt. + return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=False, + timeout=timeout, + ) + else: + start_daemon = rank == 0 +> return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=start_daemon, + timeout=timeout, + multi_tenant=True, + use_libuv=use_libuv, + ) +E torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 8889, useIpv6: false, code: -48, name: EADDRINUSE, message: address already in use + +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:198: DistNetworkError +----------------------------- Captured stdout call ----------------------------- +Distributed data parallel: gloo master at 127.0.0.1:8889 +_________________________ pytest_optimizers[False-SGD] _________________________ + +optimizer_type = 'SGD', use_zero_redundancy = False, ci_input = 'ci.json' +overwrite_data = False + + @pytest.mark.parametrize( + "optimizer_type", + ["SGD", "Adam", "Adadelta", "Adagrad", "Adamax", "AdamW", "RMSprop"], + ) + @pytest.mark.parametrize( + "use_zero_redundancy", + [False, True], + ) + def pytest_optimizers( + optimizer_type, use_zero_redundancy, ci_input="ci.json", overwrite_data=False + ): +> unittest_optimizers(optimizer_type, use_zero_redundancy, ci_input, overwrite_data) + +tests/test_optimizer.py:110: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_optimizer.py:94: in unittest_optimizers + hydragnn.run_training(config) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:80: in _ + world_size, world_rank = setup_ddp(use_deepspeed=use_deepspeed) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/utils/distributed/distributed.py:206: in setup_ddp + dist.init_process_group( +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:81: in wrapper + return func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:95: in wrapper + func_return = func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/distributed_c10d.py:1757: in init_process_group + store, rank, world_size = next(rendezvous_iterator) + ^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:278: in _env_rendezvous_handler + store = _create_c10d_store( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +hostname = '127.0.0.1', port = 8889, rank = 0, world_size = 1 +timeout = datetime.timedelta(seconds=1800), use_libuv = True + + def _create_c10d_store( + hostname, port, rank, world_size, timeout, use_libuv=True + ) -> Store: + """ + Smartly creates a c10d Store object on ``rank`` based on whether we need to reuse agent store. + + The TCPStore server is assumed to be hosted + on ``hostname:port``. + + By default, the TCPStore server uses the asynchronous implementation + ``LibUVStoreDaemon`` which utilizes libuv. + + If ``torchelastic_use_agent_store()`` is ``True``, then it is assumed that + the agent leader (node rank 0) hosts the TCPStore server (for which the + endpoint is specified by the given ``hostname:port``). Hence + ALL ranks will create and return a TCPStore client (e.g. ``start_daemon=False``). + + If ``torchelastic_use_agent_store()`` is ``False``, then rank 0 will host + the TCPStore (with multi-tenancy) and it is assumed that rank 0's hostname + and port are correctly passed via ``hostname`` and ``port``. All + non-zero ranks will create and return a TCPStore client. + """ + # check if port is uint16_t + if not 0 <= port < 2**16: + raise ValueError(f"port must have value from 0 to 65535 but was {port}.") + + if _torchelastic_use_agent_store(): + # We create a new TCPStore for every retry so no need to add prefix for each attempt. + return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=False, + timeout=timeout, + ) + else: + start_daemon = rank == 0 +> return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=start_daemon, + timeout=timeout, + multi_tenant=True, + use_libuv=use_libuv, + ) +E torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 8889, useIpv6: false, code: -48, name: EADDRINUSE, message: address already in use + +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:198: DistNetworkError +----------------------------- Captured stdout call ----------------------------- +Distributed data parallel: gloo master at 127.0.0.1:8889 +________________________ pytest_optimizers[False-Adam] _________________________ + +optimizer_type = 'Adam', use_zero_redundancy = False, ci_input = 'ci.json' +overwrite_data = False + + @pytest.mark.parametrize( + "optimizer_type", + ["SGD", "Adam", "Adadelta", "Adagrad", "Adamax", "AdamW", "RMSprop"], + ) + @pytest.mark.parametrize( + "use_zero_redundancy", + [False, True], + ) + def pytest_optimizers( + optimizer_type, use_zero_redundancy, ci_input="ci.json", overwrite_data=False + ): +> unittest_optimizers(optimizer_type, use_zero_redundancy, ci_input, overwrite_data) + +tests/test_optimizer.py:110: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_optimizer.py:94: in unittest_optimizers + hydragnn.run_training(config) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:80: in _ + world_size, world_rank = setup_ddp(use_deepspeed=use_deepspeed) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/utils/distributed/distributed.py:206: in setup_ddp + dist.init_process_group( +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:81: in wrapper + return func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:95: in wrapper + func_return = func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/distributed_c10d.py:1757: in init_process_group + store, rank, world_size = next(rendezvous_iterator) + ^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:278: in _env_rendezvous_handler + store = _create_c10d_store( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +hostname = '127.0.0.1', port = 8889, rank = 0, world_size = 1 +timeout = datetime.timedelta(seconds=1800), use_libuv = True + + def _create_c10d_store( + hostname, port, rank, world_size, timeout, use_libuv=True + ) -> Store: + """ + Smartly creates a c10d Store object on ``rank`` based on whether we need to reuse agent store. + + The TCPStore server is assumed to be hosted + on ``hostname:port``. + + By default, the TCPStore server uses the asynchronous implementation + ``LibUVStoreDaemon`` which utilizes libuv. + + If ``torchelastic_use_agent_store()`` is ``True``, then it is assumed that + the agent leader (node rank 0) hosts the TCPStore server (for which the + endpoint is specified by the given ``hostname:port``). Hence + ALL ranks will create and return a TCPStore client (e.g. ``start_daemon=False``). + + If ``torchelastic_use_agent_store()`` is ``False``, then rank 0 will host + the TCPStore (with multi-tenancy) and it is assumed that rank 0's hostname + and port are correctly passed via ``hostname`` and ``port``. All + non-zero ranks will create and return a TCPStore client. + """ + # check if port is uint16_t + if not 0 <= port < 2**16: + raise ValueError(f"port must have value from 0 to 65535 but was {port}.") + + if _torchelastic_use_agent_store(): + # We create a new TCPStore for every retry so no need to add prefix for each attempt. + return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=False, + timeout=timeout, + ) + else: + start_daemon = rank == 0 +> return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=start_daemon, + timeout=timeout, + multi_tenant=True, + use_libuv=use_libuv, + ) +E torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 8889, useIpv6: false, code: -48, name: EADDRINUSE, message: address already in use + +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:198: DistNetworkError +----------------------------- Captured stdout call ----------------------------- +Distributed data parallel: gloo master at 127.0.0.1:8889 +______________________ pytest_optimizers[False-Adadelta] _______________________ + +optimizer_type = 'Adadelta', use_zero_redundancy = False, ci_input = 'ci.json' +overwrite_data = False + + @pytest.mark.parametrize( + "optimizer_type", + ["SGD", "Adam", "Adadelta", "Adagrad", "Adamax", "AdamW", "RMSprop"], + ) + @pytest.mark.parametrize( + "use_zero_redundancy", + [False, True], + ) + def pytest_optimizers( + optimizer_type, use_zero_redundancy, ci_input="ci.json", overwrite_data=False + ): +> unittest_optimizers(optimizer_type, use_zero_redundancy, ci_input, overwrite_data) + +tests/test_optimizer.py:110: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_optimizer.py:94: in unittest_optimizers + hydragnn.run_training(config) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:80: in _ + world_size, world_rank = setup_ddp(use_deepspeed=use_deepspeed) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/utils/distributed/distributed.py:206: in setup_ddp + dist.init_process_group( +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:81: in wrapper + return func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:95: in wrapper + func_return = func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/distributed_c10d.py:1757: in init_process_group + store, rank, world_size = next(rendezvous_iterator) + ^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:278: in _env_rendezvous_handler + store = _create_c10d_store( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +hostname = '127.0.0.1', port = 8889, rank = 0, world_size = 1 +timeout = datetime.timedelta(seconds=1800), use_libuv = True + + def _create_c10d_store( + hostname, port, rank, world_size, timeout, use_libuv=True + ) -> Store: + """ + Smartly creates a c10d Store object on ``rank`` based on whether we need to reuse agent store. + + The TCPStore server is assumed to be hosted + on ``hostname:port``. + + By default, the TCPStore server uses the asynchronous implementation + ``LibUVStoreDaemon`` which utilizes libuv. + + If ``torchelastic_use_agent_store()`` is ``True``, then it is assumed that + the agent leader (node rank 0) hosts the TCPStore server (for which the + endpoint is specified by the given ``hostname:port``). Hence + ALL ranks will create and return a TCPStore client (e.g. ``start_daemon=False``). + + If ``torchelastic_use_agent_store()`` is ``False``, then rank 0 will host + the TCPStore (with multi-tenancy) and it is assumed that rank 0's hostname + and port are correctly passed via ``hostname`` and ``port``. All + non-zero ranks will create and return a TCPStore client. + """ + # check if port is uint16_t + if not 0 <= port < 2**16: + raise ValueError(f"port must have value from 0 to 65535 but was {port}.") + + if _torchelastic_use_agent_store(): + # We create a new TCPStore for every retry so no need to add prefix for each attempt. + return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=False, + timeout=timeout, + ) + else: + start_daemon = rank == 0 +> return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=start_daemon, + timeout=timeout, + multi_tenant=True, + use_libuv=use_libuv, + ) +E torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 8889, useIpv6: false, code: -48, name: EADDRINUSE, message: address already in use + +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:198: DistNetworkError +----------------------------- Captured stdout call ----------------------------- +Distributed data parallel: gloo master at 127.0.0.1:8889 +_______________________ pytest_optimizers[False-Adagrad] _______________________ + +optimizer_type = 'Adagrad', use_zero_redundancy = False, ci_input = 'ci.json' +overwrite_data = False + + @pytest.mark.parametrize( + "optimizer_type", + ["SGD", "Adam", "Adadelta", "Adagrad", "Adamax", "AdamW", "RMSprop"], + ) + @pytest.mark.parametrize( + "use_zero_redundancy", + [False, True], + ) + def pytest_optimizers( + optimizer_type, use_zero_redundancy, ci_input="ci.json", overwrite_data=False + ): +> unittest_optimizers(optimizer_type, use_zero_redundancy, ci_input, overwrite_data) + +tests/test_optimizer.py:110: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_optimizer.py:94: in unittest_optimizers + hydragnn.run_training(config) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:80: in _ + world_size, world_rank = setup_ddp(use_deepspeed=use_deepspeed) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/utils/distributed/distributed.py:206: in setup_ddp + dist.init_process_group( +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:81: in wrapper + return func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:95: in wrapper + func_return = func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/distributed_c10d.py:1757: in init_process_group + store, rank, world_size = next(rendezvous_iterator) + ^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:278: in _env_rendezvous_handler + store = _create_c10d_store( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +hostname = '127.0.0.1', port = 8889, rank = 0, world_size = 1 +timeout = datetime.timedelta(seconds=1800), use_libuv = True + + def _create_c10d_store( + hostname, port, rank, world_size, timeout, use_libuv=True + ) -> Store: + """ + Smartly creates a c10d Store object on ``rank`` based on whether we need to reuse agent store. + + The TCPStore server is assumed to be hosted + on ``hostname:port``. + + By default, the TCPStore server uses the asynchronous implementation + ``LibUVStoreDaemon`` which utilizes libuv. + + If ``torchelastic_use_agent_store()`` is ``True``, then it is assumed that + the agent leader (node rank 0) hosts the TCPStore server (for which the + endpoint is specified by the given ``hostname:port``). Hence + ALL ranks will create and return a TCPStore client (e.g. ``start_daemon=False``). + + If ``torchelastic_use_agent_store()`` is ``False``, then rank 0 will host + the TCPStore (with multi-tenancy) and it is assumed that rank 0's hostname + and port are correctly passed via ``hostname`` and ``port``. All + non-zero ranks will create and return a TCPStore client. + """ + # check if port is uint16_t + if not 0 <= port < 2**16: + raise ValueError(f"port must have value from 0 to 65535 but was {port}.") + + if _torchelastic_use_agent_store(): + # We create a new TCPStore for every retry so no need to add prefix for each attempt. + return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=False, + timeout=timeout, + ) + else: + start_daemon = rank == 0 +> return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=start_daemon, + timeout=timeout, + multi_tenant=True, + use_libuv=use_libuv, + ) +E torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 8889, useIpv6: false, code: -48, name: EADDRINUSE, message: address already in use + +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:198: DistNetworkError +----------------------------- Captured stdout call ----------------------------- +Distributed data parallel: gloo master at 127.0.0.1:8889 +_______________________ pytest_optimizers[False-Adamax] ________________________ + +optimizer_type = 'Adamax', use_zero_redundancy = False, ci_input = 'ci.json' +overwrite_data = False + + @pytest.mark.parametrize( + "optimizer_type", + ["SGD", "Adam", "Adadelta", "Adagrad", "Adamax", "AdamW", "RMSprop"], + ) + @pytest.mark.parametrize( + "use_zero_redundancy", + [False, True], + ) + def pytest_optimizers( + optimizer_type, use_zero_redundancy, ci_input="ci.json", overwrite_data=False + ): +> unittest_optimizers(optimizer_type, use_zero_redundancy, ci_input, overwrite_data) + +tests/test_optimizer.py:110: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_optimizer.py:94: in unittest_optimizers + hydragnn.run_training(config) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:80: in _ + world_size, world_rank = setup_ddp(use_deepspeed=use_deepspeed) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/utils/distributed/distributed.py:206: in setup_ddp + dist.init_process_group( +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:81: in wrapper + return func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:95: in wrapper + func_return = func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/distributed_c10d.py:1757: in init_process_group + store, rank, world_size = next(rendezvous_iterator) + ^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:278: in _env_rendezvous_handler + store = _create_c10d_store( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +hostname = '127.0.0.1', port = 8889, rank = 0, world_size = 1 +timeout = datetime.timedelta(seconds=1800), use_libuv = True + + def _create_c10d_store( + hostname, port, rank, world_size, timeout, use_libuv=True + ) -> Store: + """ + Smartly creates a c10d Store object on ``rank`` based on whether we need to reuse agent store. + + The TCPStore server is assumed to be hosted + on ``hostname:port``. + + By default, the TCPStore server uses the asynchronous implementation + ``LibUVStoreDaemon`` which utilizes libuv. + + If ``torchelastic_use_agent_store()`` is ``True``, then it is assumed that + the agent leader (node rank 0) hosts the TCPStore server (for which the + endpoint is specified by the given ``hostname:port``). Hence + ALL ranks will create and return a TCPStore client (e.g. ``start_daemon=False``). + + If ``torchelastic_use_agent_store()`` is ``False``, then rank 0 will host + the TCPStore (with multi-tenancy) and it is assumed that rank 0's hostname + and port are correctly passed via ``hostname`` and ``port``. All + non-zero ranks will create and return a TCPStore client. + """ + # check if port is uint16_t + if not 0 <= port < 2**16: + raise ValueError(f"port must have value from 0 to 65535 but was {port}.") + + if _torchelastic_use_agent_store(): + # We create a new TCPStore for every retry so no need to add prefix for each attempt. + return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=False, + timeout=timeout, + ) + else: + start_daemon = rank == 0 +> return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=start_daemon, + timeout=timeout, + multi_tenant=True, + use_libuv=use_libuv, + ) +E torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 8889, useIpv6: false, code: -48, name: EADDRINUSE, message: address already in use + +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:198: DistNetworkError +----------------------------- Captured stdout call ----------------------------- +Distributed data parallel: gloo master at 127.0.0.1:8889 +________________________ pytest_optimizers[False-AdamW] ________________________ + +optimizer_type = 'AdamW', use_zero_redundancy = False, ci_input = 'ci.json' +overwrite_data = False + + @pytest.mark.parametrize( + "optimizer_type", + ["SGD", "Adam", "Adadelta", "Adagrad", "Adamax", "AdamW", "RMSprop"], + ) + @pytest.mark.parametrize( + "use_zero_redundancy", + [False, True], + ) + def pytest_optimizers( + optimizer_type, use_zero_redundancy, ci_input="ci.json", overwrite_data=False + ): +> unittest_optimizers(optimizer_type, use_zero_redundancy, ci_input, overwrite_data) + +tests/test_optimizer.py:110: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_optimizer.py:94: in unittest_optimizers + hydragnn.run_training(config) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:80: in _ + world_size, world_rank = setup_ddp(use_deepspeed=use_deepspeed) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/utils/distributed/distributed.py:206: in setup_ddp + dist.init_process_group( +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:81: in wrapper + return func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:95: in wrapper + func_return = func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/distributed_c10d.py:1757: in init_process_group + store, rank, world_size = next(rendezvous_iterator) + ^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:278: in _env_rendezvous_handler + store = _create_c10d_store( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +hostname = '127.0.0.1', port = 8889, rank = 0, world_size = 1 +timeout = datetime.timedelta(seconds=1800), use_libuv = True + + def _create_c10d_store( + hostname, port, rank, world_size, timeout, use_libuv=True + ) -> Store: + """ + Smartly creates a c10d Store object on ``rank`` based on whether we need to reuse agent store. + + The TCPStore server is assumed to be hosted + on ``hostname:port``. + + By default, the TCPStore server uses the asynchronous implementation + ``LibUVStoreDaemon`` which utilizes libuv. + + If ``torchelastic_use_agent_store()`` is ``True``, then it is assumed that + the agent leader (node rank 0) hosts the TCPStore server (for which the + endpoint is specified by the given ``hostname:port``). Hence + ALL ranks will create and return a TCPStore client (e.g. ``start_daemon=False``). + + If ``torchelastic_use_agent_store()`` is ``False``, then rank 0 will host + the TCPStore (with multi-tenancy) and it is assumed that rank 0's hostname + and port are correctly passed via ``hostname`` and ``port``. All + non-zero ranks will create and return a TCPStore client. + """ + # check if port is uint16_t + if not 0 <= port < 2**16: + raise ValueError(f"port must have value from 0 to 65535 but was {port}.") + + if _torchelastic_use_agent_store(): + # We create a new TCPStore for every retry so no need to add prefix for each attempt. + return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=False, + timeout=timeout, + ) + else: + start_daemon = rank == 0 +> return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=start_daemon, + timeout=timeout, + multi_tenant=True, + use_libuv=use_libuv, + ) +E torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 8889, useIpv6: false, code: -48, name: EADDRINUSE, message: address already in use + +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:198: DistNetworkError +----------------------------- Captured stdout call ----------------------------- +Distributed data parallel: gloo master at 127.0.0.1:8889 +_______________________ pytest_optimizers[False-RMSprop] _______________________ + +optimizer_type = 'RMSprop', use_zero_redundancy = False, ci_input = 'ci.json' +overwrite_data = False + + @pytest.mark.parametrize( + "optimizer_type", + ["SGD", "Adam", "Adadelta", "Adagrad", "Adamax", "AdamW", "RMSprop"], + ) + @pytest.mark.parametrize( + "use_zero_redundancy", + [False, True], + ) + def pytest_optimizers( + optimizer_type, use_zero_redundancy, ci_input="ci.json", overwrite_data=False + ): +> unittest_optimizers(optimizer_type, use_zero_redundancy, ci_input, overwrite_data) + +tests/test_optimizer.py:110: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_optimizer.py:94: in unittest_optimizers + hydragnn.run_training(config) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:80: in _ + world_size, world_rank = setup_ddp(use_deepspeed=use_deepspeed) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/utils/distributed/distributed.py:206: in setup_ddp + dist.init_process_group( +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:81: in wrapper + return func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:95: in wrapper + func_return = func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/distributed_c10d.py:1757: in init_process_group + store, rank, world_size = next(rendezvous_iterator) + ^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:278: in _env_rendezvous_handler + store = _create_c10d_store( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +hostname = '127.0.0.1', port = 8889, rank = 0, world_size = 1 +timeout = datetime.timedelta(seconds=1800), use_libuv = True + + def _create_c10d_store( + hostname, port, rank, world_size, timeout, use_libuv=True + ) -> Store: + """ + Smartly creates a c10d Store object on ``rank`` based on whether we need to reuse agent store. + + The TCPStore server is assumed to be hosted + on ``hostname:port``. + + By default, the TCPStore server uses the asynchronous implementation + ``LibUVStoreDaemon`` which utilizes libuv. + + If ``torchelastic_use_agent_store()`` is ``True``, then it is assumed that + the agent leader (node rank 0) hosts the TCPStore server (for which the + endpoint is specified by the given ``hostname:port``). Hence + ALL ranks will create and return a TCPStore client (e.g. ``start_daemon=False``). + + If ``torchelastic_use_agent_store()`` is ``False``, then rank 0 will host + the TCPStore (with multi-tenancy) and it is assumed that rank 0's hostname + and port are correctly passed via ``hostname`` and ``port``. All + non-zero ranks will create and return a TCPStore client. + """ + # check if port is uint16_t + if not 0 <= port < 2**16: + raise ValueError(f"port must have value from 0 to 65535 but was {port}.") + + if _torchelastic_use_agent_store(): + # We create a new TCPStore for every retry so no need to add prefix for each attempt. + return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=False, + timeout=timeout, + ) + else: + start_daemon = rank == 0 +> return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=start_daemon, + timeout=timeout, + multi_tenant=True, + use_libuv=use_libuv, + ) +E torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 8889, useIpv6: false, code: -48, name: EADDRINUSE, message: address already in use + +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:198: DistNetworkError +----------------------------- Captured stdout call ----------------------------- +Distributed data parallel: gloo master at 127.0.0.1:8889 +_________________________ pytest_optimizers[True-SGD] __________________________ + +optimizer_type = 'SGD', use_zero_redundancy = True, ci_input = 'ci.json' +overwrite_data = False + + @pytest.mark.parametrize( + "optimizer_type", + ["SGD", "Adam", "Adadelta", "Adagrad", "Adamax", "AdamW", "RMSprop"], + ) + @pytest.mark.parametrize( + "use_zero_redundancy", + [False, True], + ) + def pytest_optimizers( + optimizer_type, use_zero_redundancy, ci_input="ci.json", overwrite_data=False + ): +> unittest_optimizers(optimizer_type, use_zero_redundancy, ci_input, overwrite_data) + +tests/test_optimizer.py:110: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_optimizer.py:94: in unittest_optimizers + hydragnn.run_training(config) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:80: in _ + world_size, world_rank = setup_ddp(use_deepspeed=use_deepspeed) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/utils/distributed/distributed.py:206: in setup_ddp + dist.init_process_group( +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:81: in wrapper + return func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:95: in wrapper + func_return = func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/distributed_c10d.py:1757: in init_process_group + store, rank, world_size = next(rendezvous_iterator) + ^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:278: in _env_rendezvous_handler + store = _create_c10d_store( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +hostname = '127.0.0.1', port = 8889, rank = 0, world_size = 1 +timeout = datetime.timedelta(seconds=1800), use_libuv = True + + def _create_c10d_store( + hostname, port, rank, world_size, timeout, use_libuv=True + ) -> Store: + """ + Smartly creates a c10d Store object on ``rank`` based on whether we need to reuse agent store. + + The TCPStore server is assumed to be hosted + on ``hostname:port``. + + By default, the TCPStore server uses the asynchronous implementation + ``LibUVStoreDaemon`` which utilizes libuv. + + If ``torchelastic_use_agent_store()`` is ``True``, then it is assumed that + the agent leader (node rank 0) hosts the TCPStore server (for which the + endpoint is specified by the given ``hostname:port``). Hence + ALL ranks will create and return a TCPStore client (e.g. ``start_daemon=False``). + + If ``torchelastic_use_agent_store()`` is ``False``, then rank 0 will host + the TCPStore (with multi-tenancy) and it is assumed that rank 0's hostname + and port are correctly passed via ``hostname`` and ``port``. All + non-zero ranks will create and return a TCPStore client. + """ + # check if port is uint16_t + if not 0 <= port < 2**16: + raise ValueError(f"port must have value from 0 to 65535 but was {port}.") + + if _torchelastic_use_agent_store(): + # We create a new TCPStore for every retry so no need to add prefix for each attempt. + return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=False, + timeout=timeout, + ) + else: + start_daemon = rank == 0 +> return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=start_daemon, + timeout=timeout, + multi_tenant=True, + use_libuv=use_libuv, + ) +E torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 8889, useIpv6: false, code: -48, name: EADDRINUSE, message: address already in use + +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:198: DistNetworkError +----------------------------- Captured stdout call ----------------------------- +Distributed data parallel: gloo master at 127.0.0.1:8889 +_________________________ pytest_optimizers[True-Adam] _________________________ + +optimizer_type = 'Adam', use_zero_redundancy = True, ci_input = 'ci.json' +overwrite_data = False + + @pytest.mark.parametrize( + "optimizer_type", + ["SGD", "Adam", "Adadelta", "Adagrad", "Adamax", "AdamW", "RMSprop"], + ) + @pytest.mark.parametrize( + "use_zero_redundancy", + [False, True], + ) + def pytest_optimizers( + optimizer_type, use_zero_redundancy, ci_input="ci.json", overwrite_data=False + ): +> unittest_optimizers(optimizer_type, use_zero_redundancy, ci_input, overwrite_data) + +tests/test_optimizer.py:110: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_optimizer.py:94: in unittest_optimizers + hydragnn.run_training(config) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:80: in _ + world_size, world_rank = setup_ddp(use_deepspeed=use_deepspeed) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/utils/distributed/distributed.py:206: in setup_ddp + dist.init_process_group( +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:81: in wrapper + return func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:95: in wrapper + func_return = func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/distributed_c10d.py:1757: in init_process_group + store, rank, world_size = next(rendezvous_iterator) + ^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:278: in _env_rendezvous_handler + store = _create_c10d_store( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +hostname = '127.0.0.1', port = 8889, rank = 0, world_size = 1 +timeout = datetime.timedelta(seconds=1800), use_libuv = True + + def _create_c10d_store( + hostname, port, rank, world_size, timeout, use_libuv=True + ) -> Store: + """ + Smartly creates a c10d Store object on ``rank`` based on whether we need to reuse agent store. + + The TCPStore server is assumed to be hosted + on ``hostname:port``. + + By default, the TCPStore server uses the asynchronous implementation + ``LibUVStoreDaemon`` which utilizes libuv. + + If ``torchelastic_use_agent_store()`` is ``True``, then it is assumed that + the agent leader (node rank 0) hosts the TCPStore server (for which the + endpoint is specified by the given ``hostname:port``). Hence + ALL ranks will create and return a TCPStore client (e.g. ``start_daemon=False``). + + If ``torchelastic_use_agent_store()`` is ``False``, then rank 0 will host + the TCPStore (with multi-tenancy) and it is assumed that rank 0's hostname + and port are correctly passed via ``hostname`` and ``port``. All + non-zero ranks will create and return a TCPStore client. + """ + # check if port is uint16_t + if not 0 <= port < 2**16: + raise ValueError(f"port must have value from 0 to 65535 but was {port}.") + + if _torchelastic_use_agent_store(): + # We create a new TCPStore for every retry so no need to add prefix for each attempt. + return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=False, + timeout=timeout, + ) + else: + start_daemon = rank == 0 +> return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=start_daemon, + timeout=timeout, + multi_tenant=True, + use_libuv=use_libuv, + ) +E torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 8889, useIpv6: false, code: -48, name: EADDRINUSE, message: address already in use + +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:198: DistNetworkError +----------------------------- Captured stdout call ----------------------------- +Distributed data parallel: gloo master at 127.0.0.1:8889 +_______________________ pytest_optimizers[True-Adadelta] _______________________ + +optimizer_type = 'Adadelta', use_zero_redundancy = True, ci_input = 'ci.json' +overwrite_data = False + + @pytest.mark.parametrize( + "optimizer_type", + ["SGD", "Adam", "Adadelta", "Adagrad", "Adamax", "AdamW", "RMSprop"], + ) + @pytest.mark.parametrize( + "use_zero_redundancy", + [False, True], + ) + def pytest_optimizers( + optimizer_type, use_zero_redundancy, ci_input="ci.json", overwrite_data=False + ): +> unittest_optimizers(optimizer_type, use_zero_redundancy, ci_input, overwrite_data) + +tests/test_optimizer.py:110: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_optimizer.py:94: in unittest_optimizers + hydragnn.run_training(config) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:80: in _ + world_size, world_rank = setup_ddp(use_deepspeed=use_deepspeed) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/utils/distributed/distributed.py:206: in setup_ddp + dist.init_process_group( +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:81: in wrapper + return func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:95: in wrapper + func_return = func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/distributed_c10d.py:1757: in init_process_group + store, rank, world_size = next(rendezvous_iterator) + ^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:278: in _env_rendezvous_handler + store = _create_c10d_store( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +hostname = '127.0.0.1', port = 8889, rank = 0, world_size = 1 +timeout = datetime.timedelta(seconds=1800), use_libuv = True + + def _create_c10d_store( + hostname, port, rank, world_size, timeout, use_libuv=True + ) -> Store: + """ + Smartly creates a c10d Store object on ``rank`` based on whether we need to reuse agent store. + + The TCPStore server is assumed to be hosted + on ``hostname:port``. + + By default, the TCPStore server uses the asynchronous implementation + ``LibUVStoreDaemon`` which utilizes libuv. + + If ``torchelastic_use_agent_store()`` is ``True``, then it is assumed that + the agent leader (node rank 0) hosts the TCPStore server (for which the + endpoint is specified by the given ``hostname:port``). Hence + ALL ranks will create and return a TCPStore client (e.g. ``start_daemon=False``). + + If ``torchelastic_use_agent_store()`` is ``False``, then rank 0 will host + the TCPStore (with multi-tenancy) and it is assumed that rank 0's hostname + and port are correctly passed via ``hostname`` and ``port``. All + non-zero ranks will create and return a TCPStore client. + """ + # check if port is uint16_t + if not 0 <= port < 2**16: + raise ValueError(f"port must have value from 0 to 65535 but was {port}.") + + if _torchelastic_use_agent_store(): + # We create a new TCPStore for every retry so no need to add prefix for each attempt. + return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=False, + timeout=timeout, + ) + else: + start_daemon = rank == 0 +> return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=start_daemon, + timeout=timeout, + multi_tenant=True, + use_libuv=use_libuv, + ) +E torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 8889, useIpv6: false, code: -48, name: EADDRINUSE, message: address already in use + +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:198: DistNetworkError +----------------------------- Captured stdout call ----------------------------- +Distributed data parallel: gloo master at 127.0.0.1:8889 +_______________________ pytest_optimizers[True-Adagrad] ________________________ + +optimizer_type = 'Adagrad', use_zero_redundancy = True, ci_input = 'ci.json' +overwrite_data = False + + @pytest.mark.parametrize( + "optimizer_type", + ["SGD", "Adam", "Adadelta", "Adagrad", "Adamax", "AdamW", "RMSprop"], + ) + @pytest.mark.parametrize( + "use_zero_redundancy", + [False, True], + ) + def pytest_optimizers( + optimizer_type, use_zero_redundancy, ci_input="ci.json", overwrite_data=False + ): +> unittest_optimizers(optimizer_type, use_zero_redundancy, ci_input, overwrite_data) + +tests/test_optimizer.py:110: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_optimizer.py:94: in unittest_optimizers + hydragnn.run_training(config) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:80: in _ + world_size, world_rank = setup_ddp(use_deepspeed=use_deepspeed) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/utils/distributed/distributed.py:206: in setup_ddp + dist.init_process_group( +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:81: in wrapper + return func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:95: in wrapper + func_return = func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/distributed_c10d.py:1757: in init_process_group + store, rank, world_size = next(rendezvous_iterator) + ^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:278: in _env_rendezvous_handler + store = _create_c10d_store( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +hostname = '127.0.0.1', port = 8889, rank = 0, world_size = 1 +timeout = datetime.timedelta(seconds=1800), use_libuv = True + + def _create_c10d_store( + hostname, port, rank, world_size, timeout, use_libuv=True + ) -> Store: + """ + Smartly creates a c10d Store object on ``rank`` based on whether we need to reuse agent store. + + The TCPStore server is assumed to be hosted + on ``hostname:port``. + + By default, the TCPStore server uses the asynchronous implementation + ``LibUVStoreDaemon`` which utilizes libuv. + + If ``torchelastic_use_agent_store()`` is ``True``, then it is assumed that + the agent leader (node rank 0) hosts the TCPStore server (for which the + endpoint is specified by the given ``hostname:port``). Hence + ALL ranks will create and return a TCPStore client (e.g. ``start_daemon=False``). + + If ``torchelastic_use_agent_store()`` is ``False``, then rank 0 will host + the TCPStore (with multi-tenancy) and it is assumed that rank 0's hostname + and port are correctly passed via ``hostname`` and ``port``. All + non-zero ranks will create and return a TCPStore client. + """ + # check if port is uint16_t + if not 0 <= port < 2**16: + raise ValueError(f"port must have value from 0 to 65535 but was {port}.") + + if _torchelastic_use_agent_store(): + # We create a new TCPStore for every retry so no need to add prefix for each attempt. + return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=False, + timeout=timeout, + ) + else: + start_daemon = rank == 0 +> return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=start_daemon, + timeout=timeout, + multi_tenant=True, + use_libuv=use_libuv, + ) +E torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 8889, useIpv6: false, code: -48, name: EADDRINUSE, message: address already in use + +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:198: DistNetworkError +----------------------------- Captured stdout call ----------------------------- +Distributed data parallel: gloo master at 127.0.0.1:8889 +________________________ pytest_optimizers[True-Adamax] ________________________ + +optimizer_type = 'Adamax', use_zero_redundancy = True, ci_input = 'ci.json' +overwrite_data = False + + @pytest.mark.parametrize( + "optimizer_type", + ["SGD", "Adam", "Adadelta", "Adagrad", "Adamax", "AdamW", "RMSprop"], + ) + @pytest.mark.parametrize( + "use_zero_redundancy", + [False, True], + ) + def pytest_optimizers( + optimizer_type, use_zero_redundancy, ci_input="ci.json", overwrite_data=False + ): +> unittest_optimizers(optimizer_type, use_zero_redundancy, ci_input, overwrite_data) + +tests/test_optimizer.py:110: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_optimizer.py:94: in unittest_optimizers + hydragnn.run_training(config) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:80: in _ + world_size, world_rank = setup_ddp(use_deepspeed=use_deepspeed) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/utils/distributed/distributed.py:206: in setup_ddp + dist.init_process_group( +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:81: in wrapper + return func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:95: in wrapper + func_return = func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/distributed_c10d.py:1757: in init_process_group + store, rank, world_size = next(rendezvous_iterator) + ^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:278: in _env_rendezvous_handler + store = _create_c10d_store( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +hostname = '127.0.0.1', port = 8889, rank = 0, world_size = 1 +timeout = datetime.timedelta(seconds=1800), use_libuv = True + + def _create_c10d_store( + hostname, port, rank, world_size, timeout, use_libuv=True + ) -> Store: + """ + Smartly creates a c10d Store object on ``rank`` based on whether we need to reuse agent store. + + The TCPStore server is assumed to be hosted + on ``hostname:port``. + + By default, the TCPStore server uses the asynchronous implementation + ``LibUVStoreDaemon`` which utilizes libuv. + + If ``torchelastic_use_agent_store()`` is ``True``, then it is assumed that + the agent leader (node rank 0) hosts the TCPStore server (for which the + endpoint is specified by the given ``hostname:port``). Hence + ALL ranks will create and return a TCPStore client (e.g. ``start_daemon=False``). + + If ``torchelastic_use_agent_store()`` is ``False``, then rank 0 will host + the TCPStore (with multi-tenancy) and it is assumed that rank 0's hostname + and port are correctly passed via ``hostname`` and ``port``. All + non-zero ranks will create and return a TCPStore client. + """ + # check if port is uint16_t + if not 0 <= port < 2**16: + raise ValueError(f"port must have value from 0 to 65535 but was {port}.") + + if _torchelastic_use_agent_store(): + # We create a new TCPStore for every retry so no need to add prefix for each attempt. + return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=False, + timeout=timeout, + ) + else: + start_daemon = rank == 0 +> return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=start_daemon, + timeout=timeout, + multi_tenant=True, + use_libuv=use_libuv, + ) +E torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 8889, useIpv6: false, code: -48, name: EADDRINUSE, message: address already in use + +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:198: DistNetworkError +----------------------------- Captured stdout call ----------------------------- +Distributed data parallel: gloo master at 127.0.0.1:8889 +________________________ pytest_optimizers[True-AdamW] _________________________ + +optimizer_type = 'AdamW', use_zero_redundancy = True, ci_input = 'ci.json' +overwrite_data = False + + @pytest.mark.parametrize( + "optimizer_type", + ["SGD", "Adam", "Adadelta", "Adagrad", "Adamax", "AdamW", "RMSprop"], + ) + @pytest.mark.parametrize( + "use_zero_redundancy", + [False, True], + ) + def pytest_optimizers( + optimizer_type, use_zero_redundancy, ci_input="ci.json", overwrite_data=False + ): +> unittest_optimizers(optimizer_type, use_zero_redundancy, ci_input, overwrite_data) + +tests/test_optimizer.py:110: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_optimizer.py:94: in unittest_optimizers + hydragnn.run_training(config) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:80: in _ + world_size, world_rank = setup_ddp(use_deepspeed=use_deepspeed) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/utils/distributed/distributed.py:206: in setup_ddp + dist.init_process_group( +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:81: in wrapper + return func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:95: in wrapper + func_return = func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/distributed_c10d.py:1757: in init_process_group + store, rank, world_size = next(rendezvous_iterator) + ^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:278: in _env_rendezvous_handler + store = _create_c10d_store( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +hostname = '127.0.0.1', port = 8889, rank = 0, world_size = 1 +timeout = datetime.timedelta(seconds=1800), use_libuv = True + + def _create_c10d_store( + hostname, port, rank, world_size, timeout, use_libuv=True + ) -> Store: + """ + Smartly creates a c10d Store object on ``rank`` based on whether we need to reuse agent store. + + The TCPStore server is assumed to be hosted + on ``hostname:port``. + + By default, the TCPStore server uses the asynchronous implementation + ``LibUVStoreDaemon`` which utilizes libuv. + + If ``torchelastic_use_agent_store()`` is ``True``, then it is assumed that + the agent leader (node rank 0) hosts the TCPStore server (for which the + endpoint is specified by the given ``hostname:port``). Hence + ALL ranks will create and return a TCPStore client (e.g. ``start_daemon=False``). + + If ``torchelastic_use_agent_store()`` is ``False``, then rank 0 will host + the TCPStore (with multi-tenancy) and it is assumed that rank 0's hostname + and port are correctly passed via ``hostname`` and ``port``. All + non-zero ranks will create and return a TCPStore client. + """ + # check if port is uint16_t + if not 0 <= port < 2**16: + raise ValueError(f"port must have value from 0 to 65535 but was {port}.") + + if _torchelastic_use_agent_store(): + # We create a new TCPStore for every retry so no need to add prefix for each attempt. + return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=False, + timeout=timeout, + ) + else: + start_daemon = rank == 0 +> return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=start_daemon, + timeout=timeout, + multi_tenant=True, + use_libuv=use_libuv, + ) +E torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 8889, useIpv6: false, code: -48, name: EADDRINUSE, message: address already in use + +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:198: DistNetworkError +----------------------------- Captured stdout call ----------------------------- +Distributed data parallel: gloo master at 127.0.0.1:8889 +_______________________ pytest_optimizers[True-RMSprop] ________________________ + +optimizer_type = 'RMSprop', use_zero_redundancy = True, ci_input = 'ci.json' +overwrite_data = False + + @pytest.mark.parametrize( + "optimizer_type", + ["SGD", "Adam", "Adadelta", "Adagrad", "Adamax", "AdamW", "RMSprop"], + ) + @pytest.mark.parametrize( + "use_zero_redundancy", + [False, True], + ) + def pytest_optimizers( + optimizer_type, use_zero_redundancy, ci_input="ci.json", overwrite_data=False + ): +> unittest_optimizers(optimizer_type, use_zero_redundancy, ci_input, overwrite_data) + +tests/test_optimizer.py:110: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_optimizer.py:94: in unittest_optimizers + hydragnn.run_training(config) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:80: in _ + world_size, world_rank = setup_ddp(use_deepspeed=use_deepspeed) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/utils/distributed/distributed.py:206: in setup_ddp + dist.init_process_group( +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:81: in wrapper + return func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:95: in wrapper + func_return = func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/distributed_c10d.py:1757: in init_process_group + store, rank, world_size = next(rendezvous_iterator) + ^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:278: in _env_rendezvous_handler + store = _create_c10d_store( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +hostname = '127.0.0.1', port = 8889, rank = 0, world_size = 1 +timeout = datetime.timedelta(seconds=1800), use_libuv = True + + def _create_c10d_store( + hostname, port, rank, world_size, timeout, use_libuv=True + ) -> Store: + """ + Smartly creates a c10d Store object on ``rank`` based on whether we need to reuse agent store. + + The TCPStore server is assumed to be hosted + on ``hostname:port``. + + By default, the TCPStore server uses the asynchronous implementation + ``LibUVStoreDaemon`` which utilizes libuv. + + If ``torchelastic_use_agent_store()`` is ``True``, then it is assumed that + the agent leader (node rank 0) hosts the TCPStore server (for which the + endpoint is specified by the given ``hostname:port``). Hence + ALL ranks will create and return a TCPStore client (e.g. ``start_daemon=False``). + + If ``torchelastic_use_agent_store()`` is ``False``, then rank 0 will host + the TCPStore (with multi-tenancy) and it is assumed that rank 0's hostname + and port are correctly passed via ``hostname`` and ``port``. All + non-zero ranks will create and return a TCPStore client. + """ + # check if port is uint16_t + if not 0 <= port < 2**16: + raise ValueError(f"port must have value from 0 to 65535 but was {port}.") + + if _torchelastic_use_agent_store(): + # We create a new TCPStore for every retry so no need to add prefix for each attempt. + return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=False, + timeout=timeout, + ) + else: + start_daemon = rank == 0 +> return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=start_daemon, + timeout=timeout, + multi_tenant=True, + use_libuv=use_libuv, + ) +E torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 8889, useIpv6: false, code: -48, name: EADDRINUSE, message: address already in use + +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:198: DistNetworkError +----------------------------- Captured stdout call ----------------------------- +Distributed data parallel: gloo master at 127.0.0.1:8889 +_______________ pytest_train_model_transforms[None-bessel-MACE] ________________ + +model_type = 'MACE', basis_function = 'bessel', distance_transform = 'None' +use_lengths = True, overwrite_data = False + + @pytest.mark.parametrize( + "model_type", + ["MACE"], + ) + @pytest.mark.parametrize("basis_function", ["bessel", "gaussian", "chebyshev"]) + @pytest.mark.parametrize("distance_transform", ["None", "Agnesi", "Soft"]) + def pytest_train_model_transforms( + model_type, + basis_function, + distance_transform, + use_lengths=True, + overwrite_data=False, + ): +> unittest_train_model( + model_type, + basis_function, + distance_transform, + "ci.json", + use_lengths, + overwrite_data, + ) + +tests/test_radial_transforms.py:201: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_radial_transforms.py:130: in unittest_train_model + hydragnn.run_training(config, use_deepspeed) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:80: in _ + world_size, world_rank = setup_ddp(use_deepspeed=use_deepspeed) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/utils/distributed/distributed.py:206: in setup_ddp + dist.init_process_group( +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:81: in wrapper + return func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:95: in wrapper + func_return = func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/distributed_c10d.py:1757: in init_process_group + store, rank, world_size = next(rendezvous_iterator) + ^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:278: in _env_rendezvous_handler + store = _create_c10d_store( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +hostname = '127.0.0.1', port = 8889, rank = 0, world_size = 1 +timeout = datetime.timedelta(seconds=1800), use_libuv = True + + def _create_c10d_store( + hostname, port, rank, world_size, timeout, use_libuv=True + ) -> Store: + """ + Smartly creates a c10d Store object on ``rank`` based on whether we need to reuse agent store. + + The TCPStore server is assumed to be hosted + on ``hostname:port``. + + By default, the TCPStore server uses the asynchronous implementation + ``LibUVStoreDaemon`` which utilizes libuv. + + If ``torchelastic_use_agent_store()`` is ``True``, then it is assumed that + the agent leader (node rank 0) hosts the TCPStore server (for which the + endpoint is specified by the given ``hostname:port``). Hence + ALL ranks will create and return a TCPStore client (e.g. ``start_daemon=False``). + + If ``torchelastic_use_agent_store()`` is ``False``, then rank 0 will host + the TCPStore (with multi-tenancy) and it is assumed that rank 0's hostname + and port are correctly passed via ``hostname`` and ``port``. All + non-zero ranks will create and return a TCPStore client. + """ + # check if port is uint16_t + if not 0 <= port < 2**16: + raise ValueError(f"port must have value from 0 to 65535 but was {port}.") + + if _torchelastic_use_agent_store(): + # We create a new TCPStore for every retry so no need to add prefix for each attempt. + return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=False, + timeout=timeout, + ) + else: + start_daemon = rank == 0 +> return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=start_daemon, + timeout=timeout, + multi_tenant=True, + use_libuv=use_libuv, + ) +E torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 8889, useIpv6: false, code: -48, name: EADDRINUSE, message: address already in use + +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:198: DistNetworkError +----------------------------- Captured stdout call ----------------------------- +Distributed data parallel: gloo master at 127.0.0.1:8889 +______________ pytest_train_model_transforms[None-gaussian-MACE] _______________ + +model_type = 'MACE', basis_function = 'gaussian', distance_transform = 'None' +use_lengths = True, overwrite_data = False + + @pytest.mark.parametrize( + "model_type", + ["MACE"], + ) + @pytest.mark.parametrize("basis_function", ["bessel", "gaussian", "chebyshev"]) + @pytest.mark.parametrize("distance_transform", ["None", "Agnesi", "Soft"]) + def pytest_train_model_transforms( + model_type, + basis_function, + distance_transform, + use_lengths=True, + overwrite_data=False, + ): +> unittest_train_model( + model_type, + basis_function, + distance_transform, + "ci.json", + use_lengths, + overwrite_data, + ) + +tests/test_radial_transforms.py:201: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_radial_transforms.py:130: in unittest_train_model + hydragnn.run_training(config, use_deepspeed) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:80: in _ + world_size, world_rank = setup_ddp(use_deepspeed=use_deepspeed) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/utils/distributed/distributed.py:206: in setup_ddp + dist.init_process_group( +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:81: in wrapper + return func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:95: in wrapper + func_return = func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/distributed_c10d.py:1757: in init_process_group + store, rank, world_size = next(rendezvous_iterator) + ^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:278: in _env_rendezvous_handler + store = _create_c10d_store( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +hostname = '127.0.0.1', port = 8889, rank = 0, world_size = 1 +timeout = datetime.timedelta(seconds=1800), use_libuv = True + + def _create_c10d_store( + hostname, port, rank, world_size, timeout, use_libuv=True + ) -> Store: + """ + Smartly creates a c10d Store object on ``rank`` based on whether we need to reuse agent store. + + The TCPStore server is assumed to be hosted + on ``hostname:port``. + + By default, the TCPStore server uses the asynchronous implementation + ``LibUVStoreDaemon`` which utilizes libuv. + + If ``torchelastic_use_agent_store()`` is ``True``, then it is assumed that + the agent leader (node rank 0) hosts the TCPStore server (for which the + endpoint is specified by the given ``hostname:port``). Hence + ALL ranks will create and return a TCPStore client (e.g. ``start_daemon=False``). + + If ``torchelastic_use_agent_store()`` is ``False``, then rank 0 will host + the TCPStore (with multi-tenancy) and it is assumed that rank 0's hostname + and port are correctly passed via ``hostname`` and ``port``. All + non-zero ranks will create and return a TCPStore client. + """ + # check if port is uint16_t + if not 0 <= port < 2**16: + raise ValueError(f"port must have value from 0 to 65535 but was {port}.") + + if _torchelastic_use_agent_store(): + # We create a new TCPStore for every retry so no need to add prefix for each attempt. + return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=False, + timeout=timeout, + ) + else: + start_daemon = rank == 0 +> return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=start_daemon, + timeout=timeout, + multi_tenant=True, + use_libuv=use_libuv, + ) +E torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 8889, useIpv6: false, code: -48, name: EADDRINUSE, message: address already in use + +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:198: DistNetworkError +----------------------------- Captured stdout call ----------------------------- +Distributed data parallel: gloo master at 127.0.0.1:8889 +______________ pytest_train_model_transforms[None-chebyshev-MACE] ______________ + +model_type = 'MACE', basis_function = 'chebyshev', distance_transform = 'None' +use_lengths = True, overwrite_data = False + + @pytest.mark.parametrize( + "model_type", + ["MACE"], + ) + @pytest.mark.parametrize("basis_function", ["bessel", "gaussian", "chebyshev"]) + @pytest.mark.parametrize("distance_transform", ["None", "Agnesi", "Soft"]) + def pytest_train_model_transforms( + model_type, + basis_function, + distance_transform, + use_lengths=True, + overwrite_data=False, + ): +> unittest_train_model( + model_type, + basis_function, + distance_transform, + "ci.json", + use_lengths, + overwrite_data, + ) + +tests/test_radial_transforms.py:201: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_radial_transforms.py:130: in unittest_train_model + hydragnn.run_training(config, use_deepspeed) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:80: in _ + world_size, world_rank = setup_ddp(use_deepspeed=use_deepspeed) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/utils/distributed/distributed.py:206: in setup_ddp + dist.init_process_group( +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:81: in wrapper + return func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:95: in wrapper + func_return = func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/distributed_c10d.py:1757: in init_process_group + store, rank, world_size = next(rendezvous_iterator) + ^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:278: in _env_rendezvous_handler + store = _create_c10d_store( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +hostname = '127.0.0.1', port = 8889, rank = 0, world_size = 1 +timeout = datetime.timedelta(seconds=1800), use_libuv = True + + def _create_c10d_store( + hostname, port, rank, world_size, timeout, use_libuv=True + ) -> Store: + """ + Smartly creates a c10d Store object on ``rank`` based on whether we need to reuse agent store. + + The TCPStore server is assumed to be hosted + on ``hostname:port``. + + By default, the TCPStore server uses the asynchronous implementation + ``LibUVStoreDaemon`` which utilizes libuv. + + If ``torchelastic_use_agent_store()`` is ``True``, then it is assumed that + the agent leader (node rank 0) hosts the TCPStore server (for which the + endpoint is specified by the given ``hostname:port``). Hence + ALL ranks will create and return a TCPStore client (e.g. ``start_daemon=False``). + + If ``torchelastic_use_agent_store()`` is ``False``, then rank 0 will host + the TCPStore (with multi-tenancy) and it is assumed that rank 0's hostname + and port are correctly passed via ``hostname`` and ``port``. All + non-zero ranks will create and return a TCPStore client. + """ + # check if port is uint16_t + if not 0 <= port < 2**16: + raise ValueError(f"port must have value from 0 to 65535 but was {port}.") + + if _torchelastic_use_agent_store(): + # We create a new TCPStore for every retry so no need to add prefix for each attempt. + return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=False, + timeout=timeout, + ) + else: + start_daemon = rank == 0 +> return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=start_daemon, + timeout=timeout, + multi_tenant=True, + use_libuv=use_libuv, + ) +E torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 8889, useIpv6: false, code: -48, name: EADDRINUSE, message: address already in use + +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:198: DistNetworkError +----------------------------- Captured stdout call ----------------------------- +Distributed data parallel: gloo master at 127.0.0.1:8889 +______________ pytest_train_model_transforms[Agnesi-bessel-MACE] _______________ + +model_type = 'MACE', basis_function = 'bessel', distance_transform = 'Agnesi' +use_lengths = True, overwrite_data = False + + @pytest.mark.parametrize( + "model_type", + ["MACE"], + ) + @pytest.mark.parametrize("basis_function", ["bessel", "gaussian", "chebyshev"]) + @pytest.mark.parametrize("distance_transform", ["None", "Agnesi", "Soft"]) + def pytest_train_model_transforms( + model_type, + basis_function, + distance_transform, + use_lengths=True, + overwrite_data=False, + ): +> unittest_train_model( + model_type, + basis_function, + distance_transform, + "ci.json", + use_lengths, + overwrite_data, + ) + +tests/test_radial_transforms.py:201: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_radial_transforms.py:130: in unittest_train_model + hydragnn.run_training(config, use_deepspeed) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:80: in _ + world_size, world_rank = setup_ddp(use_deepspeed=use_deepspeed) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/utils/distributed/distributed.py:206: in setup_ddp + dist.init_process_group( +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:81: in wrapper + return func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:95: in wrapper + func_return = func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/distributed_c10d.py:1757: in init_process_group + store, rank, world_size = next(rendezvous_iterator) + ^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:278: in _env_rendezvous_handler + store = _create_c10d_store( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +hostname = '127.0.0.1', port = 8889, rank = 0, world_size = 1 +timeout = datetime.timedelta(seconds=1800), use_libuv = True + + def _create_c10d_store( + hostname, port, rank, world_size, timeout, use_libuv=True + ) -> Store: + """ + Smartly creates a c10d Store object on ``rank`` based on whether we need to reuse agent store. + + The TCPStore server is assumed to be hosted + on ``hostname:port``. + + By default, the TCPStore server uses the asynchronous implementation + ``LibUVStoreDaemon`` which utilizes libuv. + + If ``torchelastic_use_agent_store()`` is ``True``, then it is assumed that + the agent leader (node rank 0) hosts the TCPStore server (for which the + endpoint is specified by the given ``hostname:port``). Hence + ALL ranks will create and return a TCPStore client (e.g. ``start_daemon=False``). + + If ``torchelastic_use_agent_store()`` is ``False``, then rank 0 will host + the TCPStore (with multi-tenancy) and it is assumed that rank 0's hostname + and port are correctly passed via ``hostname`` and ``port``. All + non-zero ranks will create and return a TCPStore client. + """ + # check if port is uint16_t + if not 0 <= port < 2**16: + raise ValueError(f"port must have value from 0 to 65535 but was {port}.") + + if _torchelastic_use_agent_store(): + # We create a new TCPStore for every retry so no need to add prefix for each attempt. + return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=False, + timeout=timeout, + ) + else: + start_daemon = rank == 0 +> return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=start_daemon, + timeout=timeout, + multi_tenant=True, + use_libuv=use_libuv, + ) +E torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 8889, useIpv6: false, code: -48, name: EADDRINUSE, message: address already in use + +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:198: DistNetworkError +----------------------------- Captured stdout call ----------------------------- +Distributed data parallel: gloo master at 127.0.0.1:8889 +_____________ pytest_train_model_transforms[Agnesi-gaussian-MACE] ______________ + +model_type = 'MACE', basis_function = 'gaussian', distance_transform = 'Agnesi' +use_lengths = True, overwrite_data = False + + @pytest.mark.parametrize( + "model_type", + ["MACE"], + ) + @pytest.mark.parametrize("basis_function", ["bessel", "gaussian", "chebyshev"]) + @pytest.mark.parametrize("distance_transform", ["None", "Agnesi", "Soft"]) + def pytest_train_model_transforms( + model_type, + basis_function, + distance_transform, + use_lengths=True, + overwrite_data=False, + ): +> unittest_train_model( + model_type, + basis_function, + distance_transform, + "ci.json", + use_lengths, + overwrite_data, + ) + +tests/test_radial_transforms.py:201: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_radial_transforms.py:130: in unittest_train_model + hydragnn.run_training(config, use_deepspeed) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:80: in _ + world_size, world_rank = setup_ddp(use_deepspeed=use_deepspeed) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/utils/distributed/distributed.py:206: in setup_ddp + dist.init_process_group( +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:81: in wrapper + return func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:95: in wrapper + func_return = func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/distributed_c10d.py:1757: in init_process_group + store, rank, world_size = next(rendezvous_iterator) + ^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:278: in _env_rendezvous_handler + store = _create_c10d_store( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +hostname = '127.0.0.1', port = 8889, rank = 0, world_size = 1 +timeout = datetime.timedelta(seconds=1800), use_libuv = True + + def _create_c10d_store( + hostname, port, rank, world_size, timeout, use_libuv=True + ) -> Store: + """ + Smartly creates a c10d Store object on ``rank`` based on whether we need to reuse agent store. + + The TCPStore server is assumed to be hosted + on ``hostname:port``. + + By default, the TCPStore server uses the asynchronous implementation + ``LibUVStoreDaemon`` which utilizes libuv. + + If ``torchelastic_use_agent_store()`` is ``True``, then it is assumed that + the agent leader (node rank 0) hosts the TCPStore server (for which the + endpoint is specified by the given ``hostname:port``). Hence + ALL ranks will create and return a TCPStore client (e.g. ``start_daemon=False``). + + If ``torchelastic_use_agent_store()`` is ``False``, then rank 0 will host + the TCPStore (with multi-tenancy) and it is assumed that rank 0's hostname + and port are correctly passed via ``hostname`` and ``port``. All + non-zero ranks will create and return a TCPStore client. + """ + # check if port is uint16_t + if not 0 <= port < 2**16: + raise ValueError(f"port must have value from 0 to 65535 but was {port}.") + + if _torchelastic_use_agent_store(): + # We create a new TCPStore for every retry so no need to add prefix for each attempt. + return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=False, + timeout=timeout, + ) + else: + start_daemon = rank == 0 +> return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=start_daemon, + timeout=timeout, + multi_tenant=True, + use_libuv=use_libuv, + ) +E torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 8889, useIpv6: false, code: -48, name: EADDRINUSE, message: address already in use + +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:198: DistNetworkError +----------------------------- Captured stdout call ----------------------------- +Distributed data parallel: gloo master at 127.0.0.1:8889 +_____________ pytest_train_model_transforms[Agnesi-chebyshev-MACE] _____________ + +model_type = 'MACE', basis_function = 'chebyshev', distance_transform = 'Agnesi' +use_lengths = True, overwrite_data = False + + @pytest.mark.parametrize( + "model_type", + ["MACE"], + ) + @pytest.mark.parametrize("basis_function", ["bessel", "gaussian", "chebyshev"]) + @pytest.mark.parametrize("distance_transform", ["None", "Agnesi", "Soft"]) + def pytest_train_model_transforms( + model_type, + basis_function, + distance_transform, + use_lengths=True, + overwrite_data=False, + ): +> unittest_train_model( + model_type, + basis_function, + distance_transform, + "ci.json", + use_lengths, + overwrite_data, + ) + +tests/test_radial_transforms.py:201: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_radial_transforms.py:130: in unittest_train_model + hydragnn.run_training(config, use_deepspeed) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:80: in _ + world_size, world_rank = setup_ddp(use_deepspeed=use_deepspeed) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/utils/distributed/distributed.py:206: in setup_ddp + dist.init_process_group( +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:81: in wrapper + return func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:95: in wrapper + func_return = func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/distributed_c10d.py:1757: in init_process_group + store, rank, world_size = next(rendezvous_iterator) + ^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:278: in _env_rendezvous_handler + store = _create_c10d_store( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +hostname = '127.0.0.1', port = 8889, rank = 0, world_size = 1 +timeout = datetime.timedelta(seconds=1800), use_libuv = True + + def _create_c10d_store( + hostname, port, rank, world_size, timeout, use_libuv=True + ) -> Store: + """ + Smartly creates a c10d Store object on ``rank`` based on whether we need to reuse agent store. + + The TCPStore server is assumed to be hosted + on ``hostname:port``. + + By default, the TCPStore server uses the asynchronous implementation + ``LibUVStoreDaemon`` which utilizes libuv. + + If ``torchelastic_use_agent_store()`` is ``True``, then it is assumed that + the agent leader (node rank 0) hosts the TCPStore server (for which the + endpoint is specified by the given ``hostname:port``). Hence + ALL ranks will create and return a TCPStore client (e.g. ``start_daemon=False``). + + If ``torchelastic_use_agent_store()`` is ``False``, then rank 0 will host + the TCPStore (with multi-tenancy) and it is assumed that rank 0's hostname + and port are correctly passed via ``hostname`` and ``port``. All + non-zero ranks will create and return a TCPStore client. + """ + # check if port is uint16_t + if not 0 <= port < 2**16: + raise ValueError(f"port must have value from 0 to 65535 but was {port}.") + + if _torchelastic_use_agent_store(): + # We create a new TCPStore for every retry so no need to add prefix for each attempt. + return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=False, + timeout=timeout, + ) + else: + start_daemon = rank == 0 +> return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=start_daemon, + timeout=timeout, + multi_tenant=True, + use_libuv=use_libuv, + ) +E torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 8889, useIpv6: false, code: -48, name: EADDRINUSE, message: address already in use + +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:198: DistNetworkError +----------------------------- Captured stdout call ----------------------------- +Distributed data parallel: gloo master at 127.0.0.1:8889 +_______________ pytest_train_model_transforms[Soft-bessel-MACE] ________________ + +model_type = 'MACE', basis_function = 'bessel', distance_transform = 'Soft' +use_lengths = True, overwrite_data = False + + @pytest.mark.parametrize( + "model_type", + ["MACE"], + ) + @pytest.mark.parametrize("basis_function", ["bessel", "gaussian", "chebyshev"]) + @pytest.mark.parametrize("distance_transform", ["None", "Agnesi", "Soft"]) + def pytest_train_model_transforms( + model_type, + basis_function, + distance_transform, + use_lengths=True, + overwrite_data=False, + ): +> unittest_train_model( + model_type, + basis_function, + distance_transform, + "ci.json", + use_lengths, + overwrite_data, + ) + +tests/test_radial_transforms.py:201: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_radial_transforms.py:130: in unittest_train_model + hydragnn.run_training(config, use_deepspeed) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:80: in _ + world_size, world_rank = setup_ddp(use_deepspeed=use_deepspeed) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/utils/distributed/distributed.py:206: in setup_ddp + dist.init_process_group( +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:81: in wrapper + return func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:95: in wrapper + func_return = func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/distributed_c10d.py:1757: in init_process_group + store, rank, world_size = next(rendezvous_iterator) + ^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:278: in _env_rendezvous_handler + store = _create_c10d_store( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +hostname = '127.0.0.1', port = 8889, rank = 0, world_size = 1 +timeout = datetime.timedelta(seconds=1800), use_libuv = True + + def _create_c10d_store( + hostname, port, rank, world_size, timeout, use_libuv=True + ) -> Store: + """ + Smartly creates a c10d Store object on ``rank`` based on whether we need to reuse agent store. + + The TCPStore server is assumed to be hosted + on ``hostname:port``. + + By default, the TCPStore server uses the asynchronous implementation + ``LibUVStoreDaemon`` which utilizes libuv. + + If ``torchelastic_use_agent_store()`` is ``True``, then it is assumed that + the agent leader (node rank 0) hosts the TCPStore server (for which the + endpoint is specified by the given ``hostname:port``). Hence + ALL ranks will create and return a TCPStore client (e.g. ``start_daemon=False``). + + If ``torchelastic_use_agent_store()`` is ``False``, then rank 0 will host + the TCPStore (with multi-tenancy) and it is assumed that rank 0's hostname + and port are correctly passed via ``hostname`` and ``port``. All + non-zero ranks will create and return a TCPStore client. + """ + # check if port is uint16_t + if not 0 <= port < 2**16: + raise ValueError(f"port must have value from 0 to 65535 but was {port}.") + + if _torchelastic_use_agent_store(): + # We create a new TCPStore for every retry so no need to add prefix for each attempt. + return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=False, + timeout=timeout, + ) + else: + start_daemon = rank == 0 +> return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=start_daemon, + timeout=timeout, + multi_tenant=True, + use_libuv=use_libuv, + ) +E torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 8889, useIpv6: false, code: -48, name: EADDRINUSE, message: address already in use + +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:198: DistNetworkError +----------------------------- Captured stdout call ----------------------------- +Distributed data parallel: gloo master at 127.0.0.1:8889 +______________ pytest_train_model_transforms[Soft-gaussian-MACE] _______________ + +model_type = 'MACE', basis_function = 'gaussian', distance_transform = 'Soft' +use_lengths = True, overwrite_data = False + + @pytest.mark.parametrize( + "model_type", + ["MACE"], + ) + @pytest.mark.parametrize("basis_function", ["bessel", "gaussian", "chebyshev"]) + @pytest.mark.parametrize("distance_transform", ["None", "Agnesi", "Soft"]) + def pytest_train_model_transforms( + model_type, + basis_function, + distance_transform, + use_lengths=True, + overwrite_data=False, + ): +> unittest_train_model( + model_type, + basis_function, + distance_transform, + "ci.json", + use_lengths, + overwrite_data, + ) + +tests/test_radial_transforms.py:201: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_radial_transforms.py:130: in unittest_train_model + hydragnn.run_training(config, use_deepspeed) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:80: in _ + world_size, world_rank = setup_ddp(use_deepspeed=use_deepspeed) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/utils/distributed/distributed.py:206: in setup_ddp + dist.init_process_group( +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:81: in wrapper + return func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:95: in wrapper + func_return = func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/distributed_c10d.py:1757: in init_process_group + store, rank, world_size = next(rendezvous_iterator) + ^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:278: in _env_rendezvous_handler + store = _create_c10d_store( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +hostname = '127.0.0.1', port = 8889, rank = 0, world_size = 1 +timeout = datetime.timedelta(seconds=1800), use_libuv = True + + def _create_c10d_store( + hostname, port, rank, world_size, timeout, use_libuv=True + ) -> Store: + """ + Smartly creates a c10d Store object on ``rank`` based on whether we need to reuse agent store. + + The TCPStore server is assumed to be hosted + on ``hostname:port``. + + By default, the TCPStore server uses the asynchronous implementation + ``LibUVStoreDaemon`` which utilizes libuv. + + If ``torchelastic_use_agent_store()`` is ``True``, then it is assumed that + the agent leader (node rank 0) hosts the TCPStore server (for which the + endpoint is specified by the given ``hostname:port``). Hence + ALL ranks will create and return a TCPStore client (e.g. ``start_daemon=False``). + + If ``torchelastic_use_agent_store()`` is ``False``, then rank 0 will host + the TCPStore (with multi-tenancy) and it is assumed that rank 0's hostname + and port are correctly passed via ``hostname`` and ``port``. All + non-zero ranks will create and return a TCPStore client. + """ + # check if port is uint16_t + if not 0 <= port < 2**16: + raise ValueError(f"port must have value from 0 to 65535 but was {port}.") + + if _torchelastic_use_agent_store(): + # We create a new TCPStore for every retry so no need to add prefix for each attempt. + return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=False, + timeout=timeout, + ) + else: + start_daemon = rank == 0 +> return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=start_daemon, + timeout=timeout, + multi_tenant=True, + use_libuv=use_libuv, + ) +E torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 8889, useIpv6: false, code: -48, name: EADDRINUSE, message: address already in use + +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:198: DistNetworkError +----------------------------- Captured stdout call ----------------------------- +Distributed data parallel: gloo master at 127.0.0.1:8889 +______________ pytest_train_model_transforms[Soft-chebyshev-MACE] ______________ + +model_type = 'MACE', basis_function = 'chebyshev', distance_transform = 'Soft' +use_lengths = True, overwrite_data = False + + @pytest.mark.parametrize( + "model_type", + ["MACE"], + ) + @pytest.mark.parametrize("basis_function", ["bessel", "gaussian", "chebyshev"]) + @pytest.mark.parametrize("distance_transform", ["None", "Agnesi", "Soft"]) + def pytest_train_model_transforms( + model_type, + basis_function, + distance_transform, + use_lengths=True, + overwrite_data=False, + ): +> unittest_train_model( + model_type, + basis_function, + distance_transform, + "ci.json", + use_lengths, + overwrite_data, + ) + +tests/test_radial_transforms.py:201: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_radial_transforms.py:130: in unittest_train_model + hydragnn.run_training(config, use_deepspeed) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:80: in _ + world_size, world_rank = setup_ddp(use_deepspeed=use_deepspeed) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/utils/distributed/distributed.py:206: in setup_ddp + dist.init_process_group( +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:81: in wrapper + return func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/c10d_logger.py:95: in wrapper + func_return = func(*args, **kwargs) + ^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/distributed_c10d.py:1757: in init_process_group + store, rank, world_size = next(rendezvous_iterator) + ^^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:278: in _env_rendezvous_handler + store = _create_c10d_store( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +hostname = '127.0.0.1', port = 8889, rank = 0, world_size = 1 +timeout = datetime.timedelta(seconds=1800), use_libuv = True + + def _create_c10d_store( + hostname, port, rank, world_size, timeout, use_libuv=True + ) -> Store: + """ + Smartly creates a c10d Store object on ``rank`` based on whether we need to reuse agent store. + + The TCPStore server is assumed to be hosted + on ``hostname:port``. + + By default, the TCPStore server uses the asynchronous implementation + ``LibUVStoreDaemon`` which utilizes libuv. + + If ``torchelastic_use_agent_store()`` is ``True``, then it is assumed that + the agent leader (node rank 0) hosts the TCPStore server (for which the + endpoint is specified by the given ``hostname:port``). Hence + ALL ranks will create and return a TCPStore client (e.g. ``start_daemon=False``). + + If ``torchelastic_use_agent_store()`` is ``False``, then rank 0 will host + the TCPStore (with multi-tenancy) and it is assumed that rank 0's hostname + and port are correctly passed via ``hostname`` and ``port``. All + non-zero ranks will create and return a TCPStore client. + """ + # check if port is uint16_t + if not 0 <= port < 2**16: + raise ValueError(f"port must have value from 0 to 65535 but was {port}.") + + if _torchelastic_use_agent_store(): + # We create a new TCPStore for every retry so no need to add prefix for each attempt. + return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=False, + timeout=timeout, + ) + else: + start_daemon = rank == 0 +> return TCPStore( + host_name=hostname, + port=port, + world_size=world_size, + is_master=start_daemon, + timeout=timeout, + multi_tenant=True, + use_libuv=use_libuv, + ) +E torch.distributed.DistNetworkError: The server socket has failed to listen on any local network address. port: 8889, useIpv6: false, code: -48, name: EADDRINUSE, message: address already in use + +.venv/lib/python3.13/site-packages/torch/distributed/rendezvous.py:198: DistNetworkError +----------------------------- Captured stdout call ----------------------------- +Distributed data parallel: gloo master at 127.0.0.1:8889 +=========================== short test summary info ============================ +FAILED tests/test_atomicdescriptors.py::pytest_atomicdescriptors - assert 1 == 0 +FAILED tests/test_examples.py::pytest_examples_energy[qm9-SAGE-multihead-GPS] +FAILED tests/test_examples.py::pytest_examples_energy[qm9-GIN-multihead-GPS] +FAILED tests/test_examples.py::pytest_examples_energy[qm9-GAT-multihead-GPS] +FAILED tests/test_examples.py::pytest_examples_energy[qm9-MFC-multihead-GPS] +FAILED tests/test_examples.py::pytest_examples_energy[qm9-PNA-multihead-GPS] +FAILED tests/test_examples.py::pytest_examples_energy[qm9-PNAPlus-multihead-GPS] +FAILED tests/test_examples.py::pytest_examples_energy[qm9-SchNet-multihead-GPS] +FAILED tests/test_examples.py::pytest_examples_energy[qm9-DimeNet-multihead-GPS] +FAILED tests/test_examples.py::pytest_examples_energy[qm9-EGNN-multihead-GPS] +FAILED tests/test_examples.py::pytest_examples_energy[qm9-PNAEq-multihead-GPS] +FAILED tests/test_examples.py::pytest_examples_energy[qm9-PAINN-multihead-GPS] +FAILED tests/test_examples.py::pytest_examples_energy[md17-SAGE-multihead-GPS] +FAILED tests/test_examples.py::pytest_examples_energy[md17-GIN-multihead-GPS] +FAILED tests/test_examples.py::pytest_examples_energy[md17-GAT-multihead-GPS] +FAILED tests/test_examples.py::pytest_examples_energy[md17-MFC-multihead-GPS] +FAILED tests/test_examples.py::pytest_examples_energy[md17-PNA-multihead-GPS] +FAILED tests/test_examples.py::pytest_examples_energy[md17-PNAPlus-multihead-GPS] +FAILED tests/test_examples.py::pytest_examples_energy[md17-SchNet-multihead-GPS] +FAILED tests/test_examples.py::pytest_examples_energy[md17-DimeNet-multihead-GPS] +FAILED tests/test_examples.py::pytest_examples_energy[md17-EGNN-multihead-GPS] +FAILED tests/test_examples.py::pytest_examples_energy[md17-PNAEq-multihead-GPS] +FAILED tests/test_examples.py::pytest_examples_energy[md17-PAINN-multihead-GPS] +FAILED tests/test_examples.py::pytest_examples_grad_forces[LennardJones-PNAPlus] +FAILED tests/test_examples.py::pytest_examples_grad_forces[LennardJones-SchNet] +FAILED tests/test_examples.py::pytest_examples_grad_forces[LennardJones-DimeNet] +FAILED tests/test_examples.py::pytest_examples_grad_forces[LennardJones-EGNN] +FAILED tests/test_examples.py::pytest_examples_grad_forces[LennardJones-PNAEq] +FAILED tests/test_examples.py::pytest_examples_grad_forces[LennardJones-PAINN] +FAILED tests/test_examples.py::pytest_examples_grad_forces[LennardJones-MACE] +FAILED tests/test_forces_equivariant.py::pytest_examples[SchNet-LennardJones] +FAILED tests/test_forces_equivariant.py::pytest_examples[EGNN-LennardJones] +FAILED tests/test_forces_equivariant.py::pytest_examples[DimeNet-LennardJones] +FAILED tests/test_forces_equivariant.py::pytest_examples[PAINN-LennardJones] +FAILED tests/test_forces_equivariant.py::pytest_examples[PNAPlus-LennardJones] +FAILED tests/test_forces_equivariant.py::pytest_examples[MACE-LennardJones] +FAILED tests/test_graphs.py::pytest_train_model[ci.json-SAGE] - torch.distrib... +FAILED tests/test_graphs.py::pytest_train_model[ci.json-GIN] - torch.distribu... +FAILED tests/test_graphs.py::pytest_train_model[ci.json-GAT] - torch.distribu... +FAILED tests/test_graphs.py::pytest_train_model[ci.json-MFC] - torch.distribu... +FAILED tests/test_graphs.py::pytest_train_model[ci.json-PNA] - torch.distribu... +FAILED tests/test_graphs.py::pytest_train_model[ci.json-PNAPlus] - torch.dist... +FAILED tests/test_graphs.py::pytest_train_model[ci.json-CGCNN] - torch.distri... +FAILED tests/test_graphs.py::pytest_train_model[ci.json-SchNet] - torch.distr... +FAILED tests/test_graphs.py::pytest_train_model[ci.json-DimeNet] - torch.dist... +FAILED tests/test_graphs.py::pytest_train_model[ci.json-EGNN] - torch.distrib... +FAILED tests/test_graphs.py::pytest_train_model[ci.json-PNAEq] - torch.distri... +FAILED tests/test_graphs.py::pytest_train_model[ci.json-PAINN] - torch.distri... +FAILED tests/test_graphs.py::pytest_train_model[ci.json-MACE] - torch.distrib... +FAILED tests/test_graphs.py::pytest_train_model[ci_multihead.json-SAGE] - tor... +FAILED tests/test_graphs.py::pytest_train_model[ci_multihead.json-GIN] - torc... +FAILED tests/test_graphs.py::pytest_train_model[ci_multihead.json-GAT] - torc... +FAILED tests/test_graphs.py::pytest_train_model[ci_multihead.json-MFC] - torc... +FAILED tests/test_graphs.py::pytest_train_model[ci_multihead.json-PNA] - torc... +FAILED tests/test_graphs.py::pytest_train_model[ci_multihead.json-PNAPlus] - ... +FAILED tests/test_graphs.py::pytest_train_model[ci_multihead.json-CGCNN] - to... +FAILED tests/test_graphs.py::pytest_train_model[ci_multihead.json-SchNet] - t... +FAILED tests/test_graphs.py::pytest_train_model[ci_multihead.json-DimeNet] - ... +FAILED tests/test_graphs.py::pytest_train_model[ci_multihead.json-EGNN] - tor... +FAILED tests/test_graphs.py::pytest_train_model[ci_multihead.json-PNAEq] - to... +FAILED tests/test_graphs.py::pytest_train_model[ci_multihead.json-PAINN] - to... +FAILED tests/test_graphs.py::pytest_train_model[ci_multihead.json-MACE] - tor... +FAILED tests/test_graphs.py::pytest_train_model_lengths[GAT] - torch.distribu... +FAILED tests/test_graphs.py::pytest_train_model_lengths[PNA] - torch.distribu... +FAILED tests/test_graphs.py::pytest_train_model_lengths[PNAPlus] - torch.dist... +FAILED tests/test_graphs.py::pytest_train_model_lengths[CGCNN] - torch.distri... +FAILED tests/test_graphs.py::pytest_train_model_lengths[SchNet] - torch.distr... +FAILED tests/test_graphs.py::pytest_train_model_lengths[DimeNet] - torch.dist... +FAILED tests/test_graphs.py::pytest_train_model_lengths[EGNN] - torch.distrib... +FAILED tests/test_graphs.py::pytest_train_model_lengths[PNAEq] - torch.distri... +FAILED tests/test_graphs.py::pytest_train_model_lengths[PAINN] - torch.distri... +FAILED tests/test_graphs.py::pytest_train_model_lengths_global_attention[GAT-multihead-GPS] +FAILED tests/test_graphs.py::pytest_train_model_lengths_global_attention[PNA-multihead-GPS] +FAILED tests/test_graphs.py::pytest_train_model_lengths_global_attention[PNAPlus-multihead-GPS] +FAILED tests/test_graphs.py::pytest_train_model_lengths_global_attention[CGCNN-multihead-GPS] +FAILED tests/test_graphs.py::pytest_train_model_lengths_global_attention[SchNet-multihead-GPS] +FAILED tests/test_graphs.py::pytest_train_model_lengths_global_attention[DimeNet-multihead-GPS] +FAILED tests/test_graphs.py::pytest_train_model_lengths_global_attention[EGNN-multihead-GPS] +FAILED tests/test_graphs.py::pytest_train_model_lengths_global_attention[PNAEq-multihead-GPS] +FAILED tests/test_graphs.py::pytest_train_model_lengths_global_attention[PAINN-multihead-GPS] +FAILED tests/test_graphs.py::pytest_train_mace_model_lengths[MACE] - torch.di... +FAILED tests/test_graphs.py::pytest_train_equivariant_model[EGNN] - torch.dis... +FAILED tests/test_graphs.py::pytest_train_equivariant_model[SchNet] - torch.d... +FAILED tests/test_graphs.py::pytest_train_equivariant_model[PNAEq] - torch.di... +FAILED tests/test_graphs.py::pytest_train_equivariant_model[PAINN] - torch.di... +FAILED tests/test_graphs.py::pytest_train_equivariant_model[MACE] - torch.dis... +FAILED tests/test_graphs.py::pytest_train_model_vectoroutput[GAT] - torch.dis... +FAILED tests/test_graphs.py::pytest_train_model_vectoroutput[PNA] - torch.dis... +FAILED tests/test_graphs.py::pytest_train_model_vectoroutput[PNAPlus] - torch... +FAILED tests/test_graphs.py::pytest_train_model_vectoroutput[SchNet] - torch.... +FAILED tests/test_graphs.py::pytest_train_model_vectoroutput[DimeNet] - torch... +FAILED tests/test_graphs.py::pytest_train_model_vectoroutput[EGNN] - torch.di... +FAILED tests/test_graphs.py::pytest_train_model_vectoroutput[PNAEq] - torch.d... +FAILED tests/test_graphs.py::pytest_train_model_conv_head[SAGE] - torch.distr... +FAILED tests/test_graphs.py::pytest_train_model_conv_head[GIN] - torch.distri... +FAILED tests/test_graphs.py::pytest_train_model_conv_head[GAT] - torch.distri... +FAILED tests/test_graphs.py::pytest_train_model_conv_head[MFC] - torch.distri... +FAILED tests/test_graphs.py::pytest_train_model_conv_head[PNA] - torch.distri... +FAILED tests/test_graphs.py::pytest_train_model_conv_head[PNAPlus] - torch.di... +FAILED tests/test_graphs.py::pytest_train_model_conv_head[SchNet] - torch.dis... +FAILED tests/test_graphs.py::pytest_train_model_conv_head[DimeNet] - torch.di... +FAILED tests/test_graphs.py::pytest_train_model_conv_head[EGNN] - torch.distr... +FAILED tests/test_graphs.py::pytest_train_model_conv_head[PNAEq] - torch.dist... +FAILED tests/test_graphs.py::pytest_train_model_conv_head[PAINN] - torch.dist... +FAILED tests/test_loss_and_activation_functions.py::pytest_loss_functions[mse] +FAILED tests/test_loss_and_activation_functions.py::pytest_loss_functions[mae] +FAILED tests/test_loss_and_activation_functions.py::pytest_loss_functions[rmse] +FAILED tests/test_loss_and_activation_functions.py::pytest_loss_functions[GaussianNLLLoss] +FAILED tests/test_loss_and_activation_functions.py::pytest_activation_functions_multihead[relu] +FAILED tests/test_loss_and_activation_functions.py::pytest_activation_functions_multihead[selu] +FAILED tests/test_loss_and_activation_functions.py::pytest_activation_functions_multihead[prelu] +FAILED tests/test_loss_and_activation_functions.py::pytest_activation_functions_multihead[elu] +FAILED tests/test_loss_and_activation_functions.py::pytest_activation_functions_multihead[lrelu_01] +FAILED tests/test_loss_and_activation_functions.py::pytest_activation_functions_multihead[lrelu_025] +FAILED tests/test_loss_and_activation_functions.py::pytest_activation_functions_multihead[lrelu_05] +FAILED tests/test_loss_and_activation_functions.py::pytest_activation_functions_vectoroutput[relu] +FAILED tests/test_loss_and_activation_functions.py::pytest_activation_functions_vectoroutput[selu] +FAILED tests/test_loss_and_activation_functions.py::pytest_activation_functions_vectoroutput[prelu] +FAILED tests/test_loss_and_activation_functions.py::pytest_activation_functions_vectoroutput[elu] +FAILED tests/test_loss_and_activation_functions.py::pytest_activation_functions_vectoroutput[lrelu_01] +FAILED tests/test_loss_and_activation_functions.py::pytest_activation_functions_vectoroutput[lrelu_025] +FAILED tests/test_loss_and_activation_functions.py::pytest_activation_functions_vectoroutput[lrelu_05] +FAILED tests/test_model_loadpred.py::pytest_model_loadpred - torch.distribute... +FAILED tests/test_optimizer.py::pytest_optimizers[False-SGD] - torch.distribu... +FAILED tests/test_optimizer.py::pytest_optimizers[False-Adam] - torch.distrib... +FAILED tests/test_optimizer.py::pytest_optimizers[False-Adadelta] - torch.dis... +FAILED tests/test_optimizer.py::pytest_optimizers[False-Adagrad] - torch.dist... +FAILED tests/test_optimizer.py::pytest_optimizers[False-Adamax] - torch.distr... +FAILED tests/test_optimizer.py::pytest_optimizers[False-AdamW] - torch.distri... +FAILED tests/test_optimizer.py::pytest_optimizers[False-RMSprop] - torch.dist... +FAILED tests/test_optimizer.py::pytest_optimizers[True-SGD] - torch.distribut... +FAILED tests/test_optimizer.py::pytest_optimizers[True-Adam] - torch.distribu... +FAILED tests/test_optimizer.py::pytest_optimizers[True-Adadelta] - torch.dist... +FAILED tests/test_optimizer.py::pytest_optimizers[True-Adagrad] - torch.distr... +FAILED tests/test_optimizer.py::pytest_optimizers[True-Adamax] - torch.distri... +FAILED tests/test_optimizer.py::pytest_optimizers[True-AdamW] - torch.distrib... +FAILED tests/test_optimizer.py::pytest_optimizers[True-RMSprop] - torch.distr... +FAILED tests/test_radial_transforms.py::pytest_train_model_transforms[None-bessel-MACE] +FAILED tests/test_radial_transforms.py::pytest_train_model_transforms[None-gaussian-MACE] +FAILED tests/test_radial_transforms.py::pytest_train_model_transforms[None-chebyshev-MACE] +FAILED tests/test_radial_transforms.py::pytest_train_model_transforms[Agnesi-bessel-MACE] +FAILED tests/test_radial_transforms.py::pytest_train_model_transforms[Agnesi-gaussian-MACE] +FAILED tests/test_radial_transforms.py::pytest_train_model_transforms[Agnesi-chebyshev-MACE] +FAILED tests/test_radial_transforms.py::pytest_train_model_transforms[Soft-bessel-MACE] +FAILED tests/test_radial_transforms.py::pytest_train_model_transforms[Soft-gaussian-MACE] +FAILED tests/test_radial_transforms.py::pytest_train_model_transforms[Soft-chebyshev-MACE] +============= 146 failed, 23 passed, 3 skipped in 64.63s (0:01:04) ============= diff --git a/test_output_final.log b/test_output_final.log new file mode 100644 index 000000000..79c8050fc --- /dev/null +++ b/test_output_final.log @@ -0,0 +1,2090 @@ +============================= test session starts ============================== +platform darwin -- Python 3.13.7, pytest-8.4.2, pluggy-1.6.0 -- /Users/7ml/Documents/Codes/HydraGNN/.venv/bin/python +cachedir: .pytest_cache +rootdir: /Users/7ml/Documents/Codes/HydraGNN +configfile: pytest.ini +collecting ... collected 68 items + +tests/test_graphs.py::pytest_train_model[ci.json-SAGE] PASSED [ 1%] +tests/test_graphs.py::pytest_train_model[ci.json-GIN] PASSED [ 2%] +tests/test_graphs.py::pytest_train_model[ci.json-GAT] PASSED [ 4%] +tests/test_graphs.py::pytest_train_model[ci.json-MFC] PASSED [ 5%] +tests/test_graphs.py::pytest_train_model[ci.json-PNA] PASSED [ 7%] +tests/test_graphs.py::pytest_train_model[ci.json-PNAPlus] PASSED [ 8%] +tests/test_graphs.py::pytest_train_model[ci.json-CGCNN] PASSED [ 10%] +tests/test_graphs.py::pytest_train_model[ci.json-SchNet] PASSED [ 11%] +tests/test_graphs.py::pytest_train_model[ci.json-DimeNet] PASSED [ 13%] +tests/test_graphs.py::pytest_train_model[ci.json-EGNN] PASSED [ 14%] +tests/test_graphs.py::pytest_train_model[ci.json-PNAEq] PASSED [ 16%] +tests/test_graphs.py::pytest_train_model[ci.json-PAINN] PASSED [ 17%] +tests/test_graphs.py::pytest_train_model[ci.json-MACE] PASSED [ 19%] +tests/test_graphs.py::pytest_train_model[ci_multihead.json-SAGE] PASSED [ 20%] +tests/test_graphs.py::pytest_train_model[ci_multihead.json-GIN] PASSED [ 22%] +tests/test_graphs.py::pytest_train_model[ci_multihead.json-GAT] PASSED [ 23%] +tests/test_graphs.py::pytest_train_model[ci_multihead.json-MFC] PASSED [ 25%] +tests/test_graphs.py::pytest_train_model[ci_multihead.json-PNA] PASSED [ 26%] +tests/test_graphs.py::pytest_train_model[ci_multihead.json-PNAPlus] PASSED [ 27%] +tests/test_graphs.py::pytest_train_model[ci_multihead.json-CGCNN] PASSED [ 29%] +tests/test_graphs.py::pytest_train_model[ci_multihead.json-SchNet] PASSED [ 30%] +tests/test_graphs.py::pytest_train_model[ci_multihead.json-DimeNet] PASSED [ 32%] +tests/test_graphs.py::pytest_train_model[ci_multihead.json-EGNN] PASSED [ 33%] +tests/test_graphs.py::pytest_train_model[ci_multihead.json-PNAEq] PASSED [ 35%] +tests/test_graphs.py::pytest_train_model[ci_multihead.json-PAINN] PASSED [ 36%] +tests/test_graphs.py::pytest_train_model[ci_multihead.json-MACE] PASSED [ 38%] +tests/test_graphs.py::pytest_train_model_lengths[GAT] PASSED [ 39%] +tests/test_graphs.py::pytest_train_model_lengths[PNA] PASSED [ 41%] +tests/test_graphs.py::pytest_train_model_lengths[PNAPlus] PASSED [ 42%] +tests/test_graphs.py::pytest_train_model_lengths[CGCNN] PASSED [ 44%] +tests/test_graphs.py::pytest_train_model_lengths[SchNet] PASSED [ 45%] +tests/test_graphs.py::pytest_train_model_lengths[DimeNet] PASSED [ 47%] +tests/test_graphs.py::pytest_train_model_lengths[EGNN] PASSED [ 48%] +tests/test_graphs.py::pytest_train_model_lengths[PNAEq] PASSED [ 50%] +tests/test_graphs.py::pytest_train_model_lengths[PAINN] PASSED [ 51%] +tests/test_graphs.py::pytest_train_model_lengths_global_attention[GAT-multihead-GPS] PASSED [ 52%] +tests/test_graphs.py::pytest_train_model_lengths_global_attention[PNA-multihead-GPS] PASSED [ 54%] +tests/test_graphs.py::pytest_train_model_lengths_global_attention[PNAPlus-multihead-GPS] PASSED [ 55%] +tests/test_graphs.py::pytest_train_model_lengths_global_attention[CGCNN-multihead-GPS] PASSED [ 57%] +tests/test_graphs.py::pytest_train_model_lengths_global_attention[SchNet-multihead-GPS] PASSED [ 58%] +tests/test_graphs.py::pytest_train_model_lengths_global_attention[DimeNet-multihead-GPS] PASSED [ 60%] +tests/test_graphs.py::pytest_train_model_lengths_global_attention[EGNN-multihead-GPS] PASSED [ 61%] +tests/test_graphs.py::pytest_train_model_lengths_global_attention[PNAEq-multihead-GPS] PASSED [ 63%] +tests/test_graphs.py::pytest_train_model_lengths_global_attention[PAINN-multihead-GPS] PASSED [ 64%] +tests/test_graphs.py::pytest_train_mace_model_lengths[MACE] PASSED [ 66%] +tests/test_graphs.py::pytest_train_equivariant_model[EGNN] PASSED [ 67%] +tests/test_graphs.py::pytest_train_equivariant_model[SchNet] PASSED [ 69%] +tests/test_graphs.py::pytest_train_equivariant_model[PNAEq] PASSED [ 70%] +tests/test_graphs.py::pytest_train_equivariant_model[PAINN] PASSED [ 72%] +tests/test_graphs.py::pytest_train_equivariant_model[MACE] PASSED [ 73%] +tests/test_graphs.py::pytest_train_model_vectoroutput[GAT] PASSED [ 75%] +tests/test_graphs.py::pytest_train_model_vectoroutput[PNA] FAILED [ 76%] +tests/test_graphs.py::pytest_train_model_vectoroutput[PNAPlus] PASSED [ 77%] +tests/test_graphs.py::pytest_train_model_vectoroutput[SchNet] PASSED [ 79%] +tests/test_graphs.py::pytest_train_model_vectoroutput[DimeNet] PASSED [ 80%] +tests/test_graphs.py::pytest_train_model_vectoroutput[EGNN] PASSED [ 82%] +tests/test_graphs.py::pytest_train_model_vectoroutput[PNAEq] PASSED [ 83%] +tests/test_graphs.py::pytest_train_model_conv_head[SAGE] FAILED [ 85%] +tests/test_graphs.py::pytest_train_model_conv_head[GIN] PASSED [ 86%] +tests/test_graphs.py::pytest_train_model_conv_head[GAT] FAILED [ 88%] +tests/test_graphs.py::pytest_train_model_conv_head[MFC] FAILED [ 89%] +tests/test_graphs.py::pytest_train_model_conv_head[PNA] FAILED [ 91%] +tests/test_graphs.py::pytest_train_model_conv_head[PNAPlus] FAILED [ 92%] +tests/test_graphs.py::pytest_train_model_conv_head[SchNet] FAILED [ 94%] +tests/test_graphs.py::pytest_train_model_conv_head[DimeNet] FAILED [ 95%] +tests/test_graphs.py::pytest_train_model_conv_head[EGNN] FAILED [ 97%] +tests/test_graphs.py::pytest_train_model_conv_head[PNAEq] FAILED [ 98%] +tests/test_graphs.py::pytest_train_model_conv_head[PAINN] FAILED [100%] + +=================================== FAILURES =================================== +_____________________ pytest_train_model_vectoroutput[PNA] _____________________ + +mpnn_type = 'PNA', overwrite_data = False + + @pytest.mark.parametrize( + "mpnn_type", + [ + "GAT", + "PNA", + "PNAPlus", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + ], + ) + def pytest_train_model_vectoroutput(mpnn_type, overwrite_data=False): +> unittest_train_model( + mpnn_type, None, None, "ci_vectoroutput.json", True, overwrite_data + ) + +tests/test_graphs.py:284: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +mpnn_type = 'PNA', global_attn_engine = None, global_attn_type = None +ci_input = 'ci_vectoroutput.json', use_lengths = True, overwrite_data = False +use_deepspeed = False, overwrite_config = None + + def unittest_train_model( + mpnn_type, + global_attn_engine, + global_attn_type, + ci_input, + use_lengths, + overwrite_data=False, + use_deepspeed=False, + overwrite_config=None, + ): + world_size, rank = hydragnn.utils.distributed.get_comm_size_and_rank() + + os.environ["SERIALIZED_DATA_PATH"] = os.getcwd() + + # Read in config settings and override model type. + config_file = os.path.join(os.getcwd(), "tests/inputs", ci_input) + with open(config_file, "r") as f: + config = json.load(f) + config["NeuralNetwork"]["Architecture"]["global_attn_engine"] = global_attn_engine + config["NeuralNetwork"]["Architecture"]["global_attn_type"] = global_attn_type + config["NeuralNetwork"]["Architecture"]["mpnn_type"] = mpnn_type + + # Overwrite config settings if provided + if overwrite_config: + config = merge_config(config, overwrite_config) + + """ + to test this locally, set ci.json as + "Dataset": { + ... + "path": { + "train": "serialized_dataset/unit_test_singlehead_train.pkl", + "test": "serialized_dataset/unit_test_singlehead_test.pkl", + "validate": "serialized_dataset/unit_test_singlehead_validate.pkl"} + ... + """ + # use pkl files if exist by default + for dataset_name in config["Dataset"]["path"].keys(): + if dataset_name == "total": + pkl_file = ( + os.environ["SERIALIZED_DATA_PATH"] + + "/serialized_dataset/" + + config["Dataset"]["name"] + + ".pkl" + ) + else: + pkl_file = ( + os.environ["SERIALIZED_DATA_PATH"] + + "/serialized_dataset/" + + config["Dataset"]["name"] + + "_" + + dataset_name + + ".pkl" + ) + if os.path.exists(pkl_file): + config["Dataset"]["path"][dataset_name] = pkl_file + + # In the unit test runs, it is found MFC favors graph-level features over node-level features, compared with other models; + # hence here we decrease the loss weight coefficient for graph-level head in MFC. + if mpnn_type == "MFC" and ci_input == "ci_multihead.json": + config["NeuralNetwork"]["Architecture"]["task_weights"][0] = 2 + + # Only run with edge lengths for models that support them. + if use_lengths: + config["NeuralNetwork"]["Architecture"]["edge_features"] = ["lengths"] + + if rank == 0: + num_samples_tot = 500 + # check if serialized pickle files or folders for raw files provided + pkl_input = False + if list(config["Dataset"]["path"].values())[0].endswith(".pkl"): + pkl_input = True + # only generate new datasets, if not pkl + if not pkl_input: + for dataset_name, data_path in config["Dataset"]["path"].items(): + if overwrite_data: + shutil.rmtree(data_path) + if not os.path.exists(data_path): + os.makedirs(data_path) + if dataset_name == "total": + num_samples = num_samples_tot + elif dataset_name == "train": + num_samples = int( + num_samples_tot + * config["NeuralNetwork"]["Training"]["perc_train"] + ) + elif dataset_name == "test": + num_samples = int( + num_samples_tot + * (1 - config["NeuralNetwork"]["Training"]["perc_train"]) + * 0.5 + ) + elif dataset_name == "validate": + num_samples = int( + num_samples_tot + * (1 - config["NeuralNetwork"]["Training"]["perc_train"]) + * 0.5 + ) + if not os.listdir(data_path): + tests.deterministic_graph_data( + data_path, number_configurations=num_samples + ) + MPI.COMM_WORLD.Barrier() + + # Since the config file uses PNA already, test the file overload here. + # All the other models need to use the locally modified dictionary. + if mpnn_type == "PNA" and not use_lengths: + hydragnn.run_training(config_file, use_deepspeed) + else: + hydragnn.run_training(config, use_deepspeed) + + ( + error, + error_mse_task, + true_values, + predicted_values, + ) = hydragnn.run_prediction(config, use_deepspeed) + + # Set RMSE and sample MAE error thresholds + thresholds = { + "SAGE": [0.20, 0.20], + "PNA": [0.20, 0.20], + "PNAPlus": [0.20, 0.20], + "MFC": [0.20, 0.30], + "GIN": [0.25, 0.20], + "GAT": [0.60, 0.70], + "CGCNN": [0.50, 0.40], + "SchNet": [0.20, 0.20], + "DimeNet": [0.50, 0.50], + "EGNN": [0.20, 0.20], + "PNAEq": [0.60, 0.60], + "PAINN": [0.60, 0.60], + "MACE": [0.60, 0.70], + } + if use_lengths and ("vector" not in ci_input): + thresholds["CGCNN"] = [0.175, 0.175] + thresholds["PNA"] = [0.10, 0.10] + thresholds["PNAPlus"] = [0.10, 0.10] + if use_lengths and "vector" in ci_input: + thresholds["PNA"] = [0.2, 0.15] + thresholds["PNAPlus"] = [0.2, 0.15] + if ci_input == "ci_conv_head.json": + thresholds["GIN"] = [0.26, 0.51] + thresholds["SchNet"] = [0.30, 0.30] + + verbosity = 2 + + for ihead in range(len(true_values)): + error_head_mse = error_mse_task[ihead] + error_str = ( + str("{:.6f}".format(error_head_mse)) + " < " + str(thresholds[mpnn_type][0]) + ) + hydragnn.utils.print.print_distributed(verbosity, "head: " + error_str) + assert ( + error_head_mse < thresholds[mpnn_type][0] + ), "Head RMSE checking failed for " + str(ihead) + + head_true = true_values[ihead] + head_pred = predicted_values[ihead] + # Check individual samples + mae = torch.nn.L1Loss() + sample_mean_abs_error = mae(head_true, head_pred) + error_str = ( + "{:.6f}".format(sample_mean_abs_error) + + " < " + + str(thresholds[mpnn_type][1]) + ) +> assert ( + sample_mean_abs_error < thresholds[mpnn_type][1] + ), f"MAE sample checking failed! MAE: {sample_mean_abs_error:.6f} >= threshold: {thresholds[mpnn_type][1]} for model: {mpnn_type}" +E AssertionError: MAE sample checking failed! MAE: 0.152753 >= threshold: 0.15 for model: PNA +E assert tensor(0.1528) < 0.15 + +tests/test_graphs.py:192: AssertionError +----------------------------- Captured stdout call ----------------------------- +dist.is_initialized(),sync_batch_norm,device_name: True False cpu +Using FSDP: False Sharding: ShardingStrategy.FULL_SHARD +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +dist.is_initialized(),sync_batch_norm,device_name: True False cpu +Using FSDP: False Sharding: ShardingStrategy.FULL_SHARD +get_autocast_and_scaler use_bf16: False +----------------------------- Captured stderr call ----------------------------- + Degree max: 0%| | 0/354 [00:00 unittest_train_model( + mpnn_type, None, None, "ci_conv_head.json", False, overwrite_data + ) + +tests/test_graphs.py:306: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_graphs.py:134: in unittest_train_model + hydragnn.run_training(config, use_deepspeed) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:97: in _ + model = get_distributed_model( +hydragnn/utils/distributed/distributed.py:379: in get_distributed_model + model = DDP(model, **ddp_kwargs) + ^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/nn/parallel/distributed.py:802: in __init__ + self._log_and_throw( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +self = DistributedDataParallel( + (module): SAGEStack( + (graph_convs): ModuleList( + (0): Sequential( + (0) - SA...=True, track_running_stats=True) + ) + ) + (activation_function): ReLU() + (graph_shared): ModuleDict() + ) +) +err_type = +err_msg = "Modules with uninitialized parameters can't be used with `DistributedDataParallel`. Run a dummy forward pass to correctly initialize the modules" + + def _log_and_throw(self, err_type, err_msg): + if self.logger is not None: + self.logger.set_error_and_log(f"{str(err_type)}: {err_msg}") +> raise err_type(err_msg) +E RuntimeError: Modules with uninitialized parameters can't be used with `DistributedDataParallel`. Run a dummy forward pass to correctly initialize the modules + +.venv/lib/python3.13/site-packages/torch/nn/parallel/distributed.py:1143: RuntimeError +----------------------------- Captured stdout call ----------------------------- +dist.is_initialized(),sync_batch_norm,device_name: True False cpu +Using FSDP: False Sharding: ShardingStrategy.FULL_SHARD +______________________ pytest_train_model_conv_head[GAT] _______________________ + +mpnn_type = 'GAT', overwrite_data = False + + @pytest.mark.parametrize( + "mpnn_type", + [ + "SAGE", + "GIN", + "GAT", + "MFC", + "PNA", + "PNAPlus", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + "PAINN", + ], + ) + def pytest_train_model_conv_head(mpnn_type, overwrite_data=False): +> unittest_train_model( + mpnn_type, None, None, "ci_conv_head.json", False, overwrite_data + ) + +tests/test_graphs.py:306: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_graphs.py:134: in unittest_train_model + hydragnn.run_training(config, use_deepspeed) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:97: in _ + model = get_distributed_model( +hydragnn/utils/distributed/distributed.py:379: in get_distributed_model + model = DDP(model, **ddp_kwargs) + ^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/nn/parallel/distributed.py:802: in __init__ + self._log_and_throw( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +self = DistributedDataParallel( + (module): GATStack( + (graph_convs): ModuleList( + (0): Sequential( + (0) - GAT...=True) + ) + ) + (activation_function): ReLU() + (out_lin): Identity() + (graph_shared): ModuleDict() + ) +) +err_type = +err_msg = "Modules with uninitialized parameters can't be used with `DistributedDataParallel`. Run a dummy forward pass to correctly initialize the modules" + + def _log_and_throw(self, err_type, err_msg): + if self.logger is not None: + self.logger.set_error_and_log(f"{str(err_type)}: {err_msg}") +> raise err_type(err_msg) +E RuntimeError: Modules with uninitialized parameters can't be used with `DistributedDataParallel`. Run a dummy forward pass to correctly initialize the modules + +.venv/lib/python3.13/site-packages/torch/nn/parallel/distributed.py:1143: RuntimeError +----------------------------- Captured stdout call ----------------------------- +dist.is_initialized(),sync_batch_norm,device_name: True False cpu +Using FSDP: False Sharding: ShardingStrategy.FULL_SHARD +______________________ pytest_train_model_conv_head[MFC] _______________________ + +mpnn_type = 'MFC', overwrite_data = False + + @pytest.mark.parametrize( + "mpnn_type", + [ + "SAGE", + "GIN", + "GAT", + "MFC", + "PNA", + "PNAPlus", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + "PAINN", + ], + ) + def pytest_train_model_conv_head(mpnn_type, overwrite_data=False): +> unittest_train_model( + mpnn_type, None, None, "ci_conv_head.json", False, overwrite_data + ) + +tests/test_graphs.py:306: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_graphs.py:134: in unittest_train_model + hydragnn.run_training(config, use_deepspeed) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:97: in _ + model = get_distributed_model( +hydragnn/utils/distributed/distributed.py:379: in get_distributed_model + model = DDP(model, **ddp_kwargs) + ^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/nn/parallel/distributed.py:802: in __init__ + self._log_and_throw( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +self = DistributedDataParallel( + (module): MFCStack( + (graph_convs): ModuleList( + (0): Sequential( + (0) - MFC...=True, track_running_stats=True) + ) + ) + (activation_function): ReLU() + (graph_shared): ModuleDict() + ) +) +err_type = +err_msg = "Modules with uninitialized parameters can't be used with `DistributedDataParallel`. Run a dummy forward pass to correctly initialize the modules" + + def _log_and_throw(self, err_type, err_msg): + if self.logger is not None: + self.logger.set_error_and_log(f"{str(err_type)}: {err_msg}") +> raise err_type(err_msg) +E RuntimeError: Modules with uninitialized parameters can't be used with `DistributedDataParallel`. Run a dummy forward pass to correctly initialize the modules + +.venv/lib/python3.13/site-packages/torch/nn/parallel/distributed.py:1143: RuntimeError +----------------------------- Captured stdout call ----------------------------- +dist.is_initialized(),sync_batch_norm,device_name: True False cpu +Using FSDP: False Sharding: ShardingStrategy.FULL_SHARD +______________________ pytest_train_model_conv_head[PNA] _______________________ + +mpnn_type = 'PNA', overwrite_data = False + + @pytest.mark.parametrize( + "mpnn_type", + [ + "SAGE", + "GIN", + "GAT", + "MFC", + "PNA", + "PNAPlus", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + "PAINN", + ], + ) + def pytest_train_model_conv_head(mpnn_type, overwrite_data=False): +> unittest_train_model( + mpnn_type, None, None, "ci_conv_head.json", False, overwrite_data + ) + +tests/test_graphs.py:306: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_graphs.py:132: in unittest_train_model + hydragnn.run_training(config_file, use_deepspeed) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:62: in _ + run_training(config) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:97: in _ + model = get_distributed_model( +hydragnn/utils/distributed/distributed.py:379: in get_distributed_model + model = DDP(model, **ddp_kwargs) + ^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/nn/parallel/distributed.py:802: in __init__ + self._log_and_throw( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +self = DistributedDataParallel( + (module): PNAStack( + (graph_convs): ModuleList( + (0): Sequential( + (0) - PNA...=True, track_running_stats=True) + ) + ) + (activation_function): ReLU() + (graph_shared): ModuleDict() + ) +) +err_type = +err_msg = "Modules with uninitialized parameters can't be used with `DistributedDataParallel`. Run a dummy forward pass to correctly initialize the modules" + + def _log_and_throw(self, err_type, err_msg): + if self.logger is not None: + self.logger.set_error_and_log(f"{str(err_type)}: {err_msg}") +> raise err_type(err_msg) +E RuntimeError: Modules with uninitialized parameters can't be used with `DistributedDataParallel`. Run a dummy forward pass to correctly initialize the modules + +.venv/lib/python3.13/site-packages/torch/nn/parallel/distributed.py:1143: RuntimeError +----------------------------- Captured stdout call ----------------------------- +dist.is_initialized(),sync_batch_norm,device_name: True False cpu +Using FSDP: False Sharding: ShardingStrategy.FULL_SHARD +----------------------------- Captured stderr call ----------------------------- + Degree max: 0%| | 0/350 [00:00 unittest_train_model( + mpnn_type, None, None, "ci_conv_head.json", False, overwrite_data + ) + +tests/test_graphs.py:306: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_graphs.py:134: in unittest_train_model + hydragnn.run_training(config, use_deepspeed) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:97: in _ + model = get_distributed_model( +hydragnn/utils/distributed/distributed.py:379: in get_distributed_model + model = DDP(model, **ddp_kwargs) + ^^^^^^^^^^^^^^^^^^^^^^^^ +.venv/lib/python3.13/site-packages/torch/nn/parallel/distributed.py:802: in __init__ + self._log_and_throw( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +self = DistributedDataParallel( + (module): PNAPlusStack( + (graph_convs): ModuleList( + (0): Sequential( + (0) -...unction): ReLU() + (graph_shared): ModuleDict() + (rbf): BesselBasisLayer( + (envelope): Envelope() + ) + ) +) +err_type = +err_msg = "Modules with uninitialized parameters can't be used with `DistributedDataParallel`. Run a dummy forward pass to correctly initialize the modules" + + def _log_and_throw(self, err_type, err_msg): + if self.logger is not None: + self.logger.set_error_and_log(f"{str(err_type)}: {err_msg}") +> raise err_type(err_msg) +E RuntimeError: Modules with uninitialized parameters can't be used with `DistributedDataParallel`. Run a dummy forward pass to correctly initialize the modules + +.venv/lib/python3.13/site-packages/torch/nn/parallel/distributed.py:1143: RuntimeError +----------------------------- Captured stdout call ----------------------------- +dist.is_initialized(),sync_batch_norm,device_name: True False cpu +Using FSDP: False Sharding: ShardingStrategy.FULL_SHARD +----------------------------- Captured stderr call ----------------------------- + Degree max: 0%| | 0/350 [00:00 unittest_train_model( + mpnn_type, None, None, "ci_conv_head.json", False, overwrite_data + ) + +tests/test_graphs.py:306: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +mpnn_type = 'SchNet', global_attn_engine = None, global_attn_type = None +ci_input = 'ci_conv_head.json', use_lengths = False, overwrite_data = False +use_deepspeed = False, overwrite_config = None + + def unittest_train_model( + mpnn_type, + global_attn_engine, + global_attn_type, + ci_input, + use_lengths, + overwrite_data=False, + use_deepspeed=False, + overwrite_config=None, + ): + world_size, rank = hydragnn.utils.distributed.get_comm_size_and_rank() + + os.environ["SERIALIZED_DATA_PATH"] = os.getcwd() + + # Read in config settings and override model type. + config_file = os.path.join(os.getcwd(), "tests/inputs", ci_input) + with open(config_file, "r") as f: + config = json.load(f) + config["NeuralNetwork"]["Architecture"]["global_attn_engine"] = global_attn_engine + config["NeuralNetwork"]["Architecture"]["global_attn_type"] = global_attn_type + config["NeuralNetwork"]["Architecture"]["mpnn_type"] = mpnn_type + + # Overwrite config settings if provided + if overwrite_config: + config = merge_config(config, overwrite_config) + + """ + to test this locally, set ci.json as + "Dataset": { + ... + "path": { + "train": "serialized_dataset/unit_test_singlehead_train.pkl", + "test": "serialized_dataset/unit_test_singlehead_test.pkl", + "validate": "serialized_dataset/unit_test_singlehead_validate.pkl"} + ... + """ + # use pkl files if exist by default + for dataset_name in config["Dataset"]["path"].keys(): + if dataset_name == "total": + pkl_file = ( + os.environ["SERIALIZED_DATA_PATH"] + + "/serialized_dataset/" + + config["Dataset"]["name"] + + ".pkl" + ) + else: + pkl_file = ( + os.environ["SERIALIZED_DATA_PATH"] + + "/serialized_dataset/" + + config["Dataset"]["name"] + + "_" + + dataset_name + + ".pkl" + ) + if os.path.exists(pkl_file): + config["Dataset"]["path"][dataset_name] = pkl_file + + # In the unit test runs, it is found MFC favors graph-level features over node-level features, compared with other models; + # hence here we decrease the loss weight coefficient for graph-level head in MFC. + if mpnn_type == "MFC" and ci_input == "ci_multihead.json": + config["NeuralNetwork"]["Architecture"]["task_weights"][0] = 2 + + # Only run with edge lengths for models that support them. + if use_lengths: + config["NeuralNetwork"]["Architecture"]["edge_features"] = ["lengths"] + + if rank == 0: + num_samples_tot = 500 + # check if serialized pickle files or folders for raw files provided + pkl_input = False + if list(config["Dataset"]["path"].values())[0].endswith(".pkl"): + pkl_input = True + # only generate new datasets, if not pkl + if not pkl_input: + for dataset_name, data_path in config["Dataset"]["path"].items(): + if overwrite_data: + shutil.rmtree(data_path) + if not os.path.exists(data_path): + os.makedirs(data_path) + if dataset_name == "total": + num_samples = num_samples_tot + elif dataset_name == "train": + num_samples = int( + num_samples_tot + * config["NeuralNetwork"]["Training"]["perc_train"] + ) + elif dataset_name == "test": + num_samples = int( + num_samples_tot + * (1 - config["NeuralNetwork"]["Training"]["perc_train"]) + * 0.5 + ) + elif dataset_name == "validate": + num_samples = int( + num_samples_tot + * (1 - config["NeuralNetwork"]["Training"]["perc_train"]) + * 0.5 + ) + if not os.listdir(data_path): + tests.deterministic_graph_data( + data_path, number_configurations=num_samples + ) + MPI.COMM_WORLD.Barrier() + + # Since the config file uses PNA already, test the file overload here. + # All the other models need to use the locally modified dictionary. + if mpnn_type == "PNA" and not use_lengths: + hydragnn.run_training(config_file, use_deepspeed) + else: + hydragnn.run_training(config, use_deepspeed) + + ( + error, + error_mse_task, + true_values, + predicted_values, + ) = hydragnn.run_prediction(config, use_deepspeed) + + # Set RMSE and sample MAE error thresholds + thresholds = { + "SAGE": [0.20, 0.20], + "PNA": [0.20, 0.20], + "PNAPlus": [0.20, 0.20], + "MFC": [0.20, 0.30], + "GIN": [0.25, 0.20], + "GAT": [0.60, 0.70], + "CGCNN": [0.50, 0.40], + "SchNet": [0.20, 0.20], + "DimeNet": [0.50, 0.50], + "EGNN": [0.20, 0.20], + "PNAEq": [0.60, 0.60], + "PAINN": [0.60, 0.60], + "MACE": [0.60, 0.70], + } + if use_lengths and ("vector" not in ci_input): + thresholds["CGCNN"] = [0.175, 0.175] + thresholds["PNA"] = [0.10, 0.10] + thresholds["PNAPlus"] = [0.10, 0.10] + if use_lengths and "vector" in ci_input: + thresholds["PNA"] = [0.2, 0.15] + thresholds["PNAPlus"] = [0.2, 0.15] + if ci_input == "ci_conv_head.json": + thresholds["GIN"] = [0.26, 0.51] + thresholds["SchNet"] = [0.30, 0.30] + + verbosity = 2 + + for ihead in range(len(true_values)): + error_head_mse = error_mse_task[ihead] + error_str = ( + str("{:.6f}".format(error_head_mse)) + " < " + str(thresholds[mpnn_type][0]) + ) + hydragnn.utils.print.print_distributed(verbosity, "head: " + error_str) +> assert ( + error_head_mse < thresholds[mpnn_type][0] + ), "Head RMSE checking failed for " + str(ihead) +E AssertionError: Head RMSE checking failed for 0 +E assert tensor(0.4173) < 0.3 + +tests/test_graphs.py:178: AssertionError +----------------------------- Captured stdout call ----------------------------- +dist.is_initialized(),sync_batch_norm,device_name: True False cpu +Using FSDP: False Sharding: ShardingStrategy.FULL_SHARD +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +dist.is_initialized(),sync_batch_norm,device_name: True False cpu +Using FSDP: False Sharding: ShardingStrategy.FULL_SHARD +get_autocast_and_scaler use_bf16: False +----------------------------- Captured stderr call ----------------------------- +0: Load existing model: ./logs/SchNet-r-2.0-ncl-2-hd-20-ne-100-lr-0.02-bs-32-data-unit_test-node_ft--task_weights-1.0-/SchNet-r-2.0-ncl-2-hd-20-ne-100-lr-0.02-bs-32-data-unit_test-node_ft--task_weights-1.0-.pk +0: head: 0.417273 < 0.3 +____________________ pytest_train_model_conv_head[DimeNet] _____________________ + +mpnn_type = 'DimeNet', overwrite_data = False + + @pytest.mark.parametrize( + "mpnn_type", + [ + "SAGE", + "GIN", + "GAT", + "MFC", + "PNA", + "PNAPlus", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + "PAINN", + ], + ) + def pytest_train_model_conv_head(mpnn_type, overwrite_data=False): +> unittest_train_model( + mpnn_type, None, None, "ci_conv_head.json", False, overwrite_data + ) + +tests/test_graphs.py:306: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_graphs.py:134: in unittest_train_model + hydragnn.run_training(config, use_deepspeed) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:89: in _ + model = create_model_config( +hydragnn/models/create.py:45: in create_model_config + return create_model( +hydragnn/models/create.py:369: in create_model + model = DIMEStack( +hydragnn/models/DIMEStack.py:68: in __init__ + super().__init__(input_args, conv_args, *args, **kwargs) +hydragnn/models/Base.py:174: in __init__ + self._init_conv() +hydragnn/models/DIMEStack.py:80: in _init_conv + self.get_conv( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +self = DIMEStack( + (graph_convs): ModuleList() + (feature_layers): ModuleList() + (heads_NN): ModuleList() + (convs_node_hid...eDict() + (convs_node_output): ModuleDict() + (batch_norms_node_output): ModuleDict() + (activation_function): ReLU() +) +input_dim = 0, output_dim = 20, edge_dim = None + + def get_conv(self, input_dim, output_dim, edge_dim=None): + hidden_dim = output_dim if input_dim == 1 else input_dim + assert ( +> hidden_dim > 1 + ), "DimeNet requires more than one hidden dimension between input_dim and output_dim." +E AssertionError: DimeNet requires more than one hidden dimension between input_dim and output_dim. + +hydragnn/models/DIMEStack.py:99: AssertionError +______________________ pytest_train_model_conv_head[EGNN] ______________________ + +mpnn_type = 'EGNN', overwrite_data = False + + @pytest.mark.parametrize( + "mpnn_type", + [ + "SAGE", + "GIN", + "GAT", + "MFC", + "PNA", + "PNAPlus", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + "PAINN", + ], + ) + def pytest_train_model_conv_head(mpnn_type, overwrite_data=False): +> unittest_train_model( + mpnn_type, None, None, "ci_conv_head.json", False, overwrite_data + ) + +tests/test_graphs.py:306: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +mpnn_type = 'EGNN', global_attn_engine = None, global_attn_type = None +ci_input = 'ci_conv_head.json', use_lengths = False, overwrite_data = False +use_deepspeed = False, overwrite_config = None + + def unittest_train_model( + mpnn_type, + global_attn_engine, + global_attn_type, + ci_input, + use_lengths, + overwrite_data=False, + use_deepspeed=False, + overwrite_config=None, + ): + world_size, rank = hydragnn.utils.distributed.get_comm_size_and_rank() + + os.environ["SERIALIZED_DATA_PATH"] = os.getcwd() + + # Read in config settings and override model type. + config_file = os.path.join(os.getcwd(), "tests/inputs", ci_input) + with open(config_file, "r") as f: + config = json.load(f) + config["NeuralNetwork"]["Architecture"]["global_attn_engine"] = global_attn_engine + config["NeuralNetwork"]["Architecture"]["global_attn_type"] = global_attn_type + config["NeuralNetwork"]["Architecture"]["mpnn_type"] = mpnn_type + + # Overwrite config settings if provided + if overwrite_config: + config = merge_config(config, overwrite_config) + + """ + to test this locally, set ci.json as + "Dataset": { + ... + "path": { + "train": "serialized_dataset/unit_test_singlehead_train.pkl", + "test": "serialized_dataset/unit_test_singlehead_test.pkl", + "validate": "serialized_dataset/unit_test_singlehead_validate.pkl"} + ... + """ + # use pkl files if exist by default + for dataset_name in config["Dataset"]["path"].keys(): + if dataset_name == "total": + pkl_file = ( + os.environ["SERIALIZED_DATA_PATH"] + + "/serialized_dataset/" + + config["Dataset"]["name"] + + ".pkl" + ) + else: + pkl_file = ( + os.environ["SERIALIZED_DATA_PATH"] + + "/serialized_dataset/" + + config["Dataset"]["name"] + + "_" + + dataset_name + + ".pkl" + ) + if os.path.exists(pkl_file): + config["Dataset"]["path"][dataset_name] = pkl_file + + # In the unit test runs, it is found MFC favors graph-level features over node-level features, compared with other models; + # hence here we decrease the loss weight coefficient for graph-level head in MFC. + if mpnn_type == "MFC" and ci_input == "ci_multihead.json": + config["NeuralNetwork"]["Architecture"]["task_weights"][0] = 2 + + # Only run with edge lengths for models that support them. + if use_lengths: + config["NeuralNetwork"]["Architecture"]["edge_features"] = ["lengths"] + + if rank == 0: + num_samples_tot = 500 + # check if serialized pickle files or folders for raw files provided + pkl_input = False + if list(config["Dataset"]["path"].values())[0].endswith(".pkl"): + pkl_input = True + # only generate new datasets, if not pkl + if not pkl_input: + for dataset_name, data_path in config["Dataset"]["path"].items(): + if overwrite_data: + shutil.rmtree(data_path) + if not os.path.exists(data_path): + os.makedirs(data_path) + if dataset_name == "total": + num_samples = num_samples_tot + elif dataset_name == "train": + num_samples = int( + num_samples_tot + * config["NeuralNetwork"]["Training"]["perc_train"] + ) + elif dataset_name == "test": + num_samples = int( + num_samples_tot + * (1 - config["NeuralNetwork"]["Training"]["perc_train"]) + * 0.5 + ) + elif dataset_name == "validate": + num_samples = int( + num_samples_tot + * (1 - config["NeuralNetwork"]["Training"]["perc_train"]) + * 0.5 + ) + if not os.listdir(data_path): + tests.deterministic_graph_data( + data_path, number_configurations=num_samples + ) + MPI.COMM_WORLD.Barrier() + + # Since the config file uses PNA already, test the file overload here. + # All the other models need to use the locally modified dictionary. + if mpnn_type == "PNA" and not use_lengths: + hydragnn.run_training(config_file, use_deepspeed) + else: + hydragnn.run_training(config, use_deepspeed) + + ( + error, + error_mse_task, + true_values, + predicted_values, + ) = hydragnn.run_prediction(config, use_deepspeed) + + # Set RMSE and sample MAE error thresholds + thresholds = { + "SAGE": [0.20, 0.20], + "PNA": [0.20, 0.20], + "PNAPlus": [0.20, 0.20], + "MFC": [0.20, 0.30], + "GIN": [0.25, 0.20], + "GAT": [0.60, 0.70], + "CGCNN": [0.50, 0.40], + "SchNet": [0.20, 0.20], + "DimeNet": [0.50, 0.50], + "EGNN": [0.20, 0.20], + "PNAEq": [0.60, 0.60], + "PAINN": [0.60, 0.60], + "MACE": [0.60, 0.70], + } + if use_lengths and ("vector" not in ci_input): + thresholds["CGCNN"] = [0.175, 0.175] + thresholds["PNA"] = [0.10, 0.10] + thresholds["PNAPlus"] = [0.10, 0.10] + if use_lengths and "vector" in ci_input: + thresholds["PNA"] = [0.2, 0.15] + thresholds["PNAPlus"] = [0.2, 0.15] + if ci_input == "ci_conv_head.json": + thresholds["GIN"] = [0.26, 0.51] + thresholds["SchNet"] = [0.30, 0.30] + + verbosity = 2 + + for ihead in range(len(true_values)): + error_head_mse = error_mse_task[ihead] + error_str = ( + str("{:.6f}".format(error_head_mse)) + " < " + str(thresholds[mpnn_type][0]) + ) + hydragnn.utils.print.print_distributed(verbosity, "head: " + error_str) +> assert ( + error_head_mse < thresholds[mpnn_type][0] + ), "Head RMSE checking failed for " + str(ihead) +E AssertionError: Head RMSE checking failed for 0 +E assert tensor(0.2124) < 0.2 + +tests/test_graphs.py:178: AssertionError +----------------------------- Captured stdout call ----------------------------- +dist.is_initialized(),sync_batch_norm,device_name: True False cpu +Using FSDP: False Sharding: ShardingStrategy.FULL_SHARD +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +get_autocast_and_scaler use_bf16: False +dist.is_initialized(),sync_batch_norm,device_name: True False cpu +Using FSDP: False Sharding: ShardingStrategy.FULL_SHARD +get_autocast_and_scaler use_bf16: False +----------------------------- Captured stderr call ----------------------------- +0: Load existing model: ./logs/EGNN-r-2.0-ncl-2-hd-20-ne-100-lr-0.02-bs-32-data-unit_test-node_ft--task_weights-1.0-/EGNN-r-2.0-ncl-2-hd-20-ne-100-lr-0.02-bs-32-data-unit_test-node_ft--task_weights-1.0-.pk +0: head: 0.212449 < 0.2 +_____________________ pytest_train_model_conv_head[PNAEq] ______________________ + +mpnn_type = 'PNAEq', overwrite_data = False + + @pytest.mark.parametrize( + "mpnn_type", + [ + "SAGE", + "GIN", + "GAT", + "MFC", + "PNA", + "PNAPlus", + "SchNet", + "DimeNet", + "EGNN", + "PNAEq", + "PAINN", + ], + ) + def pytest_train_model_conv_head(mpnn_type, overwrite_data=False): +> unittest_train_model( + mpnn_type, None, None, "ci_conv_head.json", False, overwrite_data + ) + +tests/test_graphs.py:306: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_graphs.py:134: in unittest_train_model + hydragnn.run_training(config, use_deepspeed) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:89: in _ + model = create_model_config( +hydragnn/models/create.py:45: in create_model_config + return create_model( +hydragnn/models/create.py:455: in create_model + model = PNAEqStack( +hydragnn/models/PNAEqStack.py:72: in __init__ + super().__init__(input_args, conv_args, *args, **kwargs) +hydragnn/models/Base.py:174: in __init__ + self._init_conv() +hydragnn/models/PNAEqStack.py:80: in _init_conv + self.get_conv( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +self = PNAEqStack( + (graph_convs): ModuleList() + (feature_layers): ModuleList() + (heads_NN): ModuleList() + (convs_node_hi...eDict() + (convs_node_output): ModuleDict() + (batch_norms_node_output): ModuleDict() + (activation_function): ReLU() +) +input_dim = 0, output_dim = 20, last_layer = False, edge_dim = None + + def get_conv(self, input_dim, output_dim, last_layer=False, edge_dim=None): + hidden_dim = output_dim if input_dim == 1 else input_dim + assert ( +> hidden_dim > 1 + ), "PNAEq requires more than one hidden dimension between input_dim and output_dim." +E AssertionError: PNAEq requires more than one hidden dimension between input_dim and output_dim. + +hydragnn/models/PNAEqStack.py:106: AssertionError +----------------------------- Captured stderr call ----------------------------- + Degree max: 0%| | 0/350 [00:00 unittest_train_model( + mpnn_type, None, None, "ci_conv_head.json", False, overwrite_data + ) + +tests/test_graphs.py:306: +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ +tests/test_graphs.py:134: in unittest_train_model + hydragnn.run_training(config, use_deepspeed) +/opt/homebrew/Cellar/python@3.13/3.13.7/Frameworks/Python.framework/Versions/3.13/lib/python3.13/functools.py:934: in wrapper + return dispatch(args[0].__class__)(*args, **kw) + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +hydragnn/run_training.py:89: in _ + model = create_model_config( +hydragnn/models/create.py:45: in create_model_config + return create_model( +hydragnn/models/create.py:428: in create_model + model = PAINNStack( +hydragnn/models/PAINNStack.py:47: in __init__ + super().__init__(input_args, conv_args, *args, **kwargs) +hydragnn/models/Base.py:174: in __init__ + self._init_conv() +hydragnn/models/PAINNStack.py:53: in _init_conv + self.get_conv( +_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ + +self = PAINNStack( + (graph_convs): ModuleList() + (feature_layers): ModuleList() + (heads_NN): ModuleList() + (convs_node_hi...eDict() + (convs_node_output): ModuleDict() + (batch_norms_node_output): ModuleDict() + (activation_function): ReLU() +) +input_dim = 0, output_dim = 20, last_layer = False, edge_dim = None + + def get_conv(self, input_dim, output_dim, last_layer=False, edge_dim=None): + hidden_dim = output_dim if input_dim == 1 else input_dim + assert ( +> hidden_dim > 1 + ), "PainnNet requires more than one hidden dimension between input_dim and output_dim." +E AssertionError: PainnNet requires more than one hidden dimension between input_dim and output_dim. + +hydragnn/models/PAINNStack.py:79: AssertionError +=========================== short test summary info ============================ +FAILED tests/test_graphs.py::pytest_train_model_vectoroutput[PNA] - Assertion... +FAILED tests/test_graphs.py::pytest_train_model_conv_head[SAGE] - RuntimeErro... +FAILED tests/test_graphs.py::pytest_train_model_conv_head[GAT] - RuntimeError... +FAILED tests/test_graphs.py::pytest_train_model_conv_head[MFC] - RuntimeError... +FAILED tests/test_graphs.py::pytest_train_model_conv_head[PNA] - RuntimeError... +FAILED tests/test_graphs.py::pytest_train_model_conv_head[PNAPlus] - RuntimeE... +FAILED tests/test_graphs.py::pytest_train_model_conv_head[SchNet] - Assertion... +FAILED tests/test_graphs.py::pytest_train_model_conv_head[DimeNet] - Assertio... +FAILED tests/test_graphs.py::pytest_train_model_conv_head[EGNN] - AssertionEr... +FAILED tests/test_graphs.py::pytest_train_model_conv_head[PNAEq] - AssertionE... +FAILED tests/test_graphs.py::pytest_train_model_conv_head[PAINN] - AssertionE... +================== 11 failed, 57 passed in 564.16s (0:09:24) =================== diff --git a/tests/inputs/ci.json b/tests/inputs/ci.json index dadf58d95..aa32529e2 100644 --- a/tests/inputs/ci.json +++ b/tests/inputs/ci.json @@ -64,10 +64,18 @@ "task_weights": [1.0] }, "Variables_of_interest": { - "input_node_features": [0], - "output_names": ["sum_x_x2_x3"], - "output_index": [0], - "type": ["graph"], + "node_features": { + "x": { + "dim": 1, + "role": "input" + } + }, + "graph_features": { + "sum_x_x2_x3": { + "dim": 1, + "role": "output" + } + }, "denormalize_output": false }, "Training": { diff --git a/tests/inputs/ci_conv_head.json b/tests/inputs/ci_conv_head.json index 9b06d8ae8..0073bceb3 100644 --- a/tests/inputs/ci_conv_head.json +++ b/tests/inputs/ci_conv_head.json @@ -55,10 +55,17 @@ "task_weights": [1.0] }, "Variables_of_interest": { - "input_node_features": [0], - "output_names": ["x"], - "output_index": [0], - "type": ["node"], + "node_features": { + "x": { + "dim": 1, + "role": "input" + }, + "x2": { + "dim": 1, + "role": "output" + } + }, + "graph_features": {}, "denormalize_output": false }, "Training": { diff --git a/tests/inputs/ci_equivariant.json b/tests/inputs/ci_equivariant.json index 4910059e2..8fb031ee7 100644 --- a/tests/inputs/ci_equivariant.json +++ b/tests/inputs/ci_equivariant.json @@ -64,10 +64,18 @@ "task_weights": [1.0] }, "Variables_of_interest": { - "input_node_features": [0], - "output_names": ["sum_x_x2_x3"], - "output_index": [0], - "type": ["graph"], + "node_features": { + "x": { + "dim": 1, + "role": "input" + } + }, + "graph_features": { + "sum_x_x2_x3": { + "dim": 1, + "role": "output" + } + }, "denormalize_output": false }, "Training": { diff --git a/tests/inputs/ci_multihead.json b/tests/inputs/ci_multihead.json index e334bac13..38ac1f50a 100644 --- a/tests/inputs/ci_multihead.json +++ b/tests/inputs/ci_multihead.json @@ -58,13 +58,29 @@ "type": "mlp" } }, - "task_weights": [20.0, 1.0, 1.0, 1.0] + "task_weights": [20.0, 1.0, 1.0] }, "Variables_of_interest": { - "input_node_features": [0], - "output_names": ["sum_x_x2_x3","x","x2","x3"], - "output_index": [0,0,1,2], - "type": ["graph","node","node","node"], + "node_features": { + "x": { + "dim": 1, + "role": "input" + }, + "x2": { + "dim": 1, + "role": "output" + }, + "x3": { + "dim": 1, + "role": "output" + } + }, + "graph_features": { + "sum_x_x2_x3": { + "dim": 1, + "role": "output" + } + }, "denormalize_output": false }, "Training": { diff --git a/tests/inputs/ci_vectoroutput.json b/tests/inputs/ci_vectoroutput.json index 696f45630..bdbe64f06 100644 --- a/tests/inputs/ci_vectoroutput.json +++ b/tests/inputs/ci_vectoroutput.json @@ -58,13 +58,37 @@ "type": "mlp" } }, - "task_weights": [1.0, 1.0, 1.0,1.0, 1.0, 1.0] + "task_weights": [1.0, 1.0, 1.0, 1.0, 1.0] }, "Variables_of_interest": { - "input_node_features": [0], - "output_names": ["x2x3_vec","sum","sums_vec","sum_linear","x","xx2_vec"], - "output_index": [ 2, 0, 1, 2, 1, 0], - "type": ["node","graph","graph","graph","node","node"], + "node_features": { + "xx2_vec": { + "dim": 2, + "role": "output" + }, + "x": { + "dim": 1, + "role": "input" + }, + "x2x3_vec": { + "dim": 2, + "role": "output" + } + }, + "graph_features": { + "sum": { + "dim": 1, + "role": "output" + }, + "sums_vec": { + "dim": 2, + "role": "output" + }, + "sum_linear": { + "dim": 1, + "role": "output" + } + }, "denormalize_output": false }, "Training": { diff --git a/tests/test_feature_config.py b/tests/test_feature_config.py new file mode 100644 index 000000000..5287f1fde --- /dev/null +++ b/tests/test_feature_config.py @@ -0,0 +1,240 @@ +############################################################################## +# Copyright (c) 2024, Oak Ridge National Laboratory # +# All rights reserved. # +# # +# This file is part of HydraGNN and is distributed under a BSD 3-clause # +# license. For the licensing terms see the LICENSE file in the top-level # +# directory. # +# # +# SPDX-License-Identifier: BSD-3-Clause # +############################################################################## + +""" +Tests for feature configuration parsing and validation. +""" + +import pytest +import torch +from torch_geometric.data import Data + +from hydragnn.utils.input_config_parsing.feature_config import ( + parse_feature_config, + validate_feature_config, + update_var_config_with_features, + print_feature_summary, + get_feature_schema_example, +) + + +def test_parse_new_format(): + """Test parsing new feature configuration format.""" + var_config = { + "node_features": { + "atomic_number": {"dim": 1, "role": "input"}, + "coordinates": {"dim": 3, "role": "input"}, + "forces": {"dim": 3, "role": "output", "output_type": "node"}, + }, + "graph_features": { + "energy": {"dim": 1, "role": "output", "output_type": "graph"} + }, + } + + parsed = parse_feature_config(var_config) + + assert parsed["node_feature_names"] == ["atomic_number", "coordinates", "forces"] + assert parsed["node_feature_dims"] == [1, 3, 3] + assert parsed["graph_feature_names"] == ["energy"] + assert parsed["graph_feature_dims"] == [1] + assert parsed["input_node_features"] == [0, 1, 2, 3] # atomic_number, coordinates + assert parsed["output_names"] == ["forces", "energy"] + assert parsed["output_dim"] == [3, 1] + assert parsed["type"] == ["node", "graph"] + + +def test_parse_legacy_format(): + """Test parsing legacy feature configuration format.""" + var_config = { + "node_feature_names": ["atomic_number", "coordinates", "forces"], + "node_feature_dims": [1, 3, 3], + "graph_feature_names": ["energy"], + "graph_feature_dims": [1], + "input_node_features": [0, 1, 2, 3], + "output_names": ["energy"], + "output_index": [0], + "output_dim": [1], + "type": ["graph"], + } + + parsed = parse_feature_config(var_config) + + assert parsed["node_feature_names"] == ["atomic_number", "coordinates", "forces"] + assert parsed["node_feature_dims"] == [1, 3, 3] + assert parsed["graph_feature_names"] == ["energy"] + assert parsed["graph_feature_dims"] == [1] + assert parsed["input_node_features"] == [0, 1, 2, 3] + assert parsed["output_names"] == ["energy"] + + +def test_validate_config_valid(): + """Test validation of valid configuration.""" + var_config = { + "node_features": { + "atomic_number": {"dim": 1, "role": "input"}, + "coordinates": {"dim": 3, "role": "input"}, + }, + "graph_features": {"energy": {"dim": 1, "role": "output"}}, + } + + is_valid, errors = validate_feature_config(var_config) + assert is_valid + assert len(errors) == 0 + + +def test_validate_config_mismatched_lengths(): + """Test validation catches mismatched lengths.""" + var_config = { + "node_feature_names": ["atomic_number", "coordinates"], + "node_feature_dims": [1], # Wrong length! + "graph_feature_names": ["energy"], + "graph_feature_dims": [1], + "input_node_features": [0], + "output_names": ["energy"], + "output_index": [0], + "output_dim": [1], + "type": ["graph"], + } + + is_valid, errors = validate_feature_config(var_config) + assert not is_valid + assert any("node_feature_names" in err for err in errors) + + +def test_validate_config_invalid_indices(): + """Test validation catches invalid indices.""" + var_config = { + "node_features": { + "atomic_number": {"dim": 1, "role": "input"}, + }, + "graph_features": { + "energy": {"dim": 1, "role": "output", "output_type": "graph"} + }, + } + # Manually add invalid index + parsed = parse_feature_config(var_config) + parsed["input_node_features"] = [0, 5] # 5 is out of range + + is_valid, errors = validate_feature_config( + {"node_features": var_config["node_features"]} + ) + # This should be valid initially + assert is_valid + + +def test_validate_against_data(): + """Test validation against actual data object.""" + var_config = { + "node_features": { + "atomic_number": {"dim": 1, "role": "input"}, + "coordinates": {"dim": 3, "role": "input"}, + } + } + + # Create matching data object + data = Data(x=torch.randn(10, 4)) # 10 nodes, 4 features (1+3) + + is_valid, errors = validate_feature_config(var_config, data) + assert is_valid + assert len(errors) == 0 + + +def test_validate_against_data_mismatch(): + """Test validation catches dimension mismatch with data.""" + var_config = { + "node_features": { + "atomic_number": {"dim": 1, "role": "input"}, + "coordinates": {"dim": 3, "role": "input"}, + } + } + + # Create mismatched data object + data = Data(x=torch.randn(10, 5)) # 10 nodes, 5 features (should be 4) + + is_valid, errors = validate_feature_config(var_config, data) + assert not is_valid + assert any("dimension" in err.lower() for err in errors) + + +def test_update_var_config(): + """Test updating var_config with parsed features.""" + var_config = { + "node_features": { + "atomic_number": {"dim": 1, "role": "input"}, + "coordinates": {"dim": 3, "role": "input"}, + }, + "graph_features": {"energy": {"dim": 1, "role": "output"}}, + } + + updated = update_var_config_with_features(var_config) + + # Should have legacy keys now + assert "node_feature_names" in updated + assert "node_feature_dims" in updated + assert "graph_feature_names" in updated + assert "graph_feature_dims" in updated + assert "input_node_features" in updated + + +def test_get_feature_schema_example(): + """Test that example schema is valid.""" + example = get_feature_schema_example() + + # Should be valid + is_valid, errors = validate_feature_config(example) + assert is_valid, f"Example schema should be valid. Errors: {errors}" + + +def test_print_feature_summary(): + """Test printing feature summary.""" + var_config = { + "node_features": { + "atomic_number": {"dim": 1, "role": "input"}, + "coordinates": {"dim": 3, "role": "input"}, + }, + "graph_features": {"energy": {"dim": 1, "role": "output"}}, + } + + summary = print_feature_summary(var_config) + + assert "Feature Configuration Summary" in summary + assert "atomic_number" in summary + assert "coordinates" in summary + assert "energy" in summary + assert "dim=1" in summary + assert "dim=3" in summary + + +def test_empty_config(): + """Test handling of empty configuration.""" + var_config = { + "node_features": {}, + "graph_features": {}, + } + + parsed = parse_feature_config(var_config) + + assert parsed["node_feature_names"] == [] + assert parsed["graph_feature_names"] == [] + assert parsed["input_node_features"] == [] + + +def test_missing_config_raises(): + """Test that completely invalid config raises error.""" + var_config = {} # No features at all + + with pytest.raises(ValueError): + parse_feature_config(var_config) + + +if __name__ == "__main__": + # Run tests + pytest.main([__file__, "-v"]) diff --git a/tests/test_feature_config_validation.py b/tests/test_feature_config_validation.py new file mode 100644 index 000000000..768e22a56 --- /dev/null +++ b/tests/test_feature_config_validation.py @@ -0,0 +1,42 @@ +import pytest +import torch +from hydragnn.utils.input_config_parsing.feature_config import ( + validate_node_feature_columns, +) + + +def test_validate_node_feature_columns_simple(): + # 2 features: dim=1 and dim=3, total columns=4 + node_feature_dims = [1, 3] + data_x = torch.zeros((10, 4)) + # Should pass + validate_node_feature_columns(data_x, node_feature_dims) + # Should fail if data_x has extra columns + + data_x_extra = torch.zeros((10, 5)) + with pytest.raises(ValueError): + validate_node_feature_columns(data_x_extra, node_feature_dims) + + +def test_validate_node_feature_columns_with_column_indices(): + # 2 features: dim=1 at col 0, dim=3 at col 2, total columns=5 + node_feature_dims = [1, 3] + column_indices = [0, 2] + data_x = torch.zeros((10, 5)) + # Should pass + validate_node_feature_columns(data_x, node_feature_dims, column_indices) + # Should fail if second feature overruns columns + bad_column_indices = [0, 3] + with pytest.raises(ValueError): + validate_node_feature_columns(data_x, node_feature_dims, bad_column_indices) + + +def test_validate_node_feature_columns_edge_cases(): + # Single feature, dim=5 + node_feature_dims = [5] + data_x = torch.zeros((10, 5)) + validate_node_feature_columns(data_x, node_feature_dims) + # Should fail if not enough columns + data_x_bad = torch.zeros((10, 4)) + with pytest.raises(ValueError): + validate_node_feature_columns(data_x_bad, node_feature_dims) diff --git a/tests/test_model_loadpred.py b/tests/test_model_loadpred.py index 8b3617959..b73ab42ce 100755 --- a/tests/test_model_loadpred.py +++ b/tests/test_model_loadpred.py @@ -14,6 +14,9 @@ import hydragnn from tests.test_graphs import unittest_train_model from hydragnn.utils.input_config_parsing.config_utils import update_config +from hydragnn.utils.input_config_parsing.feature_config import ( + update_var_config_with_features, +) def unittest_model_prediction(config): @@ -69,6 +72,10 @@ def pytest_model_loadpred(): with open(config_file, "r") as f: config = json.load(f) config["NeuralNetwork"]["Architecture"]["model_type"] = model_type + # Update var_config with features before accessing input_node_features + var_config = config["NeuralNetwork"]["Variables_of_interest"] + var_config = update_var_config_with_features(var_config) + config["NeuralNetwork"]["Variables_of_interest"] = var_config # get the directory of trained model log_name = hydragnn.utils.input_config_parsing.config_utils.get_log_name_config( config @@ -83,6 +90,10 @@ def pytest_model_loadpred(): else: with open(config_file, "r") as f: config = json.load(f) + # Update var_config after loading saved config + var_config = config["NeuralNetwork"]["Variables_of_interest"] + var_config = update_var_config_with_features(var_config) + config["NeuralNetwork"]["Variables_of_interest"] = var_config for dataset_name, raw_data_path in config["Dataset"]["path"].items(): if not os.path.isfile(raw_data_path): print(dataset_name, "datasets not found: ", raw_data_path)