MERRA-2

-

Global atmospheric reanalysis dataset produced by NASA Global Modeling and Assimilation Office (GMAO), +

Global atmospheric reanalysis produced by NASA Global Modeling and Assimilation Office (GMAO), providing physically consistent meteorological fields for weather, climate, and hazard modeling.

+
+

At-a-glance (Quick Facts)

+
+ ++++ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +

Producer

NASA GMAO (GEOS)

Spatial coverage

Global

Spatial resolution

~0.5° (lat) × 0.625° (lon)

Temporal resolution

Hourly (also 3-hourly / daily / monthly products)

Data update frequency

Published monthly; typically available ~2–3 weeks after month end.

Period of record

1980–present

Format

NetCDF4

Grid / CRS

Regular latitude–longitude grid

Typical role in PyHazards

Core meteorological driver (forcing + covariates)

+
+

Overview

MERRA-2 (Modern-Era Retrospective analysis for Research and Applications, Version 2) is a global atmospheric reanalysis that assimilates satellite and conventional observations into a numerical weather prediction system to produce gridded, time-continuous estimates of the atmospheric state.

-

It is widely used as a core meteorological driver in climate analysis and hazard prediction pipelines, +

It is widely used as a meteorological backbone for climate analysis and hazard prediction pipelines, including wildfire, hurricane, drought, and extreme heat studies.

+
+

Stats

+
+ ++++++ + + + + + + + + + + + + + + +

Spatial

Temporal

Update cadence

Coverage

0.5° × 0.625°

Hourly

Monthly (+ ~2–3 weeks latency)

Global, 1980–present

+
+
+

Data Characteristics

  • Spatial coverage: Global

  • Spatial resolution: ~0.5° latitude × 0.625° longitude

    • -

  • Temporal resolution: Hourly

    • +

  • Temporal resolution: Hourly (with derived 3-hourly/daily/monthly products)

  • Vertical structure: Surface fields and multi-level pressure profiles

    • -

  • Data format: NetCDF

    • +

  • Data format: NetCDF4

  • Coordinate system: Regular latitude–longitude grid

    • +

  • Data update frequency: Monthly publication; new data typically appear ~2–3 weeks after month end +(common operational latency for standard MERRA-2 streams)

@@ -318,11 +378,16 @@

Typical Use Cases

Access

-

MERRA-2 data can be accessed via NASA GMAO and Earthdata services:

+

MERRA-2 data can be accessed via NASA Earthdata / GES DISC services:

+
+

Note

+

For standard MERRA-2 streams, operational latency is commonly documented as +~2–3 weeks after the end of each month (monthly publication cadence).

+
@@ -339,26 +404,8 @@

Reference - -
-
- Next -
-
ERA5
-
- - - - -
-
- Previous -
- -
Datasets
- -
- + +
@@ -391,6 +438,7 @@

ReferenceMERRA-2
  • Overview
    • +
  • Stats
  • Data Characteristics
  • Variables
  • Typical Use Cases
    • diff --git a/docs/searchindex.js b/docs/searchindex.js index a070b103..72dcaad7 100644 --- a/docs/searchindex.js +++ b/docs/searchindex.js @@ -1 +1 @@ -Search.setIndex({"alltitles": {"Academic Publications": [[25, "academic-publications"]], "Access": [[8, "access"], [9, "access"], [10, "access"], [11, "access"], [12, "access"], [13, "access"], [14, "access"], [15, "access"]], "Basic Usage": [[24, "basic-usage"]], "Build a built-in model": [[22, "build-a-built-in-model"]], "Contributors": [[26, "contributors"]], "Core Components": [[17, "core-components"]], "Core Contributors": [[26, "core-contributors"]], "Core Team": [[26, null]], "Core classes": [[19, "core-classes"], [21, "core-classes"]], "Core modules": [[20, "core-modules"], [22, "core-modules"]], "Data Characteristics": [[8, "data-characteristics"], [9, "data-characteristics"], [10, "data-characteristics"], [11, "data-characteristics"], [12, "data-characteristics"], [13, "data-characteristics"], [14, "data-characteristics"], [15, "data-characteristics"]], "Dataset inspection": [[19, "dataset-inspection"]], "Datasets": [[16, "datasets"], [19, null], [19, "id1"]], "Design notes": [[22, "design-notes"]], "Distributed and devices": [[20, "distributed-and-devices"]], "ERA5": [[8, null]], "Engine": [[20, null]], "Example skeleton": [[19, "example-skeleton"]], "FIRMS": [[9, null]], "GOES-R": [[10, null]], "GPU Support": [[24, "gpu-support"]], "How to Cite": [[7, null], [17, "how-to-cite"]], "Implementation Guide": [[16, null]], "Indices and tables": [[17, "indices-and-tables"]], "Installation": [[18, null]], "Installing PyHazards": [[18, "installing-pyhazards"]], "LANDFIRE": [[11, null]], "Lead Developer": [[26, "lead-developer"]], "MERRA-2": [[12, null]], "MTBS": [[13, null]], "Mamba-based wildfire model (spatio-temporal)": [[22, "mamba-based-wildfire-model-spatio-temporal"]], "Metrics": [[16, "metrics"], [21, null]], "Models": [[16, "models"], [22, null]], "Module contents": [[1, "module-pyhazards"], [2, "module-pyhazards.datasets"], [3, "module-pyhazards.engine"], [5, "module-pyhazards.models"], [6, "module-pyhazards.utils"]], "NOAA Flood Events": [[14, null]], "Next Steps": [[24, "next-steps"]], "Overview": [[8, "overview"], [9, "overview"], [10, "overview"], [11, "overview"], [12, "overview"], [13, "overview"], [14, "overview"], [15, "overview"]], "Principal Contributors & Maintainers": [[26, "principal-contributors-maintainers"]], "Quick Start": [[24, null]], "Reference": [[8, "reference"], [9, "reference"], [10, "reference"], [11, "reference"], [12, "reference"], [13, "reference"], [14, "reference"], [15, "reference"]], "References": [[25, null]], "Register a custom model": [[22, "register-a-custom-model"]], "Requirements": [[18, "requirements"]], "Submodules": [[2, "submodules"], [3, "submodules"], [5, "submodules"], [6, "submodules"], [23, "submodules"]], "Subpackages": [[1, "subpackages"]], "Summary": [[19, "summary"], [20, "summary"], [21, "summary"], [22, "summary"], [23, "summary"]], "Toy Example (tabular classification)": [[24, "toy-example-tabular-classification"]], "Training": [[16, "training"]], "Transforms": [[16, "transforms"]], "Typical Use Cases": [[8, "typical-use-cases"], [9, "typical-use-cases"], [10, "typical-use-cases"], [11, "typical-use-cases"], [12, "typical-use-cases"], [13, "typical-use-cases"], [14, "typical-use-cases"], [15, "typical-use-cases"]], "Typical usage": [[20, "typical-usage"]], "Usage": [[21, "usage"]], "Utils": [[23, null]], "Variables": [[8, "variables"], [9, "variables"], [10, "variables"], [11, "variables"], [12, "variables"], [13, "variables"], [14, "variables"], [15, "variables"]], "WFIGS": [[15, null]], "Wildfire Mamba (spatio-temporal toy)": [[17, null]], "pyhazards": [[0, null]], "pyhazards package": [[1, null]], "pyhazards.datasets package": [[2, null]], "pyhazards.datasets.base module": [[2, "module-pyhazards.datasets.base"]], "pyhazards.datasets.hazards package": [[2, "module-pyhazards.datasets.hazards"]], "pyhazards.datasets.registry module": [[2, "module-pyhazards.datasets.registry"]], "pyhazards.datasets.transforms package": [[2, "module-pyhazards.datasets.transforms"]], "pyhazards.engine package": [[3, null]], "pyhazards.engine.distributed module": [[3, "module-pyhazards.engine.distributed"]], "pyhazards.engine.inference module": [[3, "module-pyhazards.engine.inference"]], "pyhazards.engine.trainer module": [[3, "module-pyhazards.engine.trainer"]], "pyhazards.metrics package": [[4, null]], "pyhazards.models package": [[5, null]], "pyhazards.models.backbones module": [[5, "module-pyhazards.models.backbones"]], "pyhazards.models.builder module": [[5, "module-pyhazards.models.builder"]], "pyhazards.models.heads module": [[5, "module-pyhazards.models.heads"]], "pyhazards.models.registry module": [[5, "module-pyhazards.models.registry"]], "pyhazards.utils package": [[6, null]], "pyhazards.utils.common module": [[6, "module-pyhazards.utils.common"]], "pyhazards.utils.hardware module": [[6, "module-pyhazards.utils.hardware"]]}, "docnames": ["api/modules", "api/pyhazards", "api/pyhazards.datasets", "api/pyhazards.engine", "api/pyhazards.metrics", "api/pyhazards.models", "api/pyhazards.utils", "cite", "datasets/era5", "datasets/firms", "datasets/goesr", "datasets/landfire", "datasets/merra2", "datasets/mtbs", "datasets/noaa_flood", "datasets/wfigs", "implementation", "index", "installation", "pyhazards_datasets", "pyhazards_engine", "pyhazards_metrics", "pyhazards_models", "pyhazards_utils", "quick_start", "references", "team"], "envversion": {"sphinx": 64, "sphinx.domains.c": 3, "sphinx.domains.changeset": 1, "sphinx.domains.citation": 1, "sphinx.domains.cpp": 9, "sphinx.domains.index": 1, "sphinx.domains.javascript": 3, "sphinx.domains.math": 2, "sphinx.domains.python": 4, "sphinx.domains.rst": 2, "sphinx.domains.std": 2, "sphinx.ext.viewcode": 1}, "filenames": ["api/modules.rst", "api/pyhazards.rst", "api/pyhazards.datasets.rst", "api/pyhazards.engine.rst", "api/pyhazards.metrics.rst", "api/pyhazards.models.rst", "api/pyhazards.utils.rst", "cite.rst", "datasets/era5.rst", "datasets/firms.rst", "datasets/goesr.rst", "datasets/landfire.rst", "datasets/merra2.rst", "datasets/mtbs.rst", "datasets/noaa_flood.rst", "datasets/wfigs.rst", "implementation.rst", "index.rst", "installation.rst", "pyhazards_datasets.rst", "pyhazards_engine.rst", "pyhazards_metrics.rst", "pyhazards_models.rst", "pyhazards_utils.rst", "quick_start.rst", "references.rst", "team.rst"], "indexentries": {}, "objects": {"": [[1, 0, 0, "-", "pyhazards"]], "pyhazards": [[1, 1, 1, "", "CNNPatchEncoder"], [1, 1, 1, "", "ClassificationHead"], [1, 1, 1, "", "ClassificationMetrics"], [1, 1, 1, "", "DataBundle"], [1, 1, 1, "", "DataSplit"], [1, 1, 1, "", "Dataset"], [1, 1, 1, "", "FeatureSpec"], [1, 1, 1, "", "LabelSpec"], [1, 1, 1, "", "MLPBackbone"], [1, 1, 1, "", "MetricBase"], [1, 1, 1, "", "RegressionHead"], [1, 1, 1, "", "RegressionMetrics"], [1, 1, 1, "", "SegmentationHead"], [1, 1, 1, "", "SegmentationMetrics"], [1, 1, 1, "", "TemporalEncoder"], [1, 1, 1, "", "Trainer"], [1, 4, 1, "", "available_datasets"], [1, 4, 1, "", "available_models"], [1, 4, 1, "", "build_model"], [2, 0, 0, "-", "datasets"], [3, 0, 0, "-", "engine"], [1, 4, 1, "", "load_dataset"], [4, 0, 0, "-", "metrics"], [5, 0, 0, "-", "models"], [1, 4, 1, "", "register_dataset"], [1, 4, 1, "", "register_model"], [6, 0, 0, "-", "utils"]], "pyhazards.CNNPatchEncoder": [[1, 2, 1, "", "forward"]], "pyhazards.ClassificationHead": [[1, 2, 1, "", "forward"]], "pyhazards.ClassificationMetrics": [[1, 3, 1, "", "_abc_impl"], [1, 2, 1, "", "compute"], [1, 2, 1, "", "reset"], [1, 2, 1, "", "update"]], "pyhazards.DataBundle": [[1, 3, 1, "", "feature_spec"], [1, 2, 1, "", "get_split"], [1, 3, 1, "", "label_spec"], [1, 3, 1, "", "metadata"], [1, 3, 1, "", "splits"]], "pyhazards.DataSplit": [[1, 3, 1, "", "inputs"], [1, 3, 1, "", "metadata"], [1, 3, 1, "", "targets"]], "pyhazards.Dataset": [[1, 2, 1, "", "_load"], [1, 2, 1, "", "load"], [1, 3, 1, "", "name"]], "pyhazards.FeatureSpec": [[1, 3, 1, "", "channels"], [1, 3, 1, "", "description"], [1, 3, 1, "", "extra"], [1, 3, 1, "", "input_dim"]], "pyhazards.LabelSpec": [[1, 3, 1, "", "description"], [1, 3, 1, "", "extra"], [1, 3, 1, "", "num_targets"], [1, 3, 1, "", "task_type"]], "pyhazards.MLPBackbone": [[1, 2, 1, "", "forward"]], "pyhazards.MetricBase": [[1, 3, 1, "", "_abc_impl"], [1, 2, 1, "", "compute"], [1, 2, 1, "", "reset"], [1, 2, 1, "", "update"]], "pyhazards.RegressionHead": [[1, 2, 1, "", "forward"]], "pyhazards.RegressionMetrics": [[1, 3, 1, "", "_abc_impl"], [1, 2, 1, "", "compute"], [1, 2, 1, "", "reset"], [1, 2, 1, "", "update"]], "pyhazards.SegmentationHead": [[1, 2, 1, "", "forward"]], "pyhazards.SegmentationMetrics": [[1, 3, 1, "", "_abc_impl"], [1, 2, 1, "", "compute"], [1, 2, 1, "", "reset"], [1, 2, 1, "", "update"]], "pyhazards.TemporalEncoder": [[1, 2, 1, "", "forward"]], "pyhazards.Trainer": [[1, 2, 1, "", "_make_loader"], [1, 2, 1, "", "_to_device"], [1, 2, 1, "", "evaluate"], [1, 2, 1, "", "fit"], [1, 2, 1, "", "predict"], [1, 2, 1, "", "save_checkpoint"]], "pyhazards.datasets": [[2, 1, 1, "", "DataBundle"], [2, 1, 1, "", "DataSplit"], [2, 1, 1, "", "Dataset"], [2, 1, 1, "", "FeatureSpec"], [2, 1, 1, "", "GraphTemporalDataset"], [2, 1, 1, "", "LabelSpec"], [2, 4, 1, "", "available_datasets"], [2, 0, 0, "-", "base"], [2, 4, 1, "", "graph_collate"], [2, 0, 0, "-", "hazards"], [2, 4, 1, "", "load_dataset"], [2, 4, 1, "", "register_dataset"], [2, 0, 0, "-", "registry"], [2, 0, 0, "-", "transforms"]], "pyhazards.datasets.DataBundle": [[2, 3, 1, "", "feature_spec"], [2, 2, 1, "", "get_split"], [2, 3, 1, "", "label_spec"], [2, 3, 1, "", "metadata"], [2, 3, 1, "", "splits"]], "pyhazards.datasets.DataSplit": [[2, 3, 1, "", "inputs"], [2, 3, 1, "", "metadata"], [2, 3, 1, "", "targets"]], "pyhazards.datasets.Dataset": [[2, 2, 1, "", "_load"], [2, 2, 1, "", "load"], [2, 3, 1, "", "name"]], "pyhazards.datasets.FeatureSpec": [[2, 3, 1, "", "channels"], [2, 3, 1, "", "description"], [2, 3, 1, "", "extra"], [2, 3, 1, "", "input_dim"]], "pyhazards.datasets.LabelSpec": [[2, 3, 1, "", "description"], [2, 3, 1, "", "extra"], [2, 3, 1, "", "num_targets"], [2, 3, 1, "", "task_type"]], "pyhazards.datasets.base": [[2, 1, 1, "", "DataBundle"], [2, 1, 1, "", "DataSplit"], [2, 1, 1, "", "Dataset"], [2, 1, 1, "", "FeatureSpec"], [2, 1, 1, "", "LabelSpec"], [2, 1, 1, "", "Transform"]], "pyhazards.datasets.base.DataBundle": [[2, 3, 1, "", "feature_spec"], [2, 2, 1, "", "get_split"], [2, 3, 1, "", "label_spec"], [2, 3, 1, "", "metadata"], [2, 3, 1, "", "splits"]], "pyhazards.datasets.base.DataSplit": [[2, 3, 1, "", "inputs"], [2, 3, 1, "", "metadata"], [2, 3, 1, "", "targets"]], "pyhazards.datasets.base.Dataset": [[2, 2, 1, "", "_load"], [2, 2, 1, "", "load"], [2, 3, 1, "", "name"]], "pyhazards.datasets.base.FeatureSpec": [[2, 3, 1, "", "channels"], [2, 3, 1, "", "description"], [2, 3, 1, "", "extra"], [2, 3, 1, "", "input_dim"]], "pyhazards.datasets.base.LabelSpec": [[2, 3, 1, "", "description"], [2, 3, 1, "", "extra"], [2, 3, 1, "", "num_targets"], [2, 3, 1, "", "task_type"]], "pyhazards.datasets.base.Transform": [[2, 3, 1, "", "_abc_impl"], [2, 3, 1, "", "_is_protocol"]], "pyhazards.datasets.registry": [[2, 4, 1, "", "available_datasets"], [2, 4, 1, "", "load_dataset"], [2, 4, 1, "", "register_dataset"]], "pyhazards.engine": [[3, 1, 1, "", "DistributedConfig"], [3, 1, 1, "", "SlidingWindowInference"], [3, 1, 1, "", "Trainer"], [3, 0, 0, "-", "distributed"], [3, 0, 0, "-", "inference"], [3, 4, 1, "", "select_strategy"], [3, 0, 0, "-", "trainer"]], "pyhazards.engine.DistributedConfig": [[3, 3, 1, "", "devices"], [3, 3, 1, "", "strategy"]], "pyhazards.engine.Trainer": [[3, 2, 1, "", "_make_loader"], [3, 2, 1, "", "_to_device"], [3, 2, 1, "", "evaluate"], [3, 2, 1, "", "fit"], [3, 2, 1, "", "predict"], [3, 2, 1, "", "save_checkpoint"]], "pyhazards.engine.distributed": [[3, 1, 1, "", "DistributedConfig"], [3, 4, 1, "", "select_strategy"]], "pyhazards.engine.distributed.DistributedConfig": [[3, 3, 1, "", "devices"], [3, 3, 1, "", "strategy"]], "pyhazards.engine.inference": [[3, 1, 1, "", "SlidingWindowInference"]], "pyhazards.engine.trainer": [[3, 1, 1, "", "Trainer"]], "pyhazards.engine.trainer.Trainer": [[3, 2, 1, "", "_make_loader"], [3, 2, 1, "", "_to_device"], [3, 2, 1, "", "evaluate"], [3, 2, 1, "", "fit"], [3, 2, 1, "", "predict"], [3, 2, 1, "", "save_checkpoint"]], "pyhazards.metrics": [[4, 1, 1, "", "ClassificationMetrics"], [4, 1, 1, "", "MetricBase"], [4, 1, 1, "", "RegressionMetrics"], [4, 1, 1, "", "SegmentationMetrics"]], "pyhazards.metrics.ClassificationMetrics": [[4, 3, 1, "", "_abc_impl"], [4, 2, 1, "", "compute"], [4, 2, 1, "", "reset"], [4, 2, 1, "", "update"]], "pyhazards.metrics.MetricBase": [[4, 3, 1, "", "_abc_impl"], [4, 2, 1, "", "compute"], [4, 2, 1, "", "reset"], [4, 2, 1, "", "update"]], "pyhazards.metrics.RegressionMetrics": [[4, 3, 1, "", "_abc_impl"], [4, 2, 1, "", "compute"], [4, 2, 1, "", "reset"], [4, 2, 1, "", "update"]], "pyhazards.metrics.SegmentationMetrics": [[4, 3, 1, "", "_abc_impl"], [4, 2, 1, "", "compute"], [4, 2, 1, "", "reset"], [4, 2, 1, "", "update"]], "pyhazards.models": [[5, 1, 1, "", "CNNPatchEncoder"], [5, 1, 1, "", "ClassificationHead"], [5, 1, 1, "", "MLPBackbone"], [5, 1, 1, "", "RegressionHead"], [5, 1, 1, "", "SegmentationHead"], [5, 1, 1, "", "TemporalEncoder"], [5, 1, 1, "", "TverskyLoss"], [5, 1, 1, "", "WildfireASPP"], [5, 1, 1, "", "WildfireCNNASPP"], [5, 1, 1, "", "WildfireMamba"], [5, 4, 1, "", "available_models"], [5, 0, 0, "-", "backbones"], [5, 4, 1, "", "build_model"], [5, 0, 0, "-", "builder"], [5, 4, 1, "", "cnn_aspp_builder"], [5, 0, 0, "-", "heads"], [5, 4, 1, "", "register_model"], [5, 0, 0, "-", "registry"], [5, 4, 1, "", "wildfire_aspp_builder"], [5, 4, 1, "", "wildfire_mamba_builder"]], "pyhazards.models.CNNPatchEncoder": [[5, 2, 1, "", "forward"]], "pyhazards.models.ClassificationHead": [[5, 2, 1, "", "forward"]], "pyhazards.models.MLPBackbone": [[5, 2, 1, "", "forward"]], "pyhazards.models.RegressionHead": [[5, 2, 1, "", "forward"]], "pyhazards.models.SegmentationHead": [[5, 2, 1, "", "forward"]], "pyhazards.models.TemporalEncoder": [[5, 2, 1, "", "forward"]], "pyhazards.models.TverskyLoss": [[5, 2, 1, "", "forward"]], "pyhazards.models.WildfireCNNASPP": [[5, 2, 1, "", "forward"]], "pyhazards.models.WildfireMamba": [[5, 2, 1, "", "_get_adjacency"], [5, 2, 1, "", "_temporal_delta"], [5, 2, 1, "", "forward"], [5, 2, 1, "", "set_adjacency"]], "pyhazards.models.backbones": [[5, 1, 1, "", "CNNPatchEncoder"], [5, 1, 1, "", "MLPBackbone"], [5, 1, 1, "", "TemporalEncoder"]], "pyhazards.models.backbones.CNNPatchEncoder": [[5, 2, 1, "", "forward"]], "pyhazards.models.backbones.MLPBackbone": [[5, 2, 1, "", "forward"]], "pyhazards.models.backbones.TemporalEncoder": [[5, 2, 1, "", "forward"]], "pyhazards.models.builder": [[5, 4, 1, "", "build_model"], [5, 4, 1, "", "default_builder"]], "pyhazards.models.heads": [[5, 1, 1, "", "ClassificationHead"], [5, 1, 1, "", "RegressionHead"], [5, 1, 1, "", "SegmentationHead"]], "pyhazards.models.heads.ClassificationHead": [[5, 2, 1, "", "forward"]], "pyhazards.models.heads.RegressionHead": [[5, 2, 1, "", "forward"]], "pyhazards.models.heads.SegmentationHead": [[5, 2, 1, "", "forward"]], "pyhazards.models.registry": [[5, 4, 1, "", "available_models"], [5, 4, 1, "", "get_model_config"], [5, 4, 1, "", "register_model"]], "pyhazards.utils": [[6, 4, 1, "", "auto_device"], [6, 0, 0, "-", "common"], [6, 4, 1, "", "get_device"], [6, 4, 1, "", "get_logger"], [6, 0, 0, "-", "hardware"], [6, 4, 1, "", "num_devices"], [6, 4, 1, "", "seed_all"], [6, 4, 1, "", "set_device"]], "pyhazards.utils.common": [[6, 4, 1, "", "get_logger"], [6, 4, 1, "", "seed_all"]], "pyhazards.utils.hardware": [[6, 4, 1, "", "auto_device"], [6, 4, 1, "", "get_device"], [6, 4, 1, "", "num_devices"], [6, 4, 1, "", "set_device"]]}, "objnames": {"0": ["py", "module", "Python module"], "1": ["py", "class", "Python class"], "2": ["py", "method", "Python method"], "3": ["py", "attribute", "Python attribute"], "4": ["py", "function", "Python function"]}, "objtypes": {"0": "py:module", "1": "py:class", "2": "py:method", "3": "py:attribute", "4": "py:function"}, "terms": {"": [10, 11, 13, 14, 15, 19, 26], "0": [1, 3, 5, 8, 10, 12, 16, 17, 18, 19, 22, 24], "00230": 10, "008": 9, "01": 19, "0301003": 13, "06": 5, "0758": 12, "08": 9, "1": [1, 3, 5, 10, 12, 13, 17, 22], "10": [8, 9, 10, 11, 12, 13, 16, 20], "100": [], "1000": [16, 19], "1002": 8, "1016": 9, "1071": 11, "1175": [10, 12], "12": [5, 18, 22], "128": [1, 5, 16, 22], "14": 12, "143": 9, "146": 8, "15": 10, "16": [10, 12, 16, 17, 19, 20, 22, 24], "17": 10, "18": [10, 11], "1999": 8, "1e": [5, 16, 17, 20, 22, 24], "2": [1, 5, 8, 10, 16, 17, 18, 19, 20, 22, 24], "2007": [13, 19], "2009": 11, "2013": 9, "2014": [9, 19], "2017": [10, 12, 19], "2020": [8, 19], "2024": 19, "2025": 17, "2049": 8, "21": 13, "235": 11, "24": 17, "249": 11, "25": 8, "256": [1, 5, 22], "3": [1, 5, 11, 13, 16, 17, 18, 20, 22, 24], "30": [11, 12, 13], "31": [], "32": [1, 3, 5, 17, 22], "350": [17, 24], "375": 9, "3803": 8, "4": [10, 17], "42": 6, "425": [17, 24], "48": 22, "4996": 13, "5": [5, 10, 12, 17, 22, 24], "500": [17, 24], "5419": 12, "5454": 12, "6": [5, 17, 18, 22], "60": [], "625": 12, "64": [1, 3, 5, 22], "681": 10, "698": 10, "730": 8, "8": [17, 18, 22], "800": [16, 19], "85": 9, "900": [16, 19], "96": 9, "98": 10, "A": [10, 11, 13, 17, 22], "For": [22, 24], "If": [7, 17], "It": [8, 12, 22], "Near": [8, 9, 12, 19], "Or": 24, "The": [8, 9, 10, 11, 12, 19, 20, 22, 26], "To": [16, 24], "_": [], "_abc": [1, 2, 4], "_abc_data": [1, 2, 4], "_abc_impl": [0, 1, 2, 4], "_get_adjac": [1, 5], "_is_protocol": [1, 2], "_load": [0, 1, 2, 16, 17, 19, 24], "_make_load": [0, 1, 3], "_temporal_delta": [1, 5], "_to_devic": [0, 1, 3], "abc": [1, 4], "abi": 10, "abov": [], "abstract": [1, 3, 4, 21], "accept": 16, "access": [], "accuraci": 21, "across": [11, 13, 14, 15, 19], "activ": [9, 15, 19], "adam": [16, 17, 20, 22, 24], "add": 16, "adj": 5, "adjac": [2, 5, 17, 22], "administr": 14, "adopt": 8, "advanc": 10, "aerosol": 10, "after": 16, "agenc": [14, 15], "aggreg": [15, 16, 21], "ai": 17, "al": [8, 10, 12, 19], "algorithm": 9, "allow": 19, "alpha": 5, "america": 10, "american": 10, "ametsoc": [], "amount": 8, "amp": [16, 20], "an": [2, 15, 22], "analysi": [8, 9, 10, 12, 13, 14, 15, 19], "analyz": 17, "ani": [1, 2, 3, 5, 22], "annual": [11, 13], "anomali": [9, 10], "api": [1, 3, 17, 19, 24], "append": [], "appli": [5, 22], "applic": [8, 12], "ar": [8, 9, 10, 11, 13, 14, 15, 20, 21, 22], "arcgi": 15, "architectur": [16, 17, 22], "arg": [2, 5], "argument": 16, "arrai": [], "aspp": 5, "aspp_channel": 5, "assess": [8, 9, 11, 13, 14, 17, 19], "assimil": [8, 12], "associ": [9, 15], "assum": [], "assur": 26, "atmospher": [8, 10, 12, 14, 19], "attribut": 8, "author": 17, "authorit": [15, 19], "auto": [1, 3, 20], "auto_devic": [1, 6, 20], "automat": [6, 24], "avail": [10, 13, 14, 20, 24], "available_dataset": [0, 1, 2], "available_model": [0, 1, 5, 22], "awar": [9, 10, 17], "b": [5, 8, 13], "back": 22, "backbon": [0, 1, 16, 17, 22], "background": 11, "backward": 5, "bam": 10, "band": 10, "base": [0, 1, 3, 4, 5, 9, 11, 13, 14, 15, 18, 19], "base_channel": 5, "baselin": [8, 10], "basic": 17, "batch": [2, 5, 22], "batch_siz": [1, 3, 5, 17, 22], "bcewithlogitsloss": [17, 22], "becaus": 22, "begin": 14, "behavior": [11, 19], "bell": 8, "below": 19, "benchmark": [8, 12, 19], "berrisford": 8, "beta": 5, "big": 10, "bin": 14, "binari": [5, 17, 22], "block": 22, "bool": [], "both": [], "boundari": 8, "box": 17, "branch": 22, "brewer": 13, "bright": 10, "broad": 12, "build": [1, 5, 17], "build_model": [0, 1, 5, 16, 17, 20, 22, 24], "build_path": [], "builder": [0, 1, 2, 16, 22], "built": [17, 19, 21], "bulk": 11, "bulletin": 10, "bundl": [1, 2, 17, 22], "burn": [13, 19], "burnsever": [], "c": 5, "c00648": 14, "c3": 8, "cache_dir": [1, 2], "calcul": 17, "call": 21, "callabl": 2, "can": [12, 16, 18, 22], "canopi": 11, "carri": 10, "categor": 13, "caus": 15, "cd": [8, 19], "center": [14, 15], "centr": 8, "chain": 16, "chang": [8, 13, 22], "channel": [0, 1, 2], "character": 11, "checkpoint": 20, "cheng": [17, 26], "choos": 6, "chunk": [], "class": [1, 2, 3, 4, 5, 13, 16, 17, 20, 24], "classif": [1, 5, 13, 16, 17, 19, 20, 22], "classificationhead": [0, 1, 5], "classificationmetr": [0, 1, 4, 16, 17, 20, 21, 24], "cli": 22, "clim": [], "climat": [8, 12], "closer": 10, "cloud": 10, "cnn": [1, 5, 16, 17, 22], "cnn_aspp_build": [1, 5], "cnnpatchencod": [0, 1, 5], "code": 26, "collat": [2, 17], "collate_fn": [1, 3, 17], "collect": 16, "com": [15, 17], "combin": [8, 9], "common": [0, 1, 21, 22, 23], "commonli": [15, 19], "commun": 26, "compat": 5, "compil": 14, "complex": [1, 3, 22], "compos": [22, 26], "comprehens": [8, 17], "comput": [0, 1, 2, 4, 16, 21], "concret": 2, "conda": 18, "condit": [], "confid": 9, "config": [20, 22], "consid": 7, "consist": [1, 5, 8, 11, 12], "construct": [1, 2, 19, 22], "contain": [1, 2, 15, 19], "content": [0, 19], "context": 19, "contextu": 10, "continu": [10, 12, 19], "contourf": [], "contribut": 26, "conv_kernel": 5, "conveni": 22, "convent": 12, "coordin": [8, 9, 10, 11, 12, 13, 14, 15], "copernicu": [8, 19], "core": 12, "correl": [17, 22], "correspond": 9, "count": [5, 22], "counti": [2, 5, 14, 17, 22], "coupl": 22, "covari": [8, 11, 12, 19], "cover": [11, 21], "coverag": [8, 9, 10, 11, 12, 13, 14, 15], "cpu": [16, 24], "cr": [], "creat": [16, 22], "critic": 10, "crop": 14, "crossentropyloss": [16, 17, 20, 24], "csiszar": 9, "csv": [9, 14], "cu126": 18, "cube": 22, "cuda": [18, 20, 24], "current": 2, "custom": [1, 3, 16, 17, 24], "d": [10, 12], "d2m": 22, "da": [], "dai": [2, 5, 9, 17, 22], "daili": 19, "damag": 14, "danger": 8, "dask": [], "data": [1, 2, 3, 17, 19, 22, 24], "data_bundl": [16, 20], "data_var": [], "dataandtool": [], "dataarrai": [], "databas": [14, 19], "databundl": [0, 1, 2, 16, 17, 19, 22, 24], "datafram": [], "dataload": [1, 3], "dataset": [0, 1, 9, 11, 12, 13, 14, 15, 17, 22, 24], "datasplit": [0, 1, 2, 16, 17, 19, 22, 24], "date": [15, 19], "date_end": [], "date_start": [], "datetim": [], "ddp": [3, 16, 17, 20], "dedic": 26, "def": [16, 17, 19, 22, 24], "default": [1, 5, 16, 20, 22], "default_build": [1, 5, 22], "deleg": [1, 5], "demonstr": 19, "densiti": 11, "depend": [9, 10, 11, 13, 15], "deploi": 17, "depth": [1, 5, 22], "deriv": [9, 10, 11, 13, 14, 19], "describ": [1, 2, 19], "descript": [0, 1, 2, 9, 14, 16, 17, 19, 24], "descriptor": 11, "design": [17, 21], "detail": 24, "detect": [9, 10, 15, 19, 24], "determinist": 6, "develop": 8, "devic": [1, 3, 6, 17, 23, 24], "device_str": 6, "dewpoint": 8, "df": [], "diagnost": [8, 10, 12], "dice": 21, "dict": [1, 2, 3, 4, 5, 22], "differ": [], "differenti": 22, "dilat": 5, "dimens": [], "direct": 26, "directli": 21, "directori": [], "discov": 19, "discover": 22, "discoveri": 15, "displai": [], "distanc": [17, 22], "distribut": [0, 1, 8, 9, 10, 15, 16], "distributedconfig": [1, 3], "disturb": 11, "dnbr": 13, "document": [14, 15, 24, 26], "doi": [8, 9, 10, 11, 12, 13], "dong": 26, "download": 18, "downstream": [1, 2], "dp": 3, "driven": [16, 22], "driver": [12, 14, 19], "drop": [], "dropout": 5, "drought": 12, "ds_pre": [], "ds_sfc": [], "dtype": [1, 2], "e": [9, 11, 12, 13, 15, 16, 19, 22], "each": [2, 9, 15, 19], "earli": [9, 10, 19], "earthdata": [9, 12], "earthquak": 2, "easi": [1, 2, 17], "ecmwf": [8, 19], "ecolog": [11, 13], "ecologi": 13, "ecosystem": 19, "edu": 26, "eidenshink": [13, 19], "els": [], "emerg": 14, "enabl": [5, 20], "encod": [1, 5, 16, 17, 22], "end": [11, 14, 21], "end_dat": [], "energi": 12, "engag": 26, "engin": [0, 1, 16, 17, 22, 24], "environ": [9, 18, 24], "environment": [8, 10, 12, 14, 17], "eosdi": [], "equal": [], "era": 12, "era5": [5, 17, 19, 22], "estim": [8, 12], "et": [8, 10, 12, 19], "etc": 2, "eu": [], "european": 8, "evalu": [0, 1, 3, 13, 16, 17, 20, 22, 24], "event": [9, 13, 15, 19], "evolut": 10, "exampl": [16, 17, 18], "exist": [], "explicitli": 24, "explor": 19, "export": [22, 24], "extend": [1, 3, 21], "extens": [17, 22], "extent": [13, 19], "extern": 5, "extra": [0, 1, 2, 17, 22], "extract": 14, "extractor": 22, "extrem": [8, 12], "ey": [17, 22], "f": [], "factori": [1, 2], "fake": [17, 22], "fall": 22, "fals": [1, 3, 5, 6, 17, 22], "familiar": [1, 3], "fatal": 14, "featur": [1, 2, 5, 16, 17, 19, 22, 24], "feature_spec": [0, 1, 2, 16, 17, 19, 22, 24], "featurespec": [0, 1, 2, 16, 17, 19, 22, 24], "field": [8, 10, 11, 12], "fifth": 8, "file": [13, 19], "filenotfounderror": [], "find": 7, "fire": [9, 10, 11, 13, 15, 17, 19, 22], "fireecologi": 13, "firelab": [], "firm": [15, 19], "first": [16, 17, 18, 24], "fit": [0, 1, 3, 16, 17, 20, 22, 24], "flag": [6, 9], "float": [1, 3, 4, 5, 17, 22], "flood": [2, 8, 19], "florida": 26, "flux": [8, 12], "follow": 7, "forc": [8, 12], "forecast": 8, "forest": 11, "format": [8, 9, 10, 11, 12, 13, 14, 15], "forward": [0, 1, 5], "found": [], "framework": 17, "frequenc": [10, 14, 19], "frequent": [], "from": [9, 10, 11, 13, 14, 16, 17, 19, 20, 21, 22, 24], "from_logit": 5, "frp": 9, "fsu": 26, "fuel": [9, 11, 13, 15, 19], "full": 21, "function": [2, 16, 17, 22, 24], "g": [9, 11, 12, 13, 15, 16, 19, 22], "gca": [], "gcn": 22, "gcn_hidden": 5, "gelaro": [12, 19], "gener": [5, 8, 10], "geodatabas": 13, "geograph": [9, 15], "geojson": [9, 15], "geolog": 13, "geometri": 15, "geopotenti": [8, 12], "geospati": [11, 15], "geostationari": [10, 19], "geotiff": [11, 13], "get": [16, 22, 24], "get_devic": [1, 6], "get_logg": [1, 6], "get_model_config": [1, 5], "get_split": [0, 1, 2], "gi": 15, "giglio": 9, "github": 17, "given": 22, "global": [8, 9, 12, 19], "gmao": [12, 19], "goe": [15, 19], "goesr": 19, "gov": 14, "gpu": [16, 17, 20], "graph": 17, "graph_col": [1, 2, 17, 22], "graphtemporaldataset": [1, 2, 17, 22], "grib": 8, "grid": [3, 8, 9, 10, 11, 12, 14, 19, 20], "griffith": 10, "ground": [10, 15, 19], "growth": 10, "gru": [1, 5], "gsfc": [], "guid": 24, "gunshor": 10, "h": [5, 8], "h5netcdf": [], "handl": [20, 22], "hardwar": [0, 1, 20, 23], "hazard": [0, 1, 8, 9, 10, 11, 12, 14, 16, 17, 19, 24], "head": [0, 1, 16, 17, 22], "heat": 12, "height": [11, 12], "help": 24, "helper": [17, 20, 22, 23, 24], "hemispher": 10, "hersbach": [8, 19], "hidden": [16, 22], "hidden_dim": [1, 5, 16, 22], "high": [8, 10, 19], "highlight": 22, "histor": [8, 14, 15], "hold": 19, "home": [], "hotspot": [9, 15], "hourli": [8, 12, 19], "how": [16, 19], "howard": 13, "http": [8, 9, 10, 11, 12, 13, 14, 15, 17, 18], "humid": 12, "hurrican": [2, 12], "i": [2, 5, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 20, 22, 26], "id": 14, "ident": 22, "identifi": 15, "ignit": [10, 11, 13, 15, 19], "imag": 10, "imageri": [10, 13, 19], "impact": [13, 14, 19], "implement": [2, 3, 17, 19, 24], "import": [16, 17, 19, 20, 21, 22, 24], "in_channel": [1, 5], "in_dim": [1, 5, 16, 17, 20, 22, 24], "incid": [15, 19], "includ": [8, 9, 10, 11, 12, 13, 14, 15], "index": [2, 16, 17, 18], "indic": [9, 10], "individu": 13, "infer": [0, 1, 20], "inform": [9, 14, 15], "infrar": 10, "initi": 9, "injuri": 14, "input": [0, 1, 2, 3, 5, 8, 12, 19, 22], "input_dim": [0, 1, 2, 5, 16, 17, 19, 22, 24], "ins": 22, "inspect_d": [], "int": [1, 2, 3, 6, 16, 22], "integr": [9, 10, 11, 13, 14, 15], "interag": [15, 19], "interfac": [1, 5, 17, 19, 21, 24], "intern": [11, 22], "iou": 21, "ipython": [], "isel": [], "iso": 14, "item": [], "iter": [1, 3], "j": [9, 10, 12, 13], "jcli": 12, "journal": [8, 11, 12], "k": 13, "keep": [1, 2, 5, 22], "keyerror": [], "kind": [], "km": 10, "kml": 9, "kwarg": [1, 2, 5, 16, 22], "l": 9, "label": [1, 2, 9, 14, 15, 17, 19, 22], "label_spec": [0, 1, 2, 16, 17, 19, 22, 24], "labelspec": [0, 1, 2, 16, 17, 19, 22, 24], "labrai": 17, "land": [8, 9, 12, 14], "landfir": 19, "landsat": [13, 19], "landscap": [11, 19], "landslid": 2, "larg": [3, 20], "latest": 10, "latitud": [8, 9, 10, 12], "layer": [8, 11, 13, 16, 19, 22], "learn": [8, 11, 12, 14, 17], "len": [], "let": [], "level": [8, 9, 12, 14, 15, 19], "librari": 17, "lightweight": [1, 3, 5, 19, 22], "like": [17, 22], "limit": 12, "line": 16, "linear": [16, 22], "list": [1, 3], "list_var": [], "liter": 3, "load": [0, 1, 2, 16, 17, 24], "load_dataset": [0, 1, 2, 19], "loader": 2, "local": 14, "locat": [9, 14, 15, 19], "log": [17, 23], "logger": 6, "logic": 3, "logit": [5, 17, 22], "long": [12, 13, 19], "longitud": [8, 9, 10, 12], "look": 10, "loop": [1, 3, 22], "loss": 5, "loss_fn": [1, 3, 16, 17, 20, 22, 24], "lower": [], "lr": [16, 17, 20, 22, 24], "m": [8, 9, 10, 11, 12, 13, 19], "machin": [8, 11, 12, 17], "mae": 21, "maintain": 15, "mainten": 26, "make": [1, 2], "mamba": 5, "mamba_lay": 5, "manag": [9, 11, 14, 15, 17, 23], "map": [11, 13], "mask": [1, 5], "match": 18, "matplotlib": [], "matrix": [17, 22], "max": [], "max_epoch": [1, 3, 16, 17, 20, 22, 24], "max_show": [], "max_var": [], "mccarti": 12, "mean": [], "medium": 8, "merra": 19, "merra2": 19, "merra2_sfc_": [], "merra_pres_": [], "metadata": [0, 1, 2, 5, 13, 14, 15, 19], "meteorolog": [8, 9, 10, 11, 12, 14, 15, 19], "meteorologi": [8, 12, 13, 19], "metric": [0, 1, 3, 13, 17, 20, 24], "metricbas": [0, 1, 4, 16, 21], "min": [], "minim": [1, 3, 22], "minut": 10, "miss": [], "mix": [17, 22], "mixed_precis": [1, 3, 16, 17, 20, 22, 24], "mlp": [1, 5, 16, 17, 20, 22, 24], "mlpbackbon": [0, 1, 5], "modap": [], "mode": 10, "model": [0, 1, 2, 3, 8, 9, 11, 12, 13, 14, 15, 17, 19, 20, 24], "modern": [8, 12], "modi": [9, 19], "modul": [0, 16, 17, 23], "modular": [16, 17], "moistur": 12, "monitor": [9, 10, 13, 19], "monthli": 14, "more": [22, 24], "motion": 10, "mtb": 19, "multi": [1, 5, 12, 16, 17], "multipl": [9, 10, 11, 20], "multispectr": [10, 19], "must": [], "my_custom_build": 22, "my_hazard": [16, 19], "my_mlp": [16, 22], "my_model_build": 16, "myhazard": 16, "myhazarddataset": 19, "n": 5, "n_var": [], "name": [0, 1, 2, 5, 6, 13, 15, 16, 17, 19, 20, 22, 24], "namespac": 2, "narr": 14, "nasa": [9, 12, 19], "nation": [11, 13, 14, 15], "nationwid": [11, 19], "natur": [8, 12, 17], "nc": [], "ncdc": 14, "ncei": 14, "need": [3, 21, 22], "neighbor": 22, "netcdf": [8, 10, 12], "netcdf4": [], "new": 9, "next": [5, 17, 22], "nifc": 15, "nn": [16, 17, 20, 22, 24], "no_grad": [17, 22], "noaa": [10, 19], "noaa_flood": 19, "none": [1, 2, 3, 4, 5, 6, 17, 22], "normal": [1, 2, 16], "note": 18, "np": [], "num_class": [1, 4, 5], "num_counti": [2, 5, 17, 22], "num_devic": [1, 6], "num_featur": [2, 5, 17, 22], "num_lay": [1, 5], "num_target": [0, 1, 2, 16, 17, 19, 22, 24], "num_work": [1, 3], "numer": 12, "numpi": [], "o": [], "obj": [1, 3], "object": [1, 2, 3, 4, 19], "observ": [8, 9, 10, 11, 12], "occurr": [9, 14, 15, 19], "ocean": 14, "offic": [12, 14], "offici": 15, "often": 19, "oliva": 9, "one": 17, "ongo": 15, "onlinelibrari": [], "open": [], "open_dataset": [], "open_merra": [], "opendata": 15, "oper": [9, 10, 19], "optim": [1, 3, 16, 17, 20, 22, 24], "option": [1, 2, 5, 8, 16, 17, 20, 22], "org": [8, 9, 10, 11, 12, 13, 18], "other": 23, "otherwis": 20, "our": 26, "out": 17, "out_dim": [1, 5, 16, 17, 20, 22, 24], "outdir": 19, "output": [1, 5, 19, 22], "over": [3, 10, 21, 22], "overpass": 9, "overrid": 5, "overview": 19, "own": [16, 22], "p": [8, 9, 10], "packag": 0, "page": [14, 17], "pair": [14, 19], "panda": [], "paramet": [5, 16, 17, 20, 22, 24], "particularli": [10, 14], "pass": [16, 21, 22], "past_dai": [2, 5, 17, 22], "patch": [1, 5, 16, 21, 22], "path": [1, 3], "pathlib": [], "pattern": [9, 15], "pattern_pr": [], "pattern_sfc": [], "pd": [], "per": [5, 9, 22], "perimet": [13, 15, 19], "physic": 12, "pick": [], "pip": 18, "pipelin": [8, 9, 11, 12, 15, 17], "pixel": 21, "placehold": [2, 3, 20], "plain": 22, "plan": [11, 26], "pleas": [7, 17, 24], "plt": [], "plu": [1, 2, 19, 22], "point": [9, 10, 14, 15], "polygon": [13, 15], "popul": 2, "portal": [9, 11, 13, 15], "post": [13, 19], "potenti": 9, "power": [9, 17], "pre": [], "precipit": [8, 12], "precis": 17, "pred": [1, 4, 20], "predict": [0, 1, 3, 8, 9, 11, 12, 13, 14, 16, 17, 19, 20, 21, 22], "prefer": [3, 6], "preprocess": [2, 16], "pressur": [8, 12, 19], "print": [17, 22, 24], "prithvi": [], "prob": [17, 22], "probabl": [5, 17, 22], "process": 10, "produc": [8, 11, 12, 13], "product": [9, 10, 11, 13, 15], "profil": 12, "program": [10, 11, 13, 19], "project": [10, 11, 13, 15], "properti": 14, "protocol": 2, "provid": [1, 2, 8, 9, 10, 11, 12, 14, 15, 16, 17, 19, 22], "publicli": [9, 11, 13, 14, 15], "pyhazard": [16, 17, 19, 20, 21, 22, 23, 24, 26], "pyhazards2025": 17, "pyhazards_devic": [6, 24], "pyplot": [], "python": [17, 18, 19, 24], "pytorch": [17, 18, 22], "qj": 8, "qualiti": [9, 26], "quarterli": 8, "quasi": 11, "quayl": 13, "quick": 17, "quickli": [19, 24], "r": [12, 19], "radi": 9, "radianc": 10, "radiat": 8, "rais": [], "randint": [16, 17, 19, 22, 24], "randn": [16, 17, 19, 22, 24], "random": 22, "rang": 8, "rapid": [9, 10], "raster": [1, 3, 5, 10, 11, 13, 17, 19, 20, 22], "rather": 15, "rdnbr": 13, "re": 22, "readi": [1, 2, 17], "real": [9, 10, 15, 19], "reanalysi": [8, 10, 12, 14, 19], "receiv": 22, "recommend": 18, "record": [9, 10, 14, 15, 19], "refer": 24, "refresh": 10, "regim": [11, 13, 19], "region": 14, "regist": [2, 16, 17], "register_dataset": [0, 1, 2, 16, 19], "register_model": [0, 1, 5, 16, 22], "registr": [22, 24], "registri": [0, 1, 16, 17, 19, 22], "regress": [1, 2, 5, 16, 17, 22], "regressionhead": [0, 1, 5], "regressionmetr": [0, 1, 4, 21], "regular": [8, 12], "relat": [8, 10, 14, 19], "releas": 11, "relu": [16, 22], "remot": [9, 11, 13], "replac": [1, 3, 17, 22], "report": [14, 15, 19], "repres": 15, "research": [12, 17, 26], "reset": [0, 1, 4, 16, 21], "reset_index": [], "resolut": [8, 9, 10, 11, 12, 13, 14, 15], "resourc": [9, 11], "respect": 6, "respons": 26, "result": [16, 17, 20, 24], "retrospect": 12, "return": [1, 2, 3, 4, 5, 6, 16, 17, 19, 22, 24], "reusabl": [2, 16, 22], "review": 26, "risk": [8, 11, 17, 19], "rmet": [], "rmse": 21, "rollin": 11, "root": [], "row": [], "royal": 8, "rse": 9, "run": 20, "runyang": [], "sampl": [2, 17, 22], "satellit": [9, 10, 12, 15], "save": 20, "save_checkpoint": [0, 1, 3], "scalar": [1, 5], "scale": [11, 13], "schmit": 10, "schroeder": [9, 19], "schwind": 13, "search": 17, "see": 19, "seed": [6, 17, 23], "seed_al": [1, 6], "segment": [1, 5, 16, 17, 22], "segmentationhead": [0, 1, 5], "segmentationmetr": [0, 1, 4, 21], "select": [20, 22], "select_strategi": [1, 3], "self": [16, 17, 19, 24], "sens": [9, 11, 13], "sensibl": 20, "sensor": 9, "sequenti": [16, 22], "seri": [1, 5, 10, 19, 22], "serv": 19, "servic": [8, 9, 11, 12, 15], "set": [5, 8, 12, 24], "set_adjac": [1, 5], "set_aspect": [], "set_devic": [1, 6, 24], "sever": [13, 14, 19], "sfc": [], "shape": [1, 2, 5, 17, 22], "shapefil": [9, 13, 15], "ship": 22, "short": [], "should": [1, 2], "show": 16, "shuffl": [1, 3], "sigmoid": [5, 17, 22], "signal": [1, 5], "simpl": [1, 2, 5, 16, 17], "simplifi": 19, "simul": 11, "singl": [1, 2, 8, 19, 20], "situat": [9, 10], "size": 15, "skin": 12, "skipna": [], "slide": [3, 20], "slidingwindowinfer": [1, 3], "slowli": 11, "smoke": [10, 19], "smooth": 5, "so": [16, 22], "societi": [8, 10], "softwar": 17, "soil": 12, "sort_valu": [], "sourc": [1, 2, 3, 4, 5, 6, 15], "space": 22, "spatial": [5, 8, 9, 10, 11, 12, 13, 14, 15, 22], "spatio": 5, "spatiotempor": [8, 9, 12], "spec": [1, 2, 17, 19], "specif": [1, 2, 9, 22], "spectral": [10, 13], "split": [0, 1, 2, 3, 16, 17, 19, 20, 21, 22, 24], "spread": 11, "ssm": 22, "ssr": 22, "stack": [2, 22], "stamp": 9, "standard": [5, 8, 13], "start": [14, 17], "start_dat": [], "state": [8, 11, 12, 13, 14, 15, 22, 26], "state_dim": 5, "static": [5, 11, 19], "statist": [], "statu": [15, 19], "std": [], "step": [], "stitch": 3, "store": [8, 22], "storm": [14, 19], "str": [1, 2, 3, 4, 5, 16, 22], "strateg": 26, "strategi": [1, 3, 20], "strftime": [], "structur": [8, 9, 10, 11, 12, 13, 14, 15, 19], "studi": [8, 12, 13, 14, 15, 19], "style": [2, 5, 12, 17, 22], "subclass": [1, 2, 16], "submodul": [0, 1], "subpackag": 0, "summarize_da": [], "supervis": [9, 14], "support": [11, 13, 16, 17, 19, 20, 26], "surfac": [8, 9, 10, 12], "survei": 13, "su\u00e1rez": 12, "switch": 22, "system": [8, 9, 10, 11, 12, 13, 14, 15], "t": 10, "t2m": 22, "tab": [], "tabl": [], "tabular": [1, 5, 14, 17, 19, 22], "target": [0, 1, 2, 3, 4, 5, 9, 14, 16, 19, 21, 22], "task": [1, 2, 5, 8, 16, 17, 20, 21, 22, 24], "task_typ": [0, 1, 2, 16, 17, 19, 22, 24], "technic": 26, "temperatur": [8, 10, 12], "tempor": [2, 5, 8, 9, 10, 11, 12, 13, 14, 15, 16, 19], "temporalencod": [0, 1, 5], "tensor": [1, 2, 3, 5, 9, 14, 17, 22], "term": [12, 13, 19], "test": [1, 2, 3, 16, 17, 19, 20, 24], "test_dai": [], "than": 15, "them": [2, 16, 22], "themat": 11, "thermal": [9, 10, 19], "thi": [1, 5, 14, 16, 22, 24], "through": [9, 10, 13, 15, 19], "throughout": [], "time": [1, 5, 9, 10, 12, 14, 15, 19, 22], "timestamp": 14, "titl": 17, "toi": 22, "too": [], "tool": 11, "topographi": 13, "torch": [16, 17, 18, 19, 20, 22, 24], "toyhazard": [17, 24], "tp": 22, "train": [1, 2, 3, 13, 17, 19, 20, 22, 24], "train_d": 17, "train_split": [1, 3], "trainer": [0, 1, 16, 17, 20, 21, 22, 24], "transform": [0, 1, 17], "treat": 15, "trend": [8, 12, 13], "true": [1, 2, 3, 5, 16, 17, 20, 22, 24], "truth": [10, 15, 19], "tupl": [], "tverski": 5, "tverskyloss": [1, 5], "two": [16, 22], "type": [1, 2, 3, 4, 5, 6, 11, 14, 22], "u": [11, 13, 14, 15, 19], "u10": 22, "under": 8, "unifi": [17, 19], "uniqu": 15, "unit": [11, 13, 14, 15], "univers": 26, "updat": [0, 1, 4, 9, 11, 14, 15, 16, 21], "url": [17, 18], "us": [5, 7, 16, 17, 18, 19, 20, 22, 24], "usag": [17, 22], "usda": [], "user": [16, 19], "usf": 19, "usg": 13, "util": [0, 1, 17, 19, 20, 24], "v10": 22, "val": [1, 2, 16, 17, 19, 22, 24], "val_d": 17, "val_split": [1, 3], "valid": [13, 15], "valu": [], "valueerror": [], "var": [], "vari": 11, "variabl": [19, 24], "vast": 8, "vector": 13, "veget": [11, 19], "version": [11, 12], "vertic": [8, 12], "via": [8, 11, 12, 14, 16, 17, 19, 20, 22], "view": 10, "viir": [9, 19], "visibl": 10, "visual": [], "w": [5, 9, 12], "we": 18, "weather": [8, 10, 12, 14], "western": 10, "wf08088": 11, "wfig": 19, "wheel": 18, "when": [9, 14, 19, 20], "where": [10, 14], "which": [10, 14], "whl": 18, "who": 26, "wide": [8, 9, 10, 11, 12, 13, 14, 15, 19], "wildfir": [2, 5, 8, 9, 10, 11, 12, 13, 15, 19], "wildfire_aspp_build": [1, 5], "wildfire_mamba": [17, 22], "wildfire_mamba_build": [1, 5], "wildfireaspp": [1, 5], "wildfirecnnaspp": [1, 5], "wildfiremamba": [1, 5], "wildland": [11, 15], "wilei": [], "wind": [8, 12], "window": [2, 3, 16, 17, 20], "window_fn": 3, "with_count_head": [5, 22], "work": [1, 3, 7, 22], "workflow": [12, 19], "wrap": [20, 22], "www": 14, "wxc": 12, "x": [1, 2, 5, 16, 17, 19, 22, 24], "xarrai": [], "xc25": 26, "xml": [], "xr": [], "xueqi": [17, 26], "y": [2, 16, 17, 19, 22, 24], "year": [13, 17], "you": [7, 17, 22, 24], "your": [16, 17, 22], "yushun": 26, "yyyymmdd": [], "z": 13, "zhu": 13, "zone": 14}, "titles": ["pyhazards", "pyhazards package", "pyhazards.datasets package", "pyhazards.engine package", "pyhazards.metrics package", "pyhazards.models package", "pyhazards.utils package", "How to Cite", "ERA5", "FIRMS", "GOES-R", "LANDFIRE", "MERRA-2", "MTBS", "NOAA Flood Events", "WFIGS", "Implementation Guide", "Wildfire Mamba (spatio-temporal toy)", "Installation", "Datasets", "Engine", "Metrics", "Models", "Utils", "Quick Start", "References", "Core Team"], "titleterms": {"1": [], "2": 12, "3": [], "4": [], "5": [], "6": [], "academ": 25, "access": [8, 9, 10, 11, 12, 13, 14, 15], "backbon": 5, "base": [2, 22], "basic": 24, "build": 22, "builder": 5, "built": 22, "case": [8, 9, 10, 11, 12, 13, 14, 15], "characterist": [8, 9, 10, 11, 12, 13, 14, 15], "cite": [7, 17], "class": [19, 21], "classif": 24, "common": 6, "compon": 17, "config": [], "content": [1, 2, 3, 5, 6], "contributor": 26, "coord": [], "core": [17, 19, 20, 21, 22, 26], "custom": 22, "data": [8, 9, 10, 11, 12, 13, 14, 15], "dataset": [2, 16, 19], "date": [], "design": 22, "develop": 26, "devic": 20, "dim": [], "distribut": [3, 20], "engin": [3, 20], "era5": 8, "event": 14, "exampl": [19, 24], "filenam": [], "firm": 9, "flood": 14, "goe": 10, "gpu": 24, "guid": 16, "hardwar": 6, "hazard": 2, "head": 5, "helper": [], "how": [7, 17], "implement": 16, "import": [], "indic": 17, "infer": 3, "inspect": 19, "instal": 18, "landfir": 11, "lat": [], "lead": 26, "level": [], "load": [], "lon": [], "maintain": 26, "mamba": [17, 22], "map": [], "merra": 12, "metric": [4, 16, 21], "model": [5, 16, 22], "modul": [1, 2, 3, 5, 6, 20, 22], "mtb": 13, "next": 24, "noaa": 14, "note": 22, "overview": [8, 9, 10, 11, 12, 13, 14, 15], "packag": [1, 2, 3, 4, 5, 6], "path": [], "pattern": [], "plot": [], "princip": 26, "public": 25, "pyhazard": [0, 1, 2, 3, 4, 5, 6, 18], "quick": 24, "r": 10, "refer": [8, 9, 10, 11, 12, 13, 14, 15, 25], "regist": 22, "registri": [2, 5], "requir": 18, "setup": [], "skeleton": 19, "spatio": [17, 22], "start": 24, "stat": [], "step": 24, "submodul": [2, 3, 5, 6, 23], "subpackag": 1, "summari": [19, 20, 21, 22, 23], "support": 24, "tabl": 17, "tabular": 24, "team": 26, "tempor": [17, 22], "toi": [17, 24], "train": 16, "trainer": 3, "transform": [2, 16], "typic": [8, 9, 10, 11, 12, 13, 14, 15, 20], "us": [8, 9, 10, 11, 12, 13, 14, 15], "usag": [20, 21, 24], "util": [6, 23], "variabl": [8, 9, 10, 11, 12, 13, 14, 15], "wfig": 15, "wildfir": [17, 22]}}) \ No newline at end of file +Search.setIndex({"alltitles": {"Academic Publications": [[25, "academic-publications"]], "Access": [[8, "access"], [9, "access"], [10, "access"], [11, "access"], [12, "access"], [13, "access"], [14, "access"], [15, "access"]], "At-a-glance (Quick Facts)": [[12, null]], "Basic Usage": [[24, "basic-usage"]], "Build a built-in model": [[22, "build-a-built-in-model"]], "Contributors": [[26, "contributors"]], "Core Components": [[17, "core-components"]], "Core Contributors": [[26, "core-contributors"]], "Core Team": [[26, null]], "Core classes": [[19, "core-classes"], [21, "core-classes"]], "Core modules": [[20, "core-modules"], [22, "core-modules"]], "Data Characteristics": [[8, "data-characteristics"], [9, "data-characteristics"], [10, "data-characteristics"], [11, "data-characteristics"], [12, "data-characteristics"], [13, "data-characteristics"], [14, "data-characteristics"], [15, "data-characteristics"]], "Dataset inspection": [[19, "dataset-inspection"]], "Datasets": [[16, "datasets"], [19, null], [19, "id1"]], "Design notes": [[22, "design-notes"]], "Distributed and devices": [[20, "distributed-and-devices"]], "ERA5": [[8, null]], "Engine": [[20, null]], "Example skeleton": [[19, "example-skeleton"]], "FIRMS": [[9, null]], "GOES-R": [[10, null]], "GPU Support": [[24, "gpu-support"]], "How to Cite": [[7, null], [17, "how-to-cite"]], "Implementation Guide": [[16, null]], "Indices and tables": [[17, "indices-and-tables"]], "Installation": [[18, null]], "Installing PyHazards": [[18, "installing-pyhazards"]], "LANDFIRE": [[11, null]], "Lead Developer": [[26, "lead-developer"]], "MERRA-2": [[12, null]], "MTBS": [[13, null]], "Mamba-based wildfire model (spatio-temporal)": [[22, "mamba-based-wildfire-model-spatio-temporal"]], "Metrics": [[16, "metrics"], [21, null]], "Models": [[16, "models"], [22, null]], "Module contents": [[1, "module-pyhazards"], [2, "module-pyhazards.datasets"], [3, "module-pyhazards.engine"], [5, "module-pyhazards.models"], [6, "module-pyhazards.utils"]], "NOAA Flood Events": [[14, null]], "Next Steps": [[24, "next-steps"]], "Overview": [[8, "overview"], [9, "overview"], [10, "overview"], [11, "overview"], [12, "overview"], [13, "overview"], [14, "overview"], [15, "overview"]], "Principal Contributors & Maintainers": [[26, "principal-contributors-maintainers"]], "Quick Start": [[24, null]], "Reference": [[8, "reference"], [9, "reference"], [10, "reference"], [11, "reference"], [12, "reference"], [13, "reference"], [14, "reference"], [15, "reference"]], "References": [[25, null]], "Register a custom model": [[22, "register-a-custom-model"]], "Requirements": [[18, "requirements"]], "Stats": [[12, "stats"]], "Submodules": [[2, "submodules"], [3, "submodules"], [5, "submodules"], [6, "submodules"], [23, "submodules"]], "Subpackages": [[1, "subpackages"]], "Summary": [[19, "summary"], [20, "summary"], [21, "summary"], [22, "summary"], [23, "summary"]], "Toy Example (tabular classification)": [[24, "toy-example-tabular-classification"]], "Training": [[16, "training"]], "Transforms": [[16, "transforms"]], "Typical Use Cases": [[8, "typical-use-cases"], [9, "typical-use-cases"], [10, "typical-use-cases"], [11, "typical-use-cases"], [12, "typical-use-cases"], [13, "typical-use-cases"], [14, "typical-use-cases"], [15, "typical-use-cases"]], "Typical usage": [[20, "typical-usage"]], "Usage": [[21, "usage"]], "Utils": [[23, null]], "Variables": [[8, "variables"], [9, "variables"], [10, "variables"], [11, "variables"], [12, "variables"], [13, "variables"], [14, "variables"], [15, "variables"]], "WFIGS": [[15, null]], "Wildfire Mamba (spatio-temporal toy)": [[17, null]], "pyhazards": [[0, null]], "pyhazards package": [[1, null]], "pyhazards.datasets package": [[2, null]], "pyhazards.datasets.base module": [[2, "module-pyhazards.datasets.base"]], "pyhazards.datasets.hazards package": [[2, "module-pyhazards.datasets.hazards"]], "pyhazards.datasets.registry module": [[2, "module-pyhazards.datasets.registry"]], "pyhazards.datasets.transforms package": [[2, "module-pyhazards.datasets.transforms"]], "pyhazards.engine package": [[3, null]], "pyhazards.engine.distributed module": [[3, "module-pyhazards.engine.distributed"]], "pyhazards.engine.inference module": [[3, "module-pyhazards.engine.inference"]], "pyhazards.engine.trainer module": [[3, "module-pyhazards.engine.trainer"]], "pyhazards.metrics package": [[4, null]], "pyhazards.models package": [[5, null]], "pyhazards.models.backbones module": [[5, "module-pyhazards.models.backbones"]], "pyhazards.models.builder module": [[5, "module-pyhazards.models.builder"]], "pyhazards.models.heads module": [[5, "module-pyhazards.models.heads"]], "pyhazards.models.registry module": [[5, "module-pyhazards.models.registry"]], "pyhazards.utils package": [[6, null]], "pyhazards.utils.common module": [[6, "module-pyhazards.utils.common"]], "pyhazards.utils.hardware module": [[6, "module-pyhazards.utils.hardware"]]}, "docnames": ["api/modules", "api/pyhazards", "api/pyhazards.datasets", "api/pyhazards.engine", "api/pyhazards.metrics", "api/pyhazards.models", "api/pyhazards.utils", "cite", "datasets/era5", "datasets/firms", "datasets/goesr", "datasets/landfire", "datasets/merra2", "datasets/mtbs", "datasets/noaa_flood", "datasets/wfigs", "implementation", "index", "installation", "pyhazards_datasets", "pyhazards_engine", "pyhazards_metrics", "pyhazards_models", "pyhazards_utils", "quick_start", "references", "team"], "envversion": {"sphinx": 64, "sphinx.domains.c": 3, "sphinx.domains.changeset": 1, "sphinx.domains.citation": 1, "sphinx.domains.cpp": 9, "sphinx.domains.index": 1, "sphinx.domains.javascript": 3, "sphinx.domains.math": 2, "sphinx.domains.python": 4, "sphinx.domains.rst": 2, "sphinx.domains.std": 2, "sphinx.ext.viewcode": 1}, "filenames": ["api/modules.rst", "api/pyhazards.rst", "api/pyhazards.datasets.rst", "api/pyhazards.engine.rst", "api/pyhazards.metrics.rst", "api/pyhazards.models.rst", "api/pyhazards.utils.rst", "cite.rst", "datasets/era5.rst", "datasets/firms.rst", "datasets/goesr.rst", "datasets/landfire.rst", "datasets/merra2.rst", "datasets/mtbs.rst", "datasets/noaa_flood.rst", "datasets/wfigs.rst", "implementation.rst", "index.rst", "installation.rst", "pyhazards_datasets.rst", "pyhazards_engine.rst", "pyhazards_metrics.rst", "pyhazards_models.rst", "pyhazards_utils.rst", "quick_start.rst", "references.rst", "team.rst"], "indexentries": {}, "objects": {"": [[1, 0, 0, "-", "pyhazards"]], "pyhazards": [[1, 1, 1, "", "CNNPatchEncoder"], [1, 1, 1, "", "ClassificationHead"], [1, 1, 1, "", "ClassificationMetrics"], [1, 1, 1, "", "DataBundle"], [1, 1, 1, "", "DataSplit"], [1, 1, 1, "", "Dataset"], [1, 1, 1, "", "FeatureSpec"], [1, 1, 1, "", "LabelSpec"], [1, 1, 1, "", "MLPBackbone"], [1, 1, 1, "", "MetricBase"], [1, 1, 1, "", "RegressionHead"], [1, 1, 1, "", "RegressionMetrics"], [1, 1, 1, "", "SegmentationHead"], [1, 1, 1, "", "SegmentationMetrics"], [1, 1, 1, "", "TemporalEncoder"], [1, 1, 1, "", "Trainer"], [1, 4, 1, "", "available_datasets"], [1, 4, 1, "", "available_models"], [1, 4, 1, "", "build_model"], [2, 0, 0, "-", "datasets"], [3, 0, 0, "-", "engine"], [1, 4, 1, "", "load_dataset"], [4, 0, 0, "-", "metrics"], [5, 0, 0, "-", "models"], [1, 4, 1, "", "register_dataset"], [1, 4, 1, "", "register_model"], [6, 0, 0, "-", "utils"]], "pyhazards.CNNPatchEncoder": [[1, 2, 1, "", "forward"]], "pyhazards.ClassificationHead": [[1, 2, 1, "", "forward"]], "pyhazards.ClassificationMetrics": [[1, 3, 1, "", "_abc_impl"], [1, 2, 1, "", "compute"], [1, 2, 1, "", "reset"], [1, 2, 1, "", "update"]], "pyhazards.DataBundle": [[1, 3, 1, "", "feature_spec"], [1, 2, 1, "", "get_split"], [1, 3, 1, "", "label_spec"], [1, 3, 1, "", "metadata"], [1, 3, 1, "", "splits"]], "pyhazards.DataSplit": [[1, 3, 1, "", "inputs"], [1, 3, 1, "", "metadata"], [1, 3, 1, "", "targets"]], "pyhazards.Dataset": [[1, 2, 1, "", "_load"], [1, 2, 1, "", "load"], [1, 3, 1, "", "name"]], "pyhazards.FeatureSpec": [[1, 3, 1, "", "channels"], [1, 3, 1, "", "description"], [1, 3, 1, "", "extra"], [1, 3, 1, "", "input_dim"]], "pyhazards.LabelSpec": [[1, 3, 1, "", "description"], [1, 3, 1, "", "extra"], [1, 3, 1, "", "num_targets"], [1, 3, 1, "", "task_type"]], "pyhazards.MLPBackbone": [[1, 2, 1, "", "forward"]], "pyhazards.MetricBase": [[1, 3, 1, "", "_abc_impl"], [1, 2, 1, "", "compute"], [1, 2, 1, "", "reset"], [1, 2, 1, "", "update"]], "pyhazards.RegressionHead": [[1, 2, 1, "", "forward"]], "pyhazards.RegressionMetrics": [[1, 3, 1, "", "_abc_impl"], [1, 2, 1, "", "compute"], [1, 2, 1, "", "reset"], [1, 2, 1, "", "update"]], "pyhazards.SegmentationHead": [[1, 2, 1, "", "forward"]], "pyhazards.SegmentationMetrics": [[1, 3, 1, "", "_abc_impl"], [1, 2, 1, "", "compute"], [1, 2, 1, "", "reset"], [1, 2, 1, "", "update"]], "pyhazards.TemporalEncoder": [[1, 2, 1, "", "forward"]], "pyhazards.Trainer": [[1, 2, 1, "", "_make_loader"], [1, 2, 1, "", "_to_device"], [1, 2, 1, "", "evaluate"], [1, 2, 1, "", "fit"], [1, 2, 1, "", "predict"], [1, 2, 1, "", "save_checkpoint"]], "pyhazards.datasets": [[2, 1, 1, "", "DataBundle"], [2, 1, 1, "", "DataSplit"], [2, 1, 1, "", "Dataset"], [2, 1, 1, "", "FeatureSpec"], [2, 1, 1, "", "GraphTemporalDataset"], [2, 1, 1, "", "LabelSpec"], [2, 4, 1, "", "available_datasets"], [2, 0, 0, "-", "base"], [2, 4, 1, "", "graph_collate"], [2, 0, 0, "-", "hazards"], [2, 4, 1, "", "load_dataset"], [2, 4, 1, "", "register_dataset"], [2, 0, 0, "-", "registry"], [2, 0, 0, "-", "transforms"]], "pyhazards.datasets.DataBundle": [[2, 3, 1, "", "feature_spec"], [2, 2, 1, "", "get_split"], [2, 3, 1, "", "label_spec"], [2, 3, 1, "", "metadata"], [2, 3, 1, "", "splits"]], "pyhazards.datasets.DataSplit": [[2, 3, 1, "", "inputs"], [2, 3, 1, "", "metadata"], [2, 3, 1, "", "targets"]], "pyhazards.datasets.Dataset": [[2, 2, 1, "", "_load"], [2, 2, 1, "", "load"], [2, 3, 1, "", "name"]], "pyhazards.datasets.FeatureSpec": [[2, 3, 1, "", "channels"], [2, 3, 1, "", "description"], [2, 3, 1, "", "extra"], [2, 3, 1, "", "input_dim"]], "pyhazards.datasets.LabelSpec": [[2, 3, 1, "", "description"], [2, 3, 1, "", "extra"], [2, 3, 1, "", "num_targets"], [2, 3, 1, "", "task_type"]], "pyhazards.datasets.base": [[2, 1, 1, "", "DataBundle"], [2, 1, 1, "", "DataSplit"], [2, 1, 1, "", "Dataset"], [2, 1, 1, "", "FeatureSpec"], [2, 1, 1, "", "LabelSpec"], [2, 1, 1, "", "Transform"]], "pyhazards.datasets.base.DataBundle": [[2, 3, 1, "", "feature_spec"], [2, 2, 1, "", "get_split"], [2, 3, 1, "", "label_spec"], [2, 3, 1, "", "metadata"], [2, 3, 1, "", "splits"]], "pyhazards.datasets.base.DataSplit": [[2, 3, 1, "", "inputs"], [2, 3, 1, "", "metadata"], [2, 3, 1, "", "targets"]], "pyhazards.datasets.base.Dataset": [[2, 2, 1, "", "_load"], [2, 2, 1, "", "load"], [2, 3, 1, "", "name"]], "pyhazards.datasets.base.FeatureSpec": [[2, 3, 1, "", "channels"], [2, 3, 1, "", "description"], [2, 3, 1, "", "extra"], [2, 3, 1, "", "input_dim"]], "pyhazards.datasets.base.LabelSpec": [[2, 3, 1, "", "description"], [2, 3, 1, "", "extra"], [2, 3, 1, "", "num_targets"], [2, 3, 1, "", "task_type"]], "pyhazards.datasets.base.Transform": [[2, 3, 1, "", "_abc_impl"], [2, 3, 1, "", "_is_protocol"]], "pyhazards.datasets.registry": [[2, 4, 1, "", "available_datasets"], [2, 4, 1, "", "load_dataset"], [2, 4, 1, "", "register_dataset"]], "pyhazards.engine": [[3, 1, 1, "", "DistributedConfig"], [3, 1, 1, "", "SlidingWindowInference"], [3, 1, 1, "", "Trainer"], [3, 0, 0, "-", "distributed"], [3, 0, 0, "-", "inference"], [3, 4, 1, "", "select_strategy"], [3, 0, 0, "-", "trainer"]], "pyhazards.engine.DistributedConfig": [[3, 3, 1, "", "devices"], [3, 3, 1, "", "strategy"]], "pyhazards.engine.Trainer": [[3, 2, 1, "", "_make_loader"], [3, 2, 1, "", "_to_device"], [3, 2, 1, "", "evaluate"], [3, 2, 1, "", "fit"], [3, 2, 1, "", "predict"], [3, 2, 1, "", "save_checkpoint"]], "pyhazards.engine.distributed": [[3, 1, 1, "", "DistributedConfig"], [3, 4, 1, "", "select_strategy"]], "pyhazards.engine.distributed.DistributedConfig": [[3, 3, 1, "", "devices"], [3, 3, 1, "", "strategy"]], "pyhazards.engine.inference": [[3, 1, 1, "", "SlidingWindowInference"]], "pyhazards.engine.trainer": [[3, 1, 1, "", "Trainer"]], "pyhazards.engine.trainer.Trainer": [[3, 2, 1, "", "_make_loader"], [3, 2, 1, "", "_to_device"], [3, 2, 1, "", "evaluate"], [3, 2, 1, "", "fit"], [3, 2, 1, "", "predict"], [3, 2, 1, "", "save_checkpoint"]], "pyhazards.metrics": [[4, 1, 1, "", "ClassificationMetrics"], [4, 1, 1, "", "MetricBase"], [4, 1, 1, "", "RegressionMetrics"], [4, 1, 1, "", "SegmentationMetrics"]], "pyhazards.metrics.ClassificationMetrics": [[4, 3, 1, "", "_abc_impl"], [4, 2, 1, "", "compute"], [4, 2, 1, "", "reset"], [4, 2, 1, "", "update"]], "pyhazards.metrics.MetricBase": [[4, 3, 1, "", "_abc_impl"], [4, 2, 1, "", "compute"], [4, 2, 1, "", "reset"], [4, 2, 1, "", "update"]], "pyhazards.metrics.RegressionMetrics": [[4, 3, 1, "", "_abc_impl"], [4, 2, 1, "", "compute"], [4, 2, 1, "", "reset"], [4, 2, 1, "", "update"]], "pyhazards.metrics.SegmentationMetrics": [[4, 3, 1, "", "_abc_impl"], [4, 2, 1, "", "compute"], [4, 2, 1, "", "reset"], [4, 2, 1, "", "update"]], "pyhazards.models": [[5, 1, 1, "", "CNNPatchEncoder"], [5, 1, 1, "", "ClassificationHead"], [5, 1, 1, "", "MLPBackbone"], [5, 1, 1, "", "RegressionHead"], [5, 1, 1, "", "SegmentationHead"], [5, 1, 1, "", "TemporalEncoder"], [5, 1, 1, "", "TverskyLoss"], [5, 1, 1, "", "WildfireASPP"], [5, 1, 1, "", "WildfireCNNASPP"], [5, 1, 1, "", "WildfireMamba"], [5, 4, 1, "", "available_models"], [5, 0, 0, "-", "backbones"], [5, 4, 1, "", "build_model"], [5, 0, 0, "-", "builder"], [5, 4, 1, "", "cnn_aspp_builder"], [5, 0, 0, "-", "heads"], [5, 4, 1, "", "register_model"], [5, 0, 0, "-", "registry"], [5, 4, 1, "", "wildfire_aspp_builder"], [5, 4, 1, "", "wildfire_mamba_builder"]], "pyhazards.models.CNNPatchEncoder": [[5, 2, 1, "", "forward"]], "pyhazards.models.ClassificationHead": [[5, 2, 1, "", "forward"]], "pyhazards.models.MLPBackbone": [[5, 2, 1, "", "forward"]], "pyhazards.models.RegressionHead": [[5, 2, 1, "", "forward"]], "pyhazards.models.SegmentationHead": [[5, 2, 1, "", "forward"]], "pyhazards.models.TemporalEncoder": [[5, 2, 1, "", "forward"]], "pyhazards.models.TverskyLoss": [[5, 2, 1, "", "forward"]], "pyhazards.models.WildfireCNNASPP": [[5, 2, 1, "", "forward"]], "pyhazards.models.WildfireMamba": [[5, 2, 1, "", "_get_adjacency"], [5, 2, 1, "", "_temporal_delta"], [5, 2, 1, "", "forward"], [5, 2, 1, "", "set_adjacency"]], "pyhazards.models.backbones": [[5, 1, 1, "", "CNNPatchEncoder"], [5, 1, 1, "", "MLPBackbone"], [5, 1, 1, "", "TemporalEncoder"]], "pyhazards.models.backbones.CNNPatchEncoder": [[5, 2, 1, "", "forward"]], "pyhazards.models.backbones.MLPBackbone": [[5, 2, 1, "", "forward"]], "pyhazards.models.backbones.TemporalEncoder": [[5, 2, 1, "", "forward"]], "pyhazards.models.builder": [[5, 4, 1, "", "build_model"], [5, 4, 1, "", "default_builder"]], "pyhazards.models.heads": [[5, 1, 1, "", "ClassificationHead"], [5, 1, 1, "", "RegressionHead"], [5, 1, 1, "", "SegmentationHead"]], "pyhazards.models.heads.ClassificationHead": [[5, 2, 1, "", "forward"]], "pyhazards.models.heads.RegressionHead": [[5, 2, 1, "", "forward"]], "pyhazards.models.heads.SegmentationHead": [[5, 2, 1, "", "forward"]], "pyhazards.models.registry": [[5, 4, 1, "", "available_models"], [5, 4, 1, "", "get_model_config"], [5, 4, 1, "", "register_model"]], "pyhazards.utils": [[6, 4, 1, "", "auto_device"], [6, 0, 0, "-", "common"], [6, 4, 1, "", "get_device"], [6, 4, 1, "", "get_logger"], [6, 0, 0, "-", "hardware"], [6, 4, 1, "", "num_devices"], [6, 4, 1, "", "seed_all"], [6, 4, 1, "", "set_device"]], "pyhazards.utils.common": [[6, 4, 1, "", "get_logger"], [6, 4, 1, "", "seed_all"]], "pyhazards.utils.hardware": [[6, 4, 1, "", "auto_device"], [6, 4, 1, "", "get_device"], [6, 4, 1, "", "num_devices"], [6, 4, 1, "", "set_device"]]}, "objnames": {"0": ["py", "module", "Python module"], "1": ["py", "class", "Python class"], "2": ["py", "method", "Python method"], "3": ["py", "attribute", "Python attribute"], "4": ["py", "function", "Python function"]}, "objtypes": {"0": "py:module", "1": "py:class", "2": "py:method", "3": "py:attribute", "4": "py:function"}, "terms": {"": [10, 11, 13, 14, 15, 19, 26], "0": [1, 3, 5, 8, 10, 12, 16, 17, 18, 19, 22, 24], "00230": 10, "008": 9, "01": 19, "0301003": 13, "06": 5, "0758": 12, "08": 9, "1": [1, 3, 5, 10, 12, 13, 17, 22], "10": [8, 9, 10, 11, 12, 13, 16, 20], "100": [], "1000": [16, 19], "1002": 8, "1016": 9, "1071": 11, "1175": [10, 12], "12": [5, 18, 22], "128": [1, 5, 16, 22], "14": 12, "143": 9, "146": 8, "15": 10, "16": [10, 12, 16, 17, 19, 20, 22, 24], "17": 10, "18": [10, 11], "1980": 12, "1999": 8, "1e": [5, 16, 17, 20, 22, 24], "2": [1, 5, 8, 10, 16, 17, 18, 19, 20, 22, 24], "2007": [13, 19], "2009": 11, "2013": 9, "2014": [9, 19], "2017": [10, 12, 19], "2020": [8, 19], "2024": 19, "2025": 17, "2049": 8, "21": 13, "235": 11, "24": 17, "249": 11, "25": 8, "256": [1, 5, 22], "3": [1, 5, 11, 12, 13, 16, 17, 18, 20, 22, 24], "30": [11, 12, 13], "31": [], "32": [1, 3, 5, 17, 22], "350": [17, 24], "375": 9, "3803": 8, "4": [10, 17], "42": 6, "425": [17, 24], "48": 22, "4996": 13, "5": [5, 10, 12, 17, 22, 24], "500": [17, 24], "5419": 12, "5454": 12, "6": [5, 17, 18, 22], "60": [], "625": 12, "64": [1, 3, 5, 22], "681": 10, "698": 10, "730": 8, "8": [17, 18, 22], "800": [16, 19], "85": 9, "900": [16, 19], "96": 9, "98": 10, "A": [10, 11, 13, 17, 22], "For": [12, 22, 24], "If": [7, 17], "It": [8, 12, 22], "Near": [8, 9, 12, 19], "Or": 24, "The": [8, 9, 10, 11, 12, 19, 20, 22, 26], "To": [16, 24], "_": [], "_abc": [1, 2, 4], "_abc_data": [1, 2, 4], "_abc_impl": [0, 1, 2, 4], "_get_adjac": [1, 5], "_is_protocol": [1, 2], "_load": [0, 1, 2, 16, 17, 19, 24], "_make_load": [0, 1, 3], "_temporal_delta": [1, 5], "_to_devic": [0, 1, 3], "abc": [1, 4], "abi": 10, "abov": [], "abstract": [1, 3, 4, 21], "accept": 16, "access": [], "accuraci": 21, "across": [11, 13, 14, 15, 19], "activ": [9, 15, 19], "adam": [16, 17, 20, 22, 24], "add": 16, "adj": 5, "adjac": [2, 5, 17, 22], "administr": 14, "adopt": 8, "advanc": 10, "aerosol": 10, "after": [12, 16], "agenc": [14, 15], "aggreg": [15, 16, 21], "ai": 17, "al": [8, 10, 12, 19], "algorithm": 9, "allow": 19, "alpha": 5, "also": 12, "america": 10, "american": 10, "ametsoc": [], "amount": 8, "amp": [16, 20], "an": [2, 15, 22], "analysi": [8, 9, 10, 12, 13, 14, 15, 19], "analyz": 17, "ani": [1, 2, 3, 5, 22], "annual": [11, 13], "anomali": [9, 10], "api": [1, 3, 17, 19, 24], "appear": 12, "append": [], "appli": [5, 22], "applic": [8, 12], "ar": [8, 9, 10, 11, 13, 14, 15, 20, 21, 22], "arcgi": 15, "architectur": [16, 17, 22], "arg": [2, 5], "argument": 16, "arrai": [], "aspp": 5, "aspp_channel": 5, "assess": [8, 9, 11, 13, 14, 17, 19], "assimil": [8, 12], "associ": [9, 15], "assum": [], "assur": 26, "atmospher": [8, 10, 12, 14, 19], "attribut": 8, "author": 17, "authorit": [15, 19], "auto": [1, 3, 20], "auto_devic": [1, 6, 20], "automat": [6, 24], "avail": [10, 12, 13, 14, 20, 24], "available_dataset": [0, 1, 2], "available_model": [0, 1, 5, 22], "awar": [9, 10, 17], "b": [5, 8, 13], "back": 22, "backbon": [0, 1, 12, 16, 17, 22], "background": 11, "backward": 5, "bam": 10, "band": 10, "base": [0, 1, 3, 4, 5, 9, 11, 13, 14, 15, 18, 19], "base_channel": 5, "baselin": [8, 10], "basic": 17, "batch": [2, 5, 22], "batch_siz": [1, 3, 5, 17, 22], "bcewithlogitsloss": [17, 22], "becaus": 22, "begin": 14, "behavior": [11, 19], "bell": 8, "below": 19, "benchmark": [8, 12, 19], "berrisford": 8, "beta": 5, "big": 10, "bin": 14, "binari": [5, 17, 22], "block": 22, "bool": [], "both": [], "boundari": 8, "box": 17, "branch": 22, "brewer": 13, "bright": 10, "broad": 12, "build": [1, 5, 17], "build_model": [0, 1, 5, 16, 17, 20, 22, 24], "build_path": [], "builder": [0, 1, 2, 16, 22], "built": [17, 19, 21], "bulk": 11, "bulletin": 10, "bundl": [1, 2, 17, 22], "burn": [13, 19], "burnsever": [], "c": 5, "c00648": 14, "c3": 8, "cache_dir": [1, 2], "cadenc": 12, "calcul": 17, "call": 21, "callabl": 2, "can": [12, 16, 18, 22], "canopi": 11, "carri": 10, "categor": 13, "caus": 15, "cd": [8, 19], "center": [14, 15], "centr": 8, "chain": 16, "chang": [8, 13, 22], "channel": [0, 1, 2], "character": 11, "checkpoint": 20, "cheng": [17, 26], "choos": 6, "chunk": [], "class": [1, 2, 3, 4, 5, 13, 16, 17, 20, 24], "classif": [1, 5, 13, 16, 17, 19, 20, 22], "classificationhead": [0, 1, 5], "classificationmetr": [0, 1, 4, 16, 17, 20, 21, 24], "cli": 22, "clim": [], "climat": [8, 12], "closer": 10, "cloud": 10, "cnn": [1, 5, 16, 17, 22], "cnn_aspp_build": [1, 5], "cnnpatchencod": [0, 1, 5], "code": 26, "collat": [2, 17], "collate_fn": [1, 3, 17], "collect": 16, "com": [15, 17], "combin": [8, 9], "common": [0, 1, 12, 21, 22, 23], "commonli": [12, 15, 19], "commun": 26, "compat": 5, "compil": 14, "complex": [1, 3, 22], "compos": [22, 26], "comprehens": [8, 17], "comput": [0, 1, 2, 4, 16, 21], "concret": 2, "conda": 18, "condit": [], "confid": 9, "config": [20, 22], "consid": 7, "consist": [1, 5, 8, 11, 12], "construct": [1, 2, 19, 22], "contain": [1, 2, 15, 19], "content": [0, 19], "context": 19, "contextu": 10, "continu": [10, 12, 19], "contourf": [], "contribut": 26, "conv_kernel": 5, "conveni": 22, "convent": 12, "coordin": [8, 9, 10, 11, 12, 13, 14, 15], "copernicu": [8, 19], "core": 12, "correl": [17, 22], "correspond": 9, "count": [5, 22], "counti": [2, 5, 14, 17, 22], "coupl": 22, "covari": [8, 11, 12, 19], "cover": [11, 21], "coverag": [8, 9, 10, 11, 12, 13, 14, 15], "cpu": [16, 24], "cr": 12, "creat": [16, 22], "critic": 10, "crop": 14, "crossentropyloss": [16, 17, 20, 24], "csiszar": 9, "csv": [9, 14], "cu126": 18, "cube": 22, "cuda": [18, 20, 24], "current": 2, "custom": [1, 3, 16, 17, 24], "d": [10, 12], "d2m": 22, "da": [], "dai": [2, 5, 9, 17, 22], "daili": [12, 19], "damag": 14, "danger": 8, "dask": [], "data": [1, 2, 3, 17, 19, 22, 24], "data_bundl": [16, 20], "data_var": [], "dataandtool": [], "dataarrai": [], "databas": [14, 19], "databundl": [0, 1, 2, 16, 17, 19, 22, 24], "datafram": [], "dataload": [1, 3], "dataset": [0, 1, 9, 11, 13, 14, 15, 17, 22, 24], "datasplit": [0, 1, 2, 16, 17, 19, 22, 24], "date": [15, 19], "date_end": [], "date_start": [], "datetim": [], "ddp": [3, 16, 17, 20], "dedic": 26, "def": [16, 17, 19, 22, 24], "default": [1, 5, 16, 20, 22], "default_build": [1, 5, 22], "deleg": [1, 5], "demonstr": 19, "densiti": 11, "depend": [9, 10, 11, 13, 15], "deploi": 17, "depth": [1, 5, 22], "deriv": [9, 10, 11, 12, 13, 14, 19], "describ": [1, 2, 19], "descript": [0, 1, 2, 9, 14, 16, 17, 19, 24], "descriptor": 11, "design": [17, 21], "detail": 24, "detect": [9, 10, 15, 19, 24], "determinist": 6, "develop": 8, "devic": [1, 3, 6, 17, 23, 24], "device_str": 6, "dewpoint": 8, "df": [], "diagnost": [8, 10, 12], "dice": 21, "dict": [1, 2, 3, 4, 5, 22], "differ": [], "differenti": 22, "dilat": 5, "dimens": [], "direct": 26, "directli": 21, "directori": [], "disc": 12, "discov": 19, "discover": 22, "discoveri": 15, "displai": [], "distanc": [17, 22], "distribut": [0, 1, 8, 9, 10, 15, 16], "distributedconfig": [1, 3], "disturb": 11, "dnbr": 13, "document": [12, 14, 15, 24, 26], "doi": [8, 9, 10, 11, 12, 13], "dong": 26, "download": 18, "downstream": [1, 2], "dp": 3, "driven": [16, 22], "driver": [12, 14, 19], "drop": [], "dropout": 5, "drought": 12, "ds_pre": [], "ds_sfc": [], "dtype": [1, 2], "e": [9, 11, 12, 13, 15, 16, 19, 22], "each": [2, 9, 12, 15, 19], "earli": [9, 10, 19], "earthdata": [9, 12], "earthquak": 2, "easi": [1, 2, 17], "ecmwf": [8, 19], "ecolog": [11, 13], "ecologi": 13, "ecosystem": 19, "edu": 26, "eidenshink": [13, 19], "els": [], "emerg": 14, "enabl": [5, 20], "encod": [1, 5, 16, 17, 22], "end": [11, 12, 14, 21], "end_dat": [], "energi": 12, "engag": 26, "engin": [0, 1, 16, 17, 22, 24], "environ": [9, 18, 24], "environment": [8, 10, 12, 14, 17], "eosdi": [], "equal": [], "era": 12, "era5": [5, 17, 19, 22], "estim": [8, 12], "et": [8, 10, 12, 19], "etc": 2, "eu": [], "european": 8, "evalu": [0, 1, 3, 13, 16, 17, 20, 22, 24], "event": [9, 13, 15, 19], "evolut": 10, "exampl": [16, 17, 18], "exist": [], "explicitli": 24, "explor": 19, "export": [22, 24], "extend": [1, 3, 21], "extens": [17, 22], "extent": [13, 19], "extern": 5, "extra": [0, 1, 2, 17, 22], "extract": 14, "extractor": 22, "extrem": [8, 12], "ey": [17, 22], "f": [], "factori": [1, 2], "fake": [17, 22], "fall": 22, "fals": [1, 3, 5, 6, 17, 22], "familiar": [1, 3], "fatal": 14, "featur": [1, 2, 5, 16, 17, 19, 22, 24], "feature_spec": [0, 1, 2, 16, 17, 19, 22, 24], "featurespec": [0, 1, 2, 16, 17, 19, 22, 24], "field": [8, 10, 11, 12], "fifth": 8, "file": [13, 19], "filenotfounderror": [], "find": 7, "fire": [9, 10, 11, 13, 15, 17, 19, 22], "fireecologi": 13, "firelab": [], "firm": [15, 19], "first": [16, 17, 18, 24], "fit": [0, 1, 3, 16, 17, 20, 22, 24], "flag": [6, 9], "float": [1, 3, 4, 5, 17, 22], "flood": [2, 8, 19], "florida": 26, "flux": [8, 12], "follow": 7, "forc": [8, 12], "forecast": 8, "forest": 11, "format": [8, 9, 10, 11, 12, 13, 14, 15], "forward": [0, 1, 5], "found": [], "framework": 17, "frequenc": [10, 12, 14, 19], "frequent": [], "from": [9, 10, 11, 13, 14, 16, 17, 19, 20, 21, 22, 24], "from_logit": 5, "frp": 9, "fsu": 26, "fuel": [9, 11, 13, 15, 19], "full": 21, "function": [2, 16, 17, 22, 24], "g": [9, 11, 12, 13, 15, 16, 19, 22], "gca": [], "gcn": 22, "gcn_hidden": 5, "ge": 12, "gelaro": [12, 19], "gener": [5, 8, 10], "geo": 12, "geodatabas": 13, "geograph": [9, 15], "geojson": [9, 15], "geolog": 13, "geometri": 15, "geopotenti": [8, 12], "geospati": [11, 15], "geostationari": [10, 19], "geotiff": [11, 13], "get": [16, 22, 24], "get_devic": [1, 6], "get_logg": [1, 6], "get_model_config": [1, 5], "get_split": [0, 1, 2], "gi": 15, "giglio": 9, "github": 17, "given": 22, "global": [8, 9, 12, 19], "gmao": [12, 19], "goe": [15, 19], "goesr": 19, "gov": 14, "gpu": [16, 17, 20], "graph": 17, "graph_col": [1, 2, 17, 22], "graphtemporaldataset": [1, 2, 17, 22], "grib": 8, "grid": [3, 8, 9, 10, 11, 12, 14, 19, 20], "griffith": 10, "ground": [10, 15, 19], "growth": 10, "gru": [1, 5], "gsfc": [], "guid": 24, "gunshor": 10, "h": [5, 8], "h5netcdf": [], "handl": [20, 22], "hardwar": [0, 1, 20, 23], "hazard": [0, 1, 8, 9, 10, 11, 12, 14, 16, 17, 19, 24], "head": [0, 1, 16, 17, 22], "heat": 12, "height": [11, 12], "help": 24, "helper": [17, 20, 22, 23, 24], "hemispher": 10, "hersbach": [8, 19], "hidden": [16, 22], "hidden_dim": [1, 5, 16, 22], "high": [8, 10, 19], "highlight": 22, "histor": [8, 14, 15], "hold": 19, "home": [], "hotspot": [9, 15], "hourli": [8, 12, 19], "how": [16, 19], "howard": 13, "http": [8, 9, 10, 11, 12, 13, 14, 15, 17, 18], "humid": 12, "hurrican": [2, 12], "i": [2, 5, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 20, 22, 26], "id": 14, "ident": 22, "identifi": 15, "ignit": [10, 11, 13, 15, 19], "imag": 10, "imageri": [10, 13, 19], "impact": [13, 14, 19], "implement": [2, 3, 17, 19, 24], "import": [16, 17, 19, 20, 21, 22, 24], "in_channel": [1, 5], "in_dim": [1, 5, 16, 17, 20, 22, 24], "incid": [15, 19], "includ": [8, 9, 10, 11, 12, 13, 14, 15], "index": [2, 16, 17, 18], "indic": [9, 10], "individu": 13, "infer": [0, 1, 20], "inform": [9, 14, 15], "infrar": 10, "initi": 9, "injuri": 14, "input": [0, 1, 2, 3, 5, 8, 12, 19, 22], "input_dim": [0, 1, 2, 5, 16, 17, 19, 22, 24], "ins": 22, "inspect_d": [], "int": [1, 2, 3, 6, 16, 22], "integr": [9, 10, 11, 13, 14, 15], "interag": [15, 19], "interfac": [1, 5, 17, 19, 21, 24], "intern": [11, 22], "iou": 21, "ipython": [], "isel": [], "iso": 14, "item": [], "iter": [1, 3], "j": [9, 10, 12, 13], "jcli": 12, "journal": [8, 11, 12], "k": 13, "keep": [1, 2, 5, 22], "keyerror": [], "kind": [], "km": 10, "kml": 9, "kwarg": [1, 2, 5, 16, 22], "l": 9, "label": [1, 2, 9, 14, 15, 17, 19, 22], "label_spec": [0, 1, 2, 16, 17, 19, 22, 24], "labelspec": [0, 1, 2, 16, 17, 19, 22, 24], "labrai": 17, "land": [8, 9, 12, 14], "landfir": 19, "landsat": [13, 19], "landscap": [11, 19], "landslid": 2, "larg": [3, 20], "lat": 12, "latenc": 12, "latest": 10, "latitud": [8, 9, 10, 12], "layer": [8, 11, 13, 16, 19, 22], "learn": [8, 11, 12, 14, 17], "len": [], "let": [], "level": [8, 9, 12, 14, 15, 19], "librari": 17, "lightweight": [1, 3, 5, 19, 22], "like": [17, 22], "limit": 12, "line": 16, "linear": [16, 22], "list": [1, 3], "list_var": [], "liter": 3, "load": [0, 1, 2, 16, 17, 24], "load_dataset": [0, 1, 2, 19], "loader": 2, "local": 14, "locat": [9, 14, 15, 19], "log": [17, 23], "logger": 6, "logic": 3, "logit": [5, 17, 22], "lon": 12, "long": [12, 13, 19], "longitud": [8, 9, 10, 12], "look": 10, "loop": [1, 3, 22], "loss": 5, "loss_fn": [1, 3, 16, 17, 20, 22, 24], "lower": [], "lr": [16, 17, 20, 22, 24], "m": [8, 9, 10, 11, 12, 13, 19], "machin": [8, 11, 12, 17], "mae": 21, "maintain": 15, "mainten": 26, "make": [1, 2], "mamba": 5, "mamba_lay": 5, "manag": [9, 11, 14, 15, 17, 23], "map": [11, 13], "mask": [1, 5], "match": 18, "matplotlib": [], "matrix": [17, 22], "max": [], "max_epoch": [1, 3, 16, 17, 20, 22, 24], "max_show": [], "max_var": [], "mccarti": 12, "mean": [], "medium": 8, "merra": 19, "merra2": 19, "merra2_sfc_": [], "merra_pres_": [], "metadata": [0, 1, 2, 5, 13, 14, 15, 19], "meteorolog": [8, 9, 10, 11, 12, 14, 15, 19], "meteorologi": [8, 12, 13, 19], "metric": [0, 1, 3, 13, 17, 20, 24], "metricbas": [0, 1, 4, 16, 21], "min": [], "minim": [1, 3, 22], "minut": 10, "miss": [], "mix": [17, 22], "mixed_precis": [1, 3, 16, 17, 20, 22, 24], "mlp": [1, 5, 16, 17, 20, 22, 24], "mlpbackbon": [0, 1, 5], "modap": [], "mode": 10, "model": [0, 1, 2, 3, 8, 9, 11, 12, 13, 14, 15, 17, 19, 20, 24], "modern": [8, 12], "modi": [9, 19], "modul": [0, 16, 17, 23], "modular": [16, 17], "moistur": 12, "monitor": [9, 10, 13, 19], "month": 12, "monthli": [12, 14], "more": [22, 24], "motion": 10, "mtb": 19, "multi": [1, 5, 12, 16, 17], "multipl": [9, 10, 11, 20], "multispectr": [10, 19], "must": [], "my_custom_build": 22, "my_hazard": [16, 19], "my_mlp": [16, 22], "my_model_build": 16, "myhazard": 16, "myhazarddataset": 19, "n": 5, "n_var": [], "name": [0, 1, 2, 5, 6, 13, 15, 16, 17, 19, 20, 22, 24], "namespac": 2, "narr": 14, "nasa": [9, 12, 19], "nation": [11, 13, 14, 15], "nationwid": [11, 19], "natur": [8, 12, 17], "nc": [], "ncdc": 14, "ncei": 14, "need": [3, 21, 22], "neighbor": 22, "netcdf": [8, 10], "netcdf4": 12, "new": [9, 12], "next": [5, 17, 22], "nifc": 15, "nn": [16, 17, 20, 22, 24], "no_grad": [17, 22], "noaa": [10, 19], "noaa_flood": 19, "none": [1, 2, 3, 4, 5, 6, 17, 22], "normal": [1, 2, 16], "note": 18, "np": [], "num_class": [1, 4, 5], "num_counti": [2, 5, 17, 22], "num_devic": [1, 6], "num_featur": [2, 5, 17, 22], "num_lay": [1, 5], "num_target": [0, 1, 2, 16, 17, 19, 22, 24], "num_work": [1, 3], "numer": 12, "numpi": [], "o": [], "obj": [1, 3], "object": [1, 2, 3, 4, 19], "observ": [8, 9, 10, 11, 12], "occurr": [9, 14, 15, 19], "ocean": 14, "offic": [12, 14], "offici": 15, "often": 19, "oliva": 9, "one": 17, "ongo": 15, "onlinelibrari": [], "open": [], "open_dataset": [], "open_merra": [], "opendata": 15, "oper": [9, 10, 12, 19], "optim": [1, 3, 16, 17, 20, 22, 24], "option": [1, 2, 5, 8, 16, 17, 20, 22], "org": [8, 9, 10, 11, 12, 13, 18], "other": 23, "otherwis": 20, "our": 26, "out": 17, "out_dim": [1, 5, 16, 17, 20, 22, 24], "outdir": 19, "output": [1, 5, 19, 22], "over": [3, 10, 21, 22], "overpass": 9, "overrid": 5, "overview": 19, "own": [16, 22], "p": [8, 9, 10], "packag": 0, "page": [14, 17], "pair": [14, 19], "panda": [], "paramet": [5, 16, 17, 20, 22, 24], "particularli": [10, 14], "pass": [16, 21, 22], "past_dai": [2, 5, 17, 22], "patch": [1, 5, 16, 21, 22], "path": [1, 3], "pathlib": [], "pattern": [9, 15], "pattern_pr": [], "pattern_sfc": [], "pd": [], "per": [5, 9, 22], "perimet": [13, 15, 19], "period": 12, "physic": 12, "pick": [], "pip": 18, "pipelin": [8, 9, 11, 12, 15, 17], "pixel": 21, "placehold": [2, 3, 20], "plain": 22, "plan": [11, 26], "pleas": [7, 17, 24], "plt": [], "plu": [1, 2, 19, 22], "point": [9, 10, 14, 15], "polygon": [13, 15], "popul": 2, "portal": [9, 11, 13, 15], "post": [13, 19], "potenti": 9, "power": [9, 17], "pre": [], "precipit": [8, 12], "precis": 17, "pred": [1, 4, 20], "predict": [0, 1, 3, 8, 9, 11, 12, 13, 14, 16, 17, 19, 20, 21, 22], "prefer": [3, 6], "preprocess": [2, 16], "present": 12, "pressur": [8, 12, 19], "print": [17, 22, 24], "prithvi": [], "prob": [17, 22], "probabl": [5, 17, 22], "process": 10, "produc": [8, 11, 12, 13], "product": [9, 10, 11, 12, 13, 15], "profil": 12, "program": [10, 11, 13, 19], "project": [10, 11, 13, 15], "properti": 14, "protocol": 2, "provid": [1, 2, 8, 9, 10, 11, 12, 14, 15, 16, 17, 19, 22], "public": 12, "publicli": [9, 11, 13, 14, 15], "publish": 12, "pyhazard": [12, 16, 17, 19, 20, 21, 22, 23, 24, 26], "pyhazards2025": 17, "pyhazards_devic": [6, 24], "pyplot": [], "python": [17, 18, 19, 24], "pytorch": [17, 18, 22], "qj": 8, "qualiti": [9, 26], "quarterli": 8, "quasi": 11, "quayl": 13, "quick": 17, "quickli": [19, 24], "r": [12, 19], "radi": 9, "radianc": 10, "radiat": 8, "rais": [], "randint": [16, 17, 19, 22, 24], "randn": [16, 17, 19, 22, 24], "random": 22, "rang": 8, "rapid": [9, 10], "raster": [1, 3, 5, 10, 11, 13, 17, 19, 20, 22], "rather": 15, "rdnbr": 13, "re": 22, "readi": [1, 2, 17], "real": [9, 10, 15, 19], "reanalysi": [8, 10, 12, 14, 19], "receiv": 22, "recommend": 18, "record": [9, 10, 12, 14, 15, 19], "refer": 24, "refresh": 10, "regim": [11, 13, 19], "region": 14, "regist": [2, 16, 17], "register_dataset": [0, 1, 2, 16, 19], "register_model": [0, 1, 5, 16, 22], "registr": [22, 24], "registri": [0, 1, 16, 17, 19, 22], "regress": [1, 2, 5, 16, 17, 22], "regressionhead": [0, 1, 5], "regressionmetr": [0, 1, 4, 21], "regular": [8, 12], "relat": [8, 10, 14, 19], "releas": 11, "relu": [16, 22], "remot": [9, 11, 13], "replac": [1, 3, 17, 22], "report": [14, 15, 19], "repres": 15, "research": [12, 17, 26], "reset": [0, 1, 4, 16, 21], "reset_index": [], "resolut": [8, 9, 10, 11, 12, 13, 14, 15], "resourc": [9, 11], "respect": 6, "respons": 26, "result": [16, 17, 20, 24], "retrospect": 12, "return": [1, 2, 3, 4, 5, 6, 16, 17, 19, 22, 24], "reusabl": [2, 16, 22], "review": 26, "risk": [8, 11, 17, 19], "rmet": [], "rmse": 21, "role": 12, "rollin": 11, "root": [], "row": [], "royal": 8, "rse": 9, "run": 20, "runyang": [], "sampl": [2, 17, 22], "satellit": [9, 10, 12, 15], "save": 20, "save_checkpoint": [0, 1, 3], "scalar": [1, 5], "scale": [11, 13], "schmit": 10, "schroeder": [9, 19], "schwind": 13, "search": 17, "see": 19, "seed": [6, 17, 23], "seed_al": [1, 6], "segment": [1, 5, 16, 17, 22], "segmentationhead": [0, 1, 5], "segmentationmetr": [0, 1, 4, 21], "select": [20, 22], "select_strategi": [1, 3], "self": [16, 17, 19, 24], "sens": [9, 11, 13], "sensibl": 20, "sensor": 9, "sequenti": [16, 22], "seri": [1, 5, 10, 19, 22], "serv": 19, "servic": [8, 9, 11, 12, 15], "set": [5, 8, 12, 24], "set_adjac": [1, 5], "set_aspect": [], "set_devic": [1, 6, 24], "sever": [13, 14, 19], "sfc": [], "shape": [1, 2, 5, 17, 22], "shapefil": [9, 13, 15], "ship": 22, "short": [], "should": [1, 2], "show": 16, "shuffl": [1, 3], "sigmoid": [5, 17, 22], "signal": [1, 5], "simpl": [1, 2, 5, 16, 17], "simplifi": 19, "simul": 11, "singl": [1, 2, 8, 19, 20], "situat": [9, 10], "size": 15, "skin": 12, "skipna": [], "slide": [3, 20], "slidingwindowinfer": [1, 3], "slowli": 11, "smoke": [10, 19], "smooth": 5, "so": [16, 22], "societi": [8, 10], "softwar": 17, "soil": 12, "sort_valu": [], "sourc": [1, 2, 3, 4, 5, 6, 15], "space": 22, "spatial": [5, 8, 9, 10, 11, 12, 13, 14, 15, 22], "spatio": 5, "spatiotempor": [8, 9, 12], "spec": [1, 2, 17, 19], "specif": [1, 2, 9, 22], "spectral": [10, 13], "split": [0, 1, 2, 3, 16, 17, 19, 20, 21, 22, 24], "spread": 11, "ssm": 22, "ssr": 22, "stack": [2, 22], "stamp": 9, "standard": [5, 8, 12, 13], "start": [14, 17], "start_dat": [], "state": [8, 11, 12, 13, 14, 15, 22, 26], "state_dim": 5, "static": [5, 11, 19], "statist": [], "statu": [15, 19], "std": [], "step": [], "stitch": 3, "store": [8, 22], "storm": [14, 19], "str": [1, 2, 3, 4, 5, 16, 22], "strateg": 26, "strategi": [1, 3, 20], "stream": 12, "strftime": [], "structur": [8, 9, 10, 11, 12, 13, 14, 15, 19], "studi": [8, 12, 13, 14, 15, 19], "style": [2, 5, 12, 17, 22], "subclass": [1, 2, 16], "submodul": [0, 1], "subpackag": 0, "summarize_da": [], "supervis": [9, 14], "support": [11, 13, 16, 17, 19, 20, 26], "surfac": [8, 9, 10, 12], "survei": 13, "su\u00e1rez": 12, "switch": 22, "system": [8, 9, 10, 11, 12, 13, 14, 15], "t": 10, "t2m": 22, "tab": [], "tabl": [], "tabular": [1, 5, 14, 17, 19, 22], "target": [0, 1, 2, 3, 4, 5, 9, 14, 16, 19, 21, 22], "task": [1, 2, 5, 8, 16, 17, 20, 21, 22, 24], "task_typ": [0, 1, 2, 16, 17, 19, 22, 24], "technic": 26, "temperatur": [8, 10, 12], "tempor": [2, 5, 8, 9, 10, 11, 12, 13, 14, 15, 16, 19], "temporalencod": [0, 1, 5], "tensor": [1, 2, 3, 5, 9, 14, 17, 22], "term": [12, 13, 19], "test": [1, 2, 3, 16, 17, 19, 20, 24], "test_dai": [], "than": 15, "them": [2, 16, 22], "themat": 11, "thermal": [9, 10, 19], "thi": [1, 5, 14, 16, 22, 24], "through": [9, 10, 13, 15, 19], "throughout": [], "time": [1, 5, 9, 10, 12, 14, 15, 19, 22], "timestamp": 14, "titl": 17, "toi": 22, "too": [], "tool": 11, "topographi": 13, "torch": [16, 17, 18, 19, 20, 22, 24], "toyhazard": [17, 24], "tp": 22, "train": [1, 2, 3, 13, 17, 19, 20, 22, 24], "train_d": 17, "train_split": [1, 3], "trainer": [0, 1, 16, 17, 20, 21, 22, 24], "transform": [0, 1, 17], "treat": 15, "trend": [8, 12, 13], "true": [1, 2, 3, 5, 16, 17, 20, 22, 24], "truth": [10, 15, 19], "tupl": [], "tverski": 5, "tverskyloss": [1, 5], "two": [16, 22], "type": [1, 2, 3, 4, 5, 6, 11, 14, 22], "u": [11, 13, 14, 15, 19], "u10": 22, "under": 8, "unifi": [17, 19], "uniqu": 15, "unit": [11, 13, 14, 15], "univers": 26, "updat": [0, 1, 4, 9, 11, 12, 14, 15, 16, 21], "url": [17, 18], "us": [5, 7, 16, 17, 18, 19, 20, 22, 24], "usag": [17, 22], "usda": [], "user": [16, 19], "usf": 19, "usg": 13, "util": [0, 1, 17, 19, 20, 24], "v10": 22, "val": [1, 2, 16, 17, 19, 22, 24], "val_d": 17, "val_split": [1, 3], "valid": [13, 15], "valu": [], "valueerror": [], "var": [], "vari": 11, "variabl": [19, 24], "vast": 8, "vector": 13, "veget": [11, 19], "version": [11, 12], "vertic": [8, 12], "via": [8, 11, 12, 14, 16, 17, 19, 20, 22], "view": 10, "viir": [9, 19], "visibl": 10, "visual": [], "w": [5, 9, 12], "we": 18, "weather": [8, 10, 12, 14], "week": 12, "western": 10, "wf08088": 11, "wfig": 19, "wheel": 18, "when": [9, 14, 19, 20], "where": [10, 14], "which": [10, 14], "whl": 18, "who": 26, "wide": [8, 9, 10, 11, 12, 13, 14, 15, 19], "wildfir": [2, 5, 8, 9, 10, 11, 12, 13, 15, 19], "wildfire_aspp_build": [1, 5], "wildfire_mamba": [17, 22], "wildfire_mamba_build": [1, 5], "wildfireaspp": [1, 5], "wildfirecnnaspp": [1, 5], "wildfiremamba": [1, 5], "wildland": [11, 15], "wilei": [], "wind": [8, 12], "window": [2, 3, 16, 17, 20], "window_fn": 3, "with_count_head": [5, 22], "work": [1, 3, 7, 22], "workflow": [12, 19], "wrap": [20, 22], "www": 14, "wxc": 12, "x": [1, 2, 5, 16, 17, 19, 22, 24], "xarrai": [], "xc25": 26, "xml": [], "xr": [], "xueqi": [17, 26], "y": [2, 16, 17, 19, 22, 24], "year": [13, 17], "you": [7, 17, 22, 24], "your": [16, 17, 22], "yushun": 26, "yyyymmdd": [], "z": 13, "zhu": 13, "zone": 14}, "titles": ["pyhazards", "pyhazards package", "pyhazards.datasets package", "pyhazards.engine package", "pyhazards.metrics package", "pyhazards.models package", "pyhazards.utils package", "How to Cite", "ERA5", "FIRMS", "GOES-R", "LANDFIRE", "MERRA-2", "MTBS", "NOAA Flood Events", "WFIGS", "Implementation Guide", "Wildfire Mamba (spatio-temporal toy)", "Installation", "Datasets", "Engine", "Metrics", "Models", "Utils", "Quick Start", "References", "Core Team"], "titleterms": {"1": [], "2": 12, "3": [], "4": [], "5": [], "6": [], "At": 12, "academ": 25, "access": [8, 9, 10, 11, 12, 13, 14, 15], "backbon": 5, "base": [2, 22], "basic": 24, "build": 22, "builder": 5, "built": 22, "case": [8, 9, 10, 11, 12, 13, 14, 15], "characterist": [8, 9, 10, 11, 12, 13, 14, 15], "cite": [7, 17], "class": [19, 21], "classif": 24, "common": 6, "compon": 17, "config": [], "content": [1, 2, 3, 5, 6], "contributor": 26, "coord": [], "core": [17, 19, 20, 21, 22, 26], "custom": 22, "data": [8, 9, 10, 11, 12, 13, 14, 15], "dataset": [2, 16, 19], "date": [], "design": 22, "develop": 26, "devic": 20, "dim": [], "distribut": [3, 20], "engin": [3, 20], "era5": 8, "event": 14, "exampl": [19, 24], "fact": 12, "filenam": [], "firm": 9, "flood": 14, "glanc": 12, "goe": 10, "gpu": 24, "guid": 16, "hardwar": 6, "hazard": 2, "head": 5, "helper": [], "how": [7, 17], "implement": 16, "import": [], "indic": 17, "infer": 3, "inspect": 19, "instal": 18, "landfir": 11, "lat": [], "lead": 26, "level": [], "load": [], "lon": [], "maintain": 26, "mamba": [17, 22], "map": [], "merra": 12, "metric": [4, 16, 21], "model": [5, 16, 22], "modul": [1, 2, 3, 5, 6, 20, 22], "mtb": 13, "next": 24, "noaa": 14, "note": 22, "overview": [8, 9, 10, 11, 12, 13, 14, 15], "packag": [1, 2, 3, 4, 5, 6], "path": [], "pattern": [], "plot": [], "princip": 26, "public": 25, "pyhazard": [0, 1, 2, 3, 4, 5, 6, 18], "quick": [12, 24], "r": 10, "refer": [8, 9, 10, 11, 12, 13, 14, 15, 25], "regist": 22, "registri": [2, 5], "requir": 18, "setup": [], "skeleton": 19, "spatio": [17, 22], "start": 24, "stat": 12, "step": 24, "submodul": [2, 3, 5, 6, 23], "subpackag": 1, "summari": [19, 20, 21, 22, 23], "support": 24, "tabl": 17, "tabular": 24, "team": 26, "tempor": [17, 22], "toi": [17, 24], "train": 16, "trainer": 3, "transform": [2, 16], "typic": [8, 9, 10, 11, 12, 13, 14, 15, 20], "us": [8, 9, 10, 11, 12, 13, 14, 15], "usag": [20, 21, 24], "util": [6, 23], "variabl": [8, 9, 10, 11, 12, 13, 14, 15], "wfig": 15, "wildfir": [17, 22]}}) \ No newline at end of file diff --git a/docs/source/datasets/merra2.rst b/docs/source/datasets/merra2.rst index 8a991106..74789a9c 100644 --- a/docs/source/datasets/merra2.rst +++ b/docs/source/datasets/merra2.rst @@ -1,9 +1,35 @@ MERRA-2 ======= -Global atmospheric reanalysis dataset produced by NASA Global Modeling and Assimilation Office (GMAO), +Global atmospheric reanalysis produced by NASA Global Modeling and Assimilation Office (GMAO), providing physically consistent meteorological fields for weather, climate, and hazard modeling. +.. admonition:: At-a-glance (Quick Facts) + :class: note + + .. list-table:: + :widths: 26 74 + :stub-columns: 1 + + * - Producer + - NASA GMAO (GEOS) + * - Spatial coverage + - Global + * - Spatial resolution + - ~0.5° (lat) × 0.625° (lon) + * - Temporal resolution + - Hourly (also 3-hourly / daily / monthly products) + * - **Data update frequency** + - **Published monthly; typically available ~2–3 weeks after month end.** + * - Period of record + - 1980–present + * - Format + - NetCDF4 + * - Grid / CRS + - Regular latitude–longitude grid + * - Typical role in PyHazards + - Core meteorological driver (forcing + covariates) + ---- Overview @@ -13,20 +39,40 @@ Overview atmospheric reanalysis that assimilates satellite and conventional observations into a numerical weather prediction system to produce gridded, time-continuous estimates of the atmospheric state. -It is widely used as a core meteorological driver in climate analysis and hazard prediction pipelines, +It is widely used as a meteorological backbone for climate analysis and hazard prediction pipelines, including wildfire, hurricane, drought, and extreme heat studies. ---- +Stats +----- + +.. list-table:: + :header-rows: 1 + :widths: 22 22 22 34 + + * - Spatial + - Temporal + - Update cadence + - Coverage + * - 0.5° × 0.625° + - Hourly + - Monthly (+ ~2–3 weeks latency) + - Global, 1980–present + +---- + Data Characteristics -------------------- -- **Spatial coverage:** Global -- **Spatial resolution:** ~0.5° latitude × 0.625° longitude -- **Temporal resolution:** Hourly -- **Vertical structure:** Surface fields and multi-level pressure profiles -- **Data format:** NetCDF -- **Coordinate system:** Regular latitude–longitude grid +- **Spatial coverage:** Global +- **Spatial resolution:** ~0.5° latitude × 0.625° longitude +- **Temporal resolution:** Hourly (with derived 3-hourly/daily/monthly products) +- **Vertical structure:** Surface fields and multi-level pressure profiles +- **Data format:** NetCDF4 +- **Coordinate system:** Regular latitude–longitude grid +- **Data update frequency:** Monthly publication; new data typically appear ~2–3 weeks after month end + (common operational latency for standard MERRA-2 streams) ---- @@ -55,11 +101,15 @@ Typical Use Cases Access ------ -MERRA-2 data can be accessed via NASA GMAO and Earthdata services: +MERRA-2 data can be accessed via NASA Earthdata / GES DISC services: -- `MERRA-2 overview `_ +- `MERRA-2 overview (GMAO) `_ - `NASA Earthdata `_ +.. note:: + For standard MERRA-2 streams, operational latency is commonly documented as + ~2–3 weeks after the end of each month (monthly publication cadence). + ---- Reference @@ -68,4 +118,4 @@ Reference Gelaro, R., McCarty, W., Suárez, M. J., *et al.* (2017). *The Modern-Era Retrospective Analysis for Research and Applications, Version 2 (MERRA-2)*. Journal of Climate, 30(14), 5419–5454. -`https://doi.org/10.1175/JCLI-D-16-0758.1 `_ +`https://doi.org/10.1175/JCLI-D-16-0758.1 `_ \ No newline at end of file diff --git a/pyhazards/datasets/inspection.py b/pyhazards/datasets/inspection.py index 90b39f5b..72ee900e 100644 --- a/pyhazards/datasets/inspection.py +++ b/pyhazards/datasets/inspection.py @@ -1,205 +1,999 @@ -# pyhazards/datasets/inspection.py (或你想放的任何单文件位置，但不要在 __init__ 强制 import 它) - +# pyhazards/datasets/inspection.py from __future__ import annotations import argparse +import os +import re from pathlib import Path -from typing import Optional - -def inspect_wxc_rollout( - date: str, - outdir: str, - *, - time_range: Optional[tuple[str, str]] = None, - device: Optional[str] = None, -) -> Path: +from datetime import date, timezone + +import numpy as np +import pandas as pd +import xarray as xr +import h5py +import matplotlib.pyplot as plt +import requests + + +# --------------------------------------------------------------------- +# Optional: notebook-style display, but safe for pure python +# --------------------------------------------------------------------- +try: + from IPython.display import display # type: ignore +except Exception: + def display(x): + if hasattr(x, "to_string"): + print(x.to_string(index=False)) + else: + print(x) + + +# --------------------------------------------------------------------- +# Constants / Defaults +# --------------------------------------------------------------------- +RAW_DATASETS = [ + "M2I1NXASM", # inst1_2d_asm_Nx + "M2INVASM", # alias folder name for inst3_3d_asm_Nv (normally M2I3NVASM) + "M2T1NXFLX", # tavg1_2d_flx_Nx + "M2T1NXLND", # tavg1_2d_lnd_Nx + "M2T1NXRAD", # tavg1_2d_rad_Nx + "M2C0NXCTM", # const_2d_ctm_Nx (static) +] + +PATTERN_SFC_OUT = "MERRA2_sfc_{yyyymmdd}.nc" +PATTERN_PRES_OUT = "MERRA_pres_{yyyymmdd}.nc" + +# Merge SFC variable subsets (from mergesfc.py) +ASM1_VARS = ["QV2M", "T2M", "TQI", "TQL", "TQV", "TS", "U10M", "V10M"] +FLX_VARS = ["EFLUX", "HFLUX", "Z0M"] +RAD_VARS = ["SWGNT", "SWTNT", "LWGAB", "LWGEM", "LWTUP"] +NV_VARS = ["SLP", "PS"] +LND_VARS = ["GWETROOT", "LAI"] +STATIC_VARS = ["FRACI", "FRLAND", "FROCEAN", "PHIS"] + +TARGET_N_FRAMES = 8 +TARGET_LAT = 361 +TARGET_LON = 576 +HOURS8 = [0, 3, 6, 9, 12, 15, 18, 21] + +# Merge PRES (from mergepres.py) +PRES_VARS = ["CLOUD", "H", "OMEGA", "PL", "QI", "QL", "QV", "T", "U", "V"] +PRES_LEVELS = [ + 34.0, 39.0, 41.0, 43.0, 44.0, 45.0, 48.0, + 51.0, 53.0, 56.0, 63.0, 68.0, 71.0, 72.0, +] + +# Try to import NAN_VALS (mergesfc depends on it) +try: + from PrithviWxC.definitions import NAN_VALS # type: ignore +except Exception as e: + NAN_VALS = {} + print("[WARN] Cannot import PrithviWxC.definitions.NAN_VALS. " + "NaN filling will be skipped. Error:", e) + + +# --------------------------------------------------------------------- +# Helpers: repo root inference + date formatting +# --------------------------------------------------------------------- +def infer_repo_root() -> Path: """ - Run a minimal Prithvi-WxC rollout inspection and write a NetCDF prediction file. - - This function is SAFE TO IMPORT: it does not execute anything until called. + Infer REPO_ROOT from this file location. + We want outputs to land at: + REPO_ROOT/Prithvi-WxC/data/merra-2 + REPO_ROOT/M2I1NXASM, REPO_ROOT/M2INVASM, ... + Works no matter where the repo is cloned. + + Heuristic: + - Prefer a parent directory that contains BOTH 'Prithvi-WxC/' and 'pyhazards/'. + - Fallback: any parent that contains 'Prithvi-WxC/'. """ - # Heavy/optional imports go INSIDE the function - import random - import numpy as np - import torch - import yaml - import os - import xarray as xr - - # ---- deterministic-ish setup (safe defaults) ---- - random.seed(42) - np.random.seed(42) - torch.manual_seed(42) - if torch.cuda.is_available(): - torch.cuda.manual_seed_all(42) - torch.backends.cudnn.benchmark = True - - # ---- device selection ---- - if device is None: - dev = torch.device("cuda" if torch.cuda.is_available() else "cpu") + here = Path(__file__).resolve() + candidates: list[Path] = [] + for p in [here] + list(here.parents): + if (p / "Prithvi-WxC").is_dir(): + candidates.append(p) + + if not candidates: + raise RuntimeError( + f"Cannot infer repo root from {here}. " + f"Expected a parent dir containing 'Prithvi-WxC/'. " + f"Please pass --repo-root." + ) + + for p in candidates: + if (p / "pyhazards").is_dir(): + return p + + return candidates[0] + + +def yyyymmdd(d: date) -> str: + return d.strftime("%Y%m%d") + + +def get_stream_number(d: date) -> str: + """Same logic as merra2.py.""" + y = d.year + if 1980 <= y <= 1991: + return "100" + elif 1992 <= y <= 2000: + return "200" + elif 2001 <= y <= 2010: + return "300" else: - dev = torch.device(device) - - # ---- user params ---- - yyyymmdd = date.replace("-", "") - outdir_path = Path(outdir) - outdir_path.mkdir(parents=True, exist_ok=True) - - # NOTE: give a default time_range if not provided - if time_range is None: - # TODO: choose based on `date` if you want - time_range = ("2024-01-01T00:00:00", "2024-01-01T18:00:00") - - # ---- your variable lists (copied from your file) ---- - surface_vars = [ - "EFLUX","GWETROOT","HFLUX","LAI","LWGAB","LWGEM","LWTUP","PS","QV2M","SLP", - "SWGNT","SWTNT","T2M","TQI","TQL","TQV","TS","U10M","V10M","Z0M", - ] - static_surface_vars = ["FRACI", "FRLAND", "FROCEAN", "PHIS"] - vertical_vars = ["CLOUD", "H", "OMEGA", "PL", "QI", "QL", "QV", "T", "U", "V"] - levels = [34.0,39.0,41.0,43.0,44.0,45.0,48.0,51.0,53.0,56.0,63.0,68.0,71.0,72.0] - padding = {"level": [0, 0], "lat": [0, -1], "lon": [0, 0]} - - variable_names = surface_vars + [f"{v}_level_{lv}" for v in vertical_vars for lv in levels] - - lead_time = 12 - input_time = -6 - positional_encoding = "fourier" - - # ---- paths (keep your relative layout, but do not assume cwd) ---- - # If this file sits inside pyhazards/, resolve relative to repo root if needed. - # Here: assume user runs from repo root. - surf_dir = Path("data/merra-2") - vert_dir = Path("data/merra-2") - clim_dir = Path("data/climatology") - weights_path = Path("data/weights/prithvi.wxc.rollout.2300m.v1.pt") - config_path = Path("data/config.yaml") - - # ---- Prithvi-WxC imports (also inside function) ---- - from PrithviWxC.dataloaders.merra2_rollout import Merra2RolloutDataset, preproc - from PrithviWxC.dataloaders.merra2 import input_scalers, output_scalers, static_input_scalers - from PrithviWxC.model import PrithviWxC - from PrithviWxC.rollout import rollout_iter - - dataset = Merra2RolloutDataset( - time_range=time_range, - lead_time=lead_time, - input_time=input_time, - data_path_surface=surf_dir, - data_path_vertical=vert_dir, - climatology_path_surface=clim_dir, - climatology_path_vertical=clim_dir, - surface_vars=surface_vars, - static_surface_vars=static_surface_vars, - vertical_vars=vertical_vars, - levels=levels, - positional_encoding=positional_encoding, + return "400" + + +# --------------------------------------------------------------------- +# PART 1) Download raw datasets (adapted from merra2.py, but NO hardcoded creds) +# --------------------------------------------------------------------- +PRODUCT_INFO = { + "M2I1NXASM": { + "host": "https://goldsmr4.gesdisc.eosdis.nasa.gov", + "collection": "M2I1NXASM.5.12.4", + "prefix": "inst1_2d_asm_Nx", + "has_date": True, + "data_root": "MERRA2", + }, + # Official code is M2I3NVASM, but user wants folder name M2INVASM. + "M2I3NVASM": { + "host": "https://goldsmr5.gesdisc.eosdis.nasa.gov", + "collection": "M2I3NVASM.5.12.4", + "prefix": "inst3_3d_asm_Nv", + "has_date": True, + "data_root": "MERRA2", + }, + "M2T1NXFLX": { + "host": "https://goldsmr4.gesdisc.eosdis.nasa.gov", + "collection": "M2T1NXFLX.5.12.4", + "prefix": "tavg1_2d_flx_Nx", + "has_date": True, + "data_root": "MERRA2", + }, + "M2T1NXLND": { + "host": "https://goldsmr4.gesdisc.eosdis.nasa.gov", + "collection": "M2T1NXLND.5.12.4", + "prefix": "tavg1_2d_lnd_Nx", + "has_date": True, + "data_root": "MERRA2", + }, + "M2T1NXRAD": { + "host": "https://goldsmr4.gesdisc.eosdis.nasa.gov", + "collection": "M2T1NXRAD.5.12.4", + "prefix": "tavg1_2d_rad_Nx", + "has_date": True, + "data_root": "MERRA2", + }, + "M2C0NXCTM": { + "host": "https://goldsmr4.gesdisc.eosdis.nasa.gov", + "collection": "M2C0NXCTM.5.12.4", + "prefix": "const_2d_ctm_Nx", + "has_date": False, + "filename": "MERRA2_101.const_2d_ctm_Nx.00000000.nc4", + # ✅ 常量集合通常在 MONTHLY 根目录 + "data_root": "MERRA2_MONTHLY", + # ✅ 很常见的子目录布局（你的 404 就是少了它） + "subdir": "1980", + }, +} + + +def build_file_url(product_code: str, d: date | None) -> tuple[str, str]: + """ + Build a direct HTTPS URL to the granule. + - Dated products: .../data///// + - Const products: .../data//// + """ + info = PRODUCT_INFO[product_code] + host = info["host"].rstrip("/") + collection = info["collection"].strip("/") + data_root = info.get("data_root", "MERRA2").strip("/") + base = f"{host}/data/{data_root}/{collection}" + + # const / no-date + if not info.get("has_date", True): + filename = info["filename"] + subdir = (info.get("subdir") or "").strip("/") + if subdir: + return f"{base}/{subdir}/{filename}", filename + return f"{base}/{filename}", filename + + if d is None: + raise ValueError(f"date must be provided for product {product_code}") + + stream = get_stream_number(d) + yyyy = f"{d.year:04d}" + mm = f"{d.month:02d}" + datestr = f"{d.year:04d}{d.month:02d}{d.day:02d}" + + prefix = info["prefix"] + filename = f"MERRA2_{stream}.{prefix}.{datestr}.nc4" + # ✅ MERRA2 dated granules are under /YYYY/MM/ + url = f"{base}/{yyyy}/{mm}/{filename}" + return url, filename + + +def _looks_like_html_login(resp: requests.Response) -> bool: + ctype = (resp.headers.get("Content-Type") or "").lower() + if "text/html" in ctype: + return True + # sometimes ctype is octet-stream but body is still HTML; cheap check: + head = (resp.text[:200] if resp.encoding else "") + if " str: + r = requests.get( + f"{_CMR_BASE}/collections.json", + params={"short_name": short_name, "version": version}, + timeout=30, ) - if len(dataset) == 0: - raise RuntimeError("No valid data found for the given time_range / paths.") - - surf_in_scal_path = clim_dir / "musigma_surface.nc" - vert_in_scal_path = clim_dir / "musigma_vertical.nc" - surf_out_scal_path = clim_dir / "anomaly_variance_surface.nc" - vert_out_scal_path = clim_dir / "anomaly_variance_vertical.nc" - - in_mu, in_sig = input_scalers(surface_vars, vertical_vars, levels, surf_in_scal_path, vert_in_scal_path) - out_sig = output_scalers(surface_vars, vertical_vars, levels, surf_out_scal_path, vert_out_scal_path) - static_mu, static_sig = static_input_scalers(surf_in_scal_path, static_surface_vars) - - with open(config_path, "r") as f: - config = yaml.safe_load(f) - - model = PrithviWxC( - in_channels=config["params"]["in_channels"], - input_size_time=config["params"]["input_size_time"], - in_channels_static=config["params"]["in_channels_static"], - input_scalers_mu=in_mu, - input_scalers_sigma=in_sig, - input_scalers_epsilon=config["params"]["input_scalers_epsilon"], - static_input_scalers_mu=static_mu, - static_input_scalers_sigma=static_sig, - static_input_scalers_epsilon=config["params"]["static_input_scalers_epsilon"], - output_scalers=out_sig**0.5, - n_lats_px=config["params"]["n_lats_px"], - n_lons_px=config["params"]["n_lons_px"], - patch_size_px=config["params"]["patch_size_px"], - mask_unit_size_px=config["params"]["mask_unit_size_px"], - mask_ratio_inputs=0.0, - mask_ratio_targets=0.0, - embed_dim=config["params"]["embed_dim"], - n_blocks_encoder=config["params"]["n_blocks_encoder"], - n_blocks_decoder=config["params"]["n_blocks_decoder"], - mlp_multiplier=config["params"]["mlp_multiplier"], - n_heads=config["params"]["n_heads"], - dropout=config["params"]["dropout"], - drop_path=config["params"]["drop_path"], - parameter_dropout=config["params"]["parameter_dropout"], - residual="climate", - masking_mode="both", - encoder_shifting=True, - decoder_shifting=True, - positional_encoding=positional_encoding, - checkpoint_encoder=[], - checkpoint_decoder=[], + r.raise_for_status() + entries = r.json().get("feed", {}).get("entry", []) or [] + if not entries: + raise RuntimeError(f"CMR: collection not found for {short_name} v{version}") + return entries[0]["id"] + + +def _cmr_pick_direct_data_href(collection_concept_id: str, temporal: str | None = None) -> str: + params = {"collection_concept_id": collection_concept_id, "page_size": 200} + if temporal: + params["temporal"] = temporal + + r = requests.get( + f"{_CMR_BASE}/granules.json", + params=params, + headers={"Accept": "application/json"}, + timeout=30, ) + r.raise_for_status() + entries = r.json().get("feed", {}).get("entry", []) or [] + if not entries: + raise RuntimeError("CMR: granule not found") + + def ok(h: str) -> bool: + hl = h.lower() + if not h.startswith("http"): + return False + if any(x in hl for x in ["opendap", "thredds", ".html", ".xml"]): + return False + return hl.endswith((".nc4", ".nc", ".h5")) + + # prefer links that are "data#" rel if present + for e in entries: + for link in e.get("links", []) or []: + href = link.get("href") or "" + rel = (link.get("rel") or "").lower() + if ok(href) and ("data#" in rel or "data" in rel): + return href + + # fallback: any direct-looking href + for e in entries: + for link in e.get("links", []) or []: + href = link.get("href") or "" + if ok(href): + return href + + raise RuntimeError("CMR: no direct data href") + + +def _resolve_const_ctm_urls() -> list[str]: + """ + Try multiple plausible layouts for M2C0NXCTM. + Some servers expose it as: + .../MERRA2_MONTHLY/M2C0NXCTM.../1980/ + others may differ. We'll try a short list before CMR fallback. + """ + info = PRODUCT_INFO["M2C0NXCTM"] + host = info["host"].rstrip("/") + collection = info["collection"].strip("/") + filename = info["filename"] + data_root_primary = info.get("data_root", "MERRA2").strip("/") + subdir = (info.get("subdir") or "").strip("/") + + candidates: list[str] = [] + + def add(root: str, suffix: str): + candidates.append(f"{host}/data/{root}/{collection}/{suffix}") + + # common layouts + if subdir: + add(data_root_primary, f"{subdir}/{filename}") + add(data_root_primary, f"{subdir}/01/{filename}") + add(data_root_primary, filename) + + # fallback other root + other_root = "MERRA2" if data_root_primary == "MERRA2_MONTHLY" else "MERRA2_MONTHLY" + if subdir: + add(other_root, f"{subdir}/{filename}") + add(other_root, f"{subdir}/01/{filename}") + add(other_root, filename) + + # dedupe + out: list[str] = [] + seen = set() + for u in candidates: + if u not in seen: + out.append(u) + seen.add(u) + return out + + +def _resolve_const_ctm_url_via_cmr() -> str: + """ + Resolve M2C0NXCTM direct granule URL via CMR as a robust fallback. + """ + cid = _cmr_get_collection_concept_id("M2C0NXCTM", "5.12.4") + # const file exists in 1980 year range for sure + return _cmr_pick_direct_data_href(cid, temporal="1980-01-01T00:00:00Z,1980-12-31T23:59:59Z") - state = torch.load(weights_path, map_location="cpu", weights_only=False) - if "model_state" in state: - state = state["model_state"] - model.load_state_dict(state, strict=True) - model = model.to(dev).eval() - - data = next(iter(dataset)) - batch = preproc([data], padding) - for k, v in list(batch.items()): - if isinstance(v, torch.Tensor): - batch[k] = v.to(dev) - with torch.no_grad(): - out = rollout_iter(dataset.nsteps, model, batch) +def download_raw_all(raw_base: Path, d: date, *, force: bool = False): + """ + Download required raw files for a single day + static file. + Folder layout matches user's requirement: + raw_base/M2I1NXASM/... + raw_base/M2INVASM/... + ... + raw_base/M2C0NXCTM/... + """ + # Session setup: support env creds and/or ~/.netrc + session = requests.Session() + session.trust_env = True + session.headers.update({"User-Agent": "pyhazards-merra2-inspection/1.0"}) + + username = os.getenv("EARTHDATA_USERNAME") + password = os.getenv("EARTHDATA_PASSWORD") + if username and password: + session.auth = (username, password) + else: + print("[INFO] EARTHDATA_USERNAME/PASSWORD not set. Will try ~/.netrc (machine urs.earthdata.nasa.gov).") + + # ---------- const file ---------- + const_dir = raw_base / "M2C0NXCTM" + const_dir.mkdir(parents=True, exist_ok=True) + # We always store with the canonical filename + _, const_filename = build_file_url("M2C0NXCTM", None) + const_out = const_dir / const_filename + + # First try a few deterministic URLs + last_err: Exception | None = None + for url in _resolve_const_ctm_urls(): + try: + download_file(session, url, const_out, force=force) + last_err = None + break + except Exception as e: + last_err = e + # only continue if it's a 404; otherwise fail fast + if "HTTP 404" in str(e): + continue + raise + + # If still failing, use CMR to resolve a direct href + if last_err is not None: + cmr_url = _resolve_const_ctm_url_via_cmr() + print(f"[CMR] resolved M2C0NXCTM URL: {cmr_url}") + download_file(session, cmr_url, const_out, force=force) + + # ---------- daily files ---------- + day_products = ["M2I1NXASM", "M2I3NVASM", "M2T1NXFLX", "M2T1NXLND", "M2T1NXRAD"] + + # user wants NV folder named M2INVASM + nv_folder = raw_base / "M2INVASM" + nv_folder.mkdir(parents=True, exist_ok=True) + + for prod in day_products: + url, filename = build_file_url(prod, d) + if prod == "M2I3NVASM": + out_dir = nv_folder + else: + out_dir = raw_base / prod + download_file(session, url, out_dir / filename, force=force) + + +# --------------------------------------------------------------------- +# PART 2) timefix (shared, derived from mergesfc/mergepres timefix sections) +# --------------------------------------------------------------------- +def parse_ymd_from_name(path: Path) -> tuple[int, str]: + m = re.search(r"(\d{8})", path.name) + if not m: + raise ValueError(f"Cannot find YYYYMMDD in filename: {path.name}") + return int(m.group(1)), m.group(1) + + +def timefix_one_file(path: Path): + """ + Keep the same behavior: rewrite time to int32 [0, step, 2*step,...] (minutes), + and set begin_date/begin_time/units/calendar. + """ + path = Path(path) + if not path.exists(): + raise FileNotFoundError(f"timefix target missing: {path}") + + ymd_int, ymd_str = parse_ymd_from_name(path) + + with h5py.File(path, "r+") as f: + if "time" not in f: + raise RuntimeError(f"{path}: missing 'time' variable for timefix") + + t = f["time"] + data = np.array(t[...]) + if data.size == 0: + raise RuntimeError(f"{path}: time length is 0") + + # step from old time + if data.size > 1: + step = int(data[1] - data[0]) + else: + step = 180 + if step <= 0: + step = 180 + + new_time = np.arange(0, step * data.size, step, dtype="int32") + t[...] = new_time.astype("int32") + + t.attrs["begin_date"] = np.array([ymd_int], dtype="int32") + t.attrs["begin_time"] = np.array([0], dtype="int32") + t.attrs["units"] = f"minutes since {ymd_str[0:4]}-{ymd_str[4:6]}-{ymd_str[6:8]} 00:00:00" + t.attrs["calendar"] = "proleptic_gregorian" + + +# --------------------------------------------------------------------- +# PART 3) Merge SFC (adapted from mergesfc.py; same merge logic, path parameterized) +# --------------------------------------------------------------------- +def _rename_latlon(ds: xr.Dataset) -> xr.Dataset: + rename_map = {} + if "latitude" in ds.coords and "lat" not in ds.coords: + rename_map["latitude"] = "lat" + if "longitude" in ds.coords and "lon" not in ds.coords: + rename_map["longitude"] = "lon" + if rename_map: + ds = ds.rename(rename_map) + return ds + + +def _ensure_order(ds: xr.Dataset) -> xr.Dataset: + if "lat" in ds.coords and ds["lat"].size > 1: + lat = ds["lat"].values + if np.nanmean(np.diff(lat)) < 0: + ds = ds.sortby("lat") + if "lon" in ds.coords and ds["lon"].size > 1: + lon = ds["lon"].values + if np.nanmean(np.diff(lon)) < 0: + ds = ds.sortby("lon") + return ds + + +def _coerce_to_shape(ds: xr.Dataset, target_lat=TARGET_LAT, target_lon=TARGET_LON) -> xr.Dataset: + if "lat" in ds.dims and ds.dims["lat"] != target_lat: + raise ValueError(f"lat size mismatch: {ds.dims['lat']} != {target_lat}") + if "lon" in ds.dims and ds.dims["lon"] != target_lon: + raise ValueError(f"lon size mismatch: {ds.dims['lon']} != {target_lon}") + return ds + + +def _select_8_from_24(da: xr.DataArray): + return da.isel(time=[0, 3, 6, 9, 12, 15, 18, 21]) + + +def _repeat_to_n(da: xr.DataArray, n: int = TARGET_N_FRAMES) -> xr.DataArray: + if "time" not in da.dims: + tiles = [da.expand_dims(time=[i]) for i in range(n)] + return xr.concat(tiles, dim="time") + T = da.sizes["time"] + rep = int(np.ceil(n / T)) + out = xr.concat([da] * rep, dim="time").isel(time=slice(0, n)) + return out + + +def _average_into_n(da: xr.DataArray, n: int = TARGET_N_FRAMES) -> xr.DataArray: + T = da.sizes["time"] + splits = np.array_split(np.arange(T), n) + tiles = [da.isel(time=idx).mean(dim="time") for idx in splits] + return xr.concat(tiles, dim="time") + + +def _to_8_frames_data(da: xr.DataArray) -> xr.DataArray: + if "time" not in da.dims: + return _repeat_to_n(da, TARGET_N_FRAMES) + T = da.sizes["time"] + if T == TARGET_N_FRAMES: + return da + if T == 24: + return _select_8_from_24(da) + if T == 1: + return _repeat_to_n(da, TARGET_N_FRAMES) + if T in (2, 4): + return _repeat_to_n(da, TARGET_N_FRAMES) + if T > TARGET_N_FRAMES: + return _average_into_n(da, TARGET_N_FRAMES) + return _repeat_to_n(da, TARGET_N_FRAMES) + + +def _make_time8_from_base_time(base_time: xr.DataArray) -> xr.DataArray: + t = pd.to_datetime(base_time.values) + if t.size >= 24: + idx = [0, 3, 6, 9, 12, 15, 18, 21] + t8 = t[idx] + elif t.size == 8: + t8 = t + else: + t0 = pd.Timestamp(t[0]).normalize() if t.size > 0 else pd.Timestamp("2000-01-01") + t8 = [t0 + pd.Timedelta(hours=h) for h in HOURS8] + return xr.DataArray(np.array(t8, dtype="datetime64[ns]"), dims=["time"], name="time") - # ---- write a small set of outputs ---- - important_surface = ["T2M","QV2M","TQV","U10M","V10M","GWETROOT","TS","LAI","EFLUX","HFLUX","SWGNT","SWTNT","LWGAB","LWGEM"] - out_np = out.detach().cpu().numpy().astype("float32") - T, C, H, W = out_np.shape +def _align_and_put(ds_out: xr.Dataset, name: str, da: xr.DataArray): + if "lev" in da.dims: + da = da.isel(lev=0) - time_array = np.arange(T) - lat = np.linspace(-90, 90, H) - lon = np.linspace(-180, 180, W) + da = da.transpose(*[d for d in da.dims if d in ["time", "lat", "lon"]]) + da8 = _to_8_frames_data(da) - data_vars = {} - for name in important_surface: - if name not in variable_names: + if set(["lat", "lon"]).issubset(set(da8.dims)): + order = ("time", "lat", "lon") if "time" in da8.dims else ("lat", "lon") + da8 = da8.transpose(*order) + ds_out[name] = xr.DataArray(da8.values, dims=order, attrs=da.attrs) + else: + ds_out[name] = xr.DataArray(da8.values, dims=da8.dims, attrs=da.attrs) + + +def _load_and_prepare(path: Path) -> xr.Dataset: + print(f"[DEBUG] open_dataset: {path}") + last_err = None + for engine in [None, "netcdf4", "h5netcdf"]: + try: + print(f" try engine={engine}") + ds = xr.open_dataset(path, engine=engine) if engine else xr.open_dataset(path) + ds = _rename_latlon(ds) + ds = _ensure_order(ds) + ds = _coerce_to_shape(ds, TARGET_LAT, TARGET_LON) + print(f" OK with engine={engine}") + return ds + except Exception as e: + print(f" engine={engine} FAILED:", e) + last_err = e + raise last_err + + +def _squeeze_static_2d(da: xr.DataArray, name: str) -> xr.DataArray: + if "time" in da.dims: + print(f"[STATIC] {name}: has time dim={da.sizes['time']}, taking time=0") + da = da.isel(time=0, drop=True) + if "lev" in da.dims: + print(f"[STATIC] {name}: has lev dim={da.sizes['lev']}, taking lev=0") + da = da.isel(lev=0, drop=True) + + if not {"lat", "lon"}.issubset(da.dims): + raise ValueError(f"STATIC {name} missing lat/lon dims: dims={da.dims}") + + da2 = da.transpose("lat", "lon") + if da2.sizes["lat"] != TARGET_LAT or da2.sizes["lon"] != TARGET_LON: + raise ValueError(f"STATIC {name} size not (361,576): {da2.sizes}") + return da2 + + +def _merge_global_attrs(*datasets) -> dict: + merged = {} + for ds in datasets: + if ds is None: continue - ch = variable_names.index(name) - data_vars[name] = xr.DataArray(out_np[:, ch, :, :], coords={"time": time_array, "lat": lat, "lon": lon}, dims=("time","lat","lon")) + for k, v in ds.attrs.items(): + if k not in merged: + merged[k] = v + return merged + + +def _raw_paths_for_day(raw_base: Path, d: date) -> dict[str, Path]: + datestr = yyyymmdd(d) + stream = get_stream_number(d) + + # note: NV folder is M2INVASM (user requirement) + return { + "ASM1": raw_base / "M2I1NXASM" / f"MERRA2_{stream}.inst1_2d_asm_Nx.{datestr}.nc4", + "FLX": raw_base / "M2T1NXFLX" / f"MERRA2_{stream}.tavg1_2d_flx_Nx.{datestr}.nc4", + "RAD": raw_base / "M2T1NXRAD" / f"MERRA2_{stream}.tavg1_2d_rad_Nx.{datestr}.nc4", + "NV": raw_base / "M2INVASM" / f"MERRA2_{stream}.inst3_3d_asm_Nv.{datestr}.nc4", + "LND": raw_base / "M2T1NXLND" / f"MERRA2_{stream}.tavg1_2d_lnd_Nx.{datestr}.nc4", + "STATIC": raw_base / "M2C0NXCTM" / "MERRA2_101.const_2d_ctm_Nx.00000000.nc4", + } + + +def merge_sfc(raw_base: Path, merged_dir: Path, d: date) -> Path: + paths = _raw_paths_for_day(raw_base, d) + for k, p in paths.items(): + if not p.exists(): + raise FileNotFoundError(f"Missing raw file for {k}: {p}") + + ds_asm1 = _load_and_prepare(paths["ASM1"]) + ds_flx = _load_and_prepare(paths["FLX"]) + ds_rad = _load_and_prepare(paths["RAD"]) + ds_nv = _load_and_prepare(paths["NV"]) + ds_lnd = _load_and_prepare(paths["LND"]) + ds_static = _load_and_prepare(paths["STATIC"]) + + keep = [v for v in ASM1_VARS if v in ds_asm1] + if not keep: + raise RuntimeError("ASM1 has none of requested variables.") + asm1_vars_8 = {v: _to_8_frames_data(ds_asm1[v]) for v in keep} + + if "time" in ds_asm1.coords and ds_asm1.sizes.get("time", 0) > 0: + time8 = _make_time8_from_base_time(ds_asm1["time"]) + else: + t0 = pd.Timestamp("2000-01-01") + time8 = xr.DataArray( + np.array([t0 + pd.Timedelta(hours=h) for h in HOURS8], dtype="datetime64[ns]"), + dims=["time"], name="time" + ) + + base = xr.DataArray( + np.empty((TARGET_N_FRAMES, TARGET_LAT, TARGET_LON), dtype="float32"), + dims=("time", "lat", "lon") + ).assign_coords( + time=time8, + lat=ds_asm1["lat"] if "lat" in ds_asm1.coords else np.arange(TARGET_LAT), + lon=ds_asm1["lon"] if "lon" in ds_asm1.coords else np.arange(TARGET_LON), + ) + + ds_out = xr.Dataset(coords={"time": base["time"], "lat": base["lat"], "lon": base["lon"]}) + + for v, da8 in asm1_vars_8.items(): + ds_out[v] = xr.DataArray(da8.values, dims=("time", "lat", "lon"), attrs=ds_asm1[v].attrs) + + for v in FLX_VARS: + if v not in ds_flx: + raise RuntimeError(f"FLX missing var: {v}") + _align_and_put(ds_out, v, ds_flx[v]) - ds = xr.Dataset(data_vars) + for v in RAD_VARS: + if v not in ds_rad: + raise RuntimeError(f"RAD missing var: {v}") + _align_and_put(ds_out, v, ds_rad[v]) - out_path = outdir_path / f"pred_{yyyymmdd}.nc" - if out_path.exists(): - os.remove(out_path) + for v in NV_VARS: + if v not in ds_nv: + print(f"[WARN] NV missing var: {v}; skip") + continue + _align_and_put(ds_out, v, ds_nv[v]) - ds.load() - ds.to_netcdf(out_path, mode="w", engine="scipy") + for v in LND_VARS: + if v not in ds_lnd: + print(f"[WARN] LND missing var: {v}; skip") + continue + _align_and_put(ds_out, v, ds_lnd[v]) + + for v in STATIC_VARS: + if v not in ds_static: + raise RuntimeError(f"STATIC missing var: {v}") + da2 = _squeeze_static_2d(ds_static[v], v) + ds_out[v] = xr.DataArray(da2.values, dims=("lat", "lon"), attrs=ds_static[v].attrs) + + ds_out.attrs = _merge_global_attrs(ds_asm1, ds_flx, ds_rad, ds_nv, ds_lnd, ds_static) + + # NaN fill (same intent as mergesfc.py) + if NAN_VALS: + for var in ds_out.data_vars: + if var in NAN_VALS: + nan_val = NAN_VALS[var] + ds_out[var].data[:] = np.nan_to_num(ds_out[var].data, nan=nan_val) + + # time encoding + timefix afterwards (same structure as mergesfc.py) + t0 = ds_out["time"].values[0] + ds_out["time"] = ((ds_out["time"].values - t0).astype("timedelta64[m]").astype("int32")) + ds_out.time.attrs = {"begin_time": 0, "begin_date": int(yyyymmdd(d))} + + encoding = {name: {"zlib": True} for name in ds_out.data_vars} + encoding["time"] = {"dtype": "int32"} + + merged_dir.mkdir(parents=True, exist_ok=True) + out_path = merged_dir / PATTERN_SFC_OUT.format(yyyymmdd=yyyymmdd(d)) + + try: + print("[INFO] Writing SFC with engine=h5netcdf...") + ds_out.to_netcdf(out_path, encoding=encoding, engine="h5netcdf") + except Exception as e: + print("[WARN] h5netcdf failed:", e) + print("[INFO] Falling back to engine=netcdf4") + ds_out.to_netcdf(out_path, encoding=encoding, engine="netcdf4") + + timefix_one_file(out_path) + print(f"[OK] Wrote SFC: {out_path}") return out_path -def _build_argparser() -> argparse.ArgumentParser: - p = argparse.ArgumentParser() - p.add_argument("--date", required=True, help="YYYY-MM-DD") - p.add_argument("--outdir", required=True) - p.add_argument("--device", default=None, help='e.g., "cuda", "cuda:0", "cpu"') +# --------------------------------------------------------------------- +# PART 4) Merge PRES (adapted from mergepres.py; same merge logic, path parameterized) +# --------------------------------------------------------------------- +def _fmt_date_any(ts): + ts_pd = pd.to_datetime(ts) + if ts_pd.tzinfo is None: + ts_pd = ts_pd.tz_localize(timezone.utc) + else: + ts_pd = ts_pd.tz_convert(timezone.utc) + return ts_pd.strftime("%Y-%m-%d"), ts_pd.strftime("%H:%M:%S.%f") + + +def _open_nv_any_engine(path: Path) -> xr.Dataset: + print(f"[DEBUG] open_dataset: {path}") + last_err = None + for engine in [None, "netcdf4", "h5netcdf"]: + try: + print(f" try engine={engine}") + ds = xr.open_dataset(path, engine=engine) if engine else xr.open_dataset(path) + print(f" OK with engine={engine}") + return ds + except Exception as e: + print(f" engine={engine} FAILED:", e) + last_err = e + raise last_err + + +def _write_any_engine(ds: xr.Dataset, out_path: Path): + engines = [("h5netcdf", True), ("netcdf4", True), ("scipy", False)] + last_err = None + out_path.parent.mkdir(parents=True, exist_ok=True) + + for engine, can_compress in engines: + try: + print(f"[INFO] Writing to {out_path} (engine={engine}, compression={'on' if can_compress else 'off'})") + if can_compress: + comp = dict(zlib=True, complevel=1, shuffle=True) + encoding = {v: comp for v in ds.data_vars} + ds.to_netcdf(out_path, engine=engine, mode="w", encoding=encoding) + else: + ds.to_netcdf(out_path, engine=engine, mode="w") + print(f"[INFO] Write OK with engine={engine}") + return + except Exception as e: + print(f"[WARN] engine={engine} FAILED:", e) + last_err = e + raise last_err + + +def merge_pres(raw_base: Path, merged_dir: Path, d: date) -> Path: + datestr = yyyymmdd(d) + stream = get_stream_number(d) + in_vert = raw_base / "M2INVASM" / f"MERRA2_{stream}.inst3_3d_asm_Nv.{datestr}.nc4" + if not in_vert.exists(): + raise FileNotFoundError(f"Missing NV raw file for PRES merge: {in_vert}") + + merged_dir.mkdir(parents=True, exist_ok=True) + out_vert = merged_dir / PATTERN_PRES_OUT.format(yyyymmdd=datestr) + + ds = _open_nv_any_engine(in_vert) + + keep_vars = [v for v in PRES_VARS if v in ds.data_vars] + if not keep_vars: + raise ValueError(f"No target vars found. Wanted: {PRES_VARS}") + ds = ds[keep_vars] + + if "lev" not in ds.coords: + raise KeyError("Missing 'lev' coordinate in NV file.") + + lev_vals = np.array(ds["lev"].values, dtype=float).tolist() + missing = [lv for lv in PRES_LEVELS if lv not in lev_vals] + if missing: + raise ValueError(f"Missing levels: {missing}\nAvailable lev: {lev_vals}") + + ds = ds.sel(lev=xr.DataArray(PRES_LEVELS, dims="lev")) + ds = ds.sortby("lev", ascending=False) + + attrs = dict(ds.attrs) if ds.attrs else {} + if "time" in ds.coords and ds.sizes.get("time", 0) > 0: + tvals = ds["time"].values + tmin, tmax = tvals.min(), tvals.max() + beg_date, beg_time = _fmt_date_any(tmin) + end_date, end_time = _fmt_date_any(tmax) + attrs.update({ + "RangeBeginningDate": beg_date, + "RangeBeginningTime": beg_time, + "RangeEndingDate": end_date, + "RangeEndingTime": end_time, + }) + ds = ds.assign_attrs(attrs) + + _write_any_engine(ds, out_vert) + ds.close() + + timefix_one_file(out_vert) + + # quick check + try: + ds2 = xr.open_dataset(out_vert, engine="h5netcdf") + print(f"[OK] Wrote PRES: {out_vert}") + print("vars:", list(ds2.data_vars)) + print("shape:", {k: int(v) for k, v in ds2.sizes.items()}) + ds2.close() + except Exception as e: + print("[WARN] PRES written & timefixed, but reopen check failed:", e) + + return out_vert + + +# --------------------------------------------------------------------- +# PART 5) Inspection (SFC/PRES merged products) +# --------------------------------------------------------------------- +def list_vars(ds: xr.Dataset, max_show: int = 60) -> pd.DataFrame: + rows = [] + for name, da in ds.data_vars.items(): + rows.append({ + "var": name, + "dims": str(da.dims), + "shape": str(tuple(da.shape)), + "dtype": str(da.dtype), + }) + df = pd.DataFrame(rows).sort_values("var").reset_index(drop=True) + return df.head(max_show) if len(df) > max_show else df + + +def inspect_ds(ds: xr.Dataset, name: str, max_vars: int = 60): + print(f"\n=== {name} ===") + print("dims:", dict(ds.sizes)) + print("coords:", list(ds.coords)) + print("n_vars:", len(ds.data_vars)) + display(list_vars(ds, max_show=max_vars)) + + +def summarize_da(da: xr.DataArray) -> pd.Series: + s = xr.Dataset({ + "min": da.min(skipna=True), + "max": da.max(skipna=True), + "mean": da.mean(skipna=True), + "std": da.std(skipna=True), + }).compute() + return pd.Series({k: float(s[k].values) for k in s.data_vars}) + + +def run_inspection(merged_dir: Path, outdir: Path, d: date, var: str = "T2M"): + sfc_path = merged_dir / PATTERN_SFC_OUT.format(yyyymmdd=yyyymmdd(d)) + pres_path = merged_dir / PATTERN_PRES_OUT.format(yyyymmdd=yyyymmdd(d)) + + ds_sfc = xr.open_dataset(sfc_path, engine="h5netcdf") + ds_pres = xr.open_dataset(pres_path, engine="h5netcdf") + + inspect_ds(ds_sfc, "SFC (one day)") + inspect_ds(ds_pres, "PRES (one day)") + + # save var tables + outdir.mkdir(parents=True, exist_ok=True) + list_vars(ds_sfc).to_csv(outdir / f"sfc_vars_{d.isoformat()}.csv", index=False) + list_vars(ds_pres).to_csv(outdir / f"pres_vars_{d.isoformat()}.csv", index=False) + + if var not in ds_sfc: + raise KeyError(f"{var} not found in SFC merged file. See vars csv in {outdir}") + + da = ds_sfc[var] + print(f"\n{var} dims :", da.dims) + print(f"{var} shape:", da.shape) + print(f"\n{var} summary:") + print(summarize_da(da)) + + # plot one timestep + t = 0 + Z = da.isel(time=t).compute() + plt.figure() + plt.contourf(ds_sfc["lon"], ds_sfc["lat"], Z, 100) + plt.gca().set_aspect("equal") + plt.title(f"{var} (t={t})") + + out_pdf = outdir / f"{var.lower()}_{d.isoformat()}.pdf" + plt.savefig(out_pdf) + print(f"\nSaved: {out_pdf}") + + ds_sfc.close() + ds_pres.close() + + +# --------------------------------------------------------------------- +# CLI +# --------------------------------------------------------------------- +def build_parser() -> argparse.ArgumentParser: + p = argparse.ArgumentParser( + prog="python -m pyhazards.datasets.inspection", + description="One-shot pipeline: download raw MERRA-2 -> merge SFC+PRES -> inspection.", + ) + + # ✅ positional date + p.add_argument( + "date", + help="Date to run. Accepts YYYYMMDD (e.g., 20251111) or YYYY-MM-DD (e.g., 2025-11-11).", + ) + + # ✅ default outputs (relative to repo root unless absolute) + p.add_argument("--outdir", default="outputs", help="Output directory (default: outputs under repo root)") + + # ✅ auto infer repo root if omitted + p.add_argument("--repo-root", default=None, help="Repo root (auto-infer if omitted)") + p.add_argument("--raw-base", default=None, help="Raw datasets base dir (default: REPO_ROOT)") + p.add_argument("--merged-dir", default=None, help="Merged dir (default: REPO_ROOT/Prithvi-WxC/data/merra-2)") + + p.add_argument("--skip-download", action="store_true", help="Assume raw files already exist") + p.add_argument("--skip-merge", action="store_true", help="Assume merged nc already exist") + p.add_argument("--force-download", action="store_true", help="Force re-download even if file exists") + p.add_argument("--var", default="T2M", help="Variable to summarize/plot from SFC") + return p -def main() -> None: - args = _build_argparser().parse_args() - out_path = inspect_wxc_rollout(args.date, args.outdir, device=args.device) - print(f"[OK] wrote: {out_path}") +def main(argv=None) -> int: + args = build_parser().parse_args(argv) + + s = args.date.strip() + if len(s) == 8 and s.isdigit(): + d = date(int(s[:4]), int(s[4:6]), int(s[6:8])) + else: + d = date.fromisoformat(s) + + repo_root = Path(args.repo_root).expanduser().resolve() if args.repo_root else infer_repo_root() + raw_base = Path(args.raw_base).expanduser().resolve() if args.raw_base else repo_root + merged_dir = Path(args.merged_dir).expanduser().resolve() if args.merged_dir else (repo_root / "Prithvi-WxC" / "data" / "merra-2") + + # outdir: relative -> repo_root/outdir; absolute -> keep + outdir = (repo_root / args.outdir) if not Path(args.outdir).is_absolute() else Path(args.outdir) + outdir = outdir.expanduser().resolve() + + print("repo_root :", repo_root) + print("raw_base :", raw_base) + print("merged_dir:", merged_dir) + print("outdir :", outdir) + print("date :", d) + + raw_base.mkdir(parents=True, exist_ok=True) + merged_dir.mkdir(parents=True, exist_ok=True) + outdir.mkdir(parents=True, exist_ok=True) + + if not args.skip_download: + print("\n=== STEP 1: Download raw datasets ===") + download_raw_all(raw_base, d, force=args.force_download) + + if not args.skip_merge: + print("\n=== STEP 2: Merge SFC ===") + merge_sfc(raw_base, merged_dir, d) + + print("\n=== STEP 3: Merge PRES ===") + merge_pres(raw_base, merged_dir, d) + + print("\n=== STEP 4: Inspection ===") + run_inspection(merged_dir, outdir, d, var=args.var) + + print("\n[DONE] Full pipeline finished.") + return 0 if __name__ == "__main__": - main() + raise SystemExit(main()) diff --git a/pyhazards/datasets/outputs/t2m_pred.pdf b/pyhazards/datasets/outputs/t2m_pred.pdf new file mode 100644 index 00000000..88fbb3f0 Binary files /dev/null and b/pyhazards/datasets/outputs/t2m_pred.pdf differ