Skip to content

Add functionality to allow users to download smaller data subset (Closes #4) #5

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
leahgibson wants to merge 7 commits into
base: main
Choose a base branch
from
Open

Add functionality to allow users to download smaller data subset (Closes #4) #5

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
7 commits
Select commit Hold shift + click to select a range
e4d7995
add functionality for subselection of data based on variable and time
leahgibson Feb 12, 2025
0c79682
bug fixes for subselection
leahgibson Feb 12, 2025
67e1a57
updating readme to reflect additional options for downloading data
leahgibson Feb 12, 2025
05b5427
clarifications to readme
leahgibson Feb 13, 2025
3476ab6
edits to easily work with tests and final bug fixes
leahgibson Feb 13, 2025
16de513
upload test file for edits to download_gauss.sh
leahgibson Feb 13, 2025
34e7a15
edits to README instructions for data download
leahgibson Feb 13, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 8 additions & 2 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ pip install -r requirements.txt
```

### B. Data Download
Download the GAUSS data from Globus (requires around 140GB of disk space):
Download the GAUSS data from Globus (requires up to 140GB of disk space if all data is downloaded):
1. First install the [Globus CLI](https://docs.globus.org/cli/):
```bash
pip install globus-cli
Expand All @@ -39,10 +39,16 @@ Download the GAUSS data from Globus (requires around 140GB of disk space):
```bash
globus login
```
3. Download the data:
3. Download the data. You can download all data (~140GB) using:
```bash
sh scripts/download_gauss.sh 2e01e83a-5180-47f7-a6ab-c98b626ad9e4 <YOUR ENDPOINT ID> data/gauss/
```

Or you can select a subset of the data to download (e.g. `tas` and `monthly`) using:
```bash
sh scripts/download_gauss.sh 2e01e83a-5180-47f7-a6ab-c98b626ad9e4 <YOUR ENDPOINT ID> data/gauss/ --tas --monthly
```
If only a subset of the data is downloaded, it will affect which of the following steps can successfully run. To fit the regression models, the monthly temperature (--tas --monthly) data must be downloaded at a minimum. At this time, it is only possible to subselect temperature (--tas) or precipitation (--pr) data.

### C. Data Processing
1. Process the daily data to create monthly values. You can process all the daily data (both `tas` and `pr`) using:
Expand Down
229 changes: 155 additions & 74 deletions scripts/download_gauss.sh
100644 → 100755
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Argument validation check
if [ "$#" -ne 3 ]; then
echo "Usage: $0 <source_endpoint_id> <destination_endpoint_id> <local_destination>"
if [ "$#" -lt 3 ] || [ "$#" -gt 5 ]; then
echo "Usage: $0 <source_endpoint_id> <destination_endpoint_id> <local_destination> [--tas | --pr] [--daily | --monthly]"
exit 1
fi

Expand All @@ -9,6 +9,51 @@ source_endpoint_id=$1
destination_endpoint_id=$2
local_destination=$3

# Initialize optional variables
download_tas=false
download_pr=false
download_daily=false
download_monthly=false

# Check for variable flags
for arg in "${@:4}"
do
case $arg in
--tas)
if [ "$download_pr" = true ]; then
echo "Error: You cannot select both ---tas and --pr."
exit 1
fi
download_tas=true
;;
--pr)
if [ "$download_tas" = true ]; then
echo "Error: You cannot select both --tas and --pr."
exit 1
fi
download_pr=true
;;
--daily)
if [ "$download_monthly" = true ]; then
echo "Error: You cannot select both --daily and --monthly."
exit 1
fi
download_daily=true
;;
--monthly)
if [ "$download_daily" = true ]; then
echo "Error: You cannot select both --daily and --monthly."
exit 1
fi
download_monthly=true
;;
*)
echo "Invalid option: $arg"
exit 1
;;
esac
done

# Create the local destination directory if it does not exist
mkdir -p $local_destination

Expand All @@ -23,85 +68,121 @@ do
base_dirs+=("/SSP245-MA-GAUSS-LOWER-1.0.00$i/atm/proc/tseries/month_1")
done

# Loop through each base directory and download the files
for dir in "${base_dirs[@]}"
do
echo "Listing files in $dir..."
# List files in the directory and filter for specific patterns
files=$(globus ls $source_endpoint_id:$dir | grep -E "\.TREFHT\.|\.\PRECT\.|\.TREFHTMN\.|\.TREFHTMX\.|\.QFLX\." | grep -E "\.h0\.")
# Download monthly files as long as user disn't request just daily
if [ "$download_daily" = false ]; then
# Define the file pattern based on the user's selection
file_pattern=""
if [ "$download_tas" = true ]; then
file_pattern="\.TREFHT\."
fi
if [ "$download_pr" = true ]; then
file_pattern="\.\PRECT\."
fi
if [ "$download_tas" = false ] && [ "$download_pr" = false ]; then
file_pattern="\.TREFHT\.|\.\PRECT\.|\.TREFHTMN\.|\.TREFHTMX\.|\.QFLX\."
fi

# Loop through filtered files and initiate transfer
for file in $files
# Loop through each base directory and download the files
for dir in "${base_dirs[@]}"
do
# Check if file already exists at destination
if globus ls $destination_endpoint_id:$local_destination | grep -q "^$file$"; then
echo "Skipping $file, already exists at destination."
else
echo "Transferring $file from $dir..."
globus transfer $source_endpoint_id:$dir/$file $destination_endpoint_id:$local_destination$file
fi
done
done
echo "Listing files in $dir... with file pattern $file_pattern"
# List files in the directory and filter for specific patterns
files=$(globus ls $source_endpoint_id:$dir | grep -E "$file_pattern" | grep -E "\.h0\.")

# For MA-BASELINE.002 and MA-BASELINE.003, have to download PRECC and PRECL instead of PRECT
for i in {2..3}
do
echo "Listing files in /MA-BASELINE.00$i/atm/proc/tseries/month_1..."
files=$(globus ls $source_endpoint_id:/MA-BASELINE.00$i/atm/proc/tseries/month_1 | grep -E "\.PRECL\.|\.\PRECC\." | grep -E "\.h0\.")
for file in $files
do
if globus ls $destination_endpoint_id:$local_destination | grep -q "^$file$"; then
echo "Skipping $file, already exists at destination."
else
echo "Transferring $file from /MA-BASELINE.00$i/atm/proc/tseries/month_1..."
globus transfer $source_endpoint_id:/MA-BASELINE.00$i/atm/proc/tseries/month_1/$file $destination_endpoint_id:$local_destination$file
fi
# Loop through filtered files and initiate transfer
for file in $files
do
# Check if file already exists at destination
if globus ls $destination_endpoint_id:$local_destination | grep -q "^$file$"; then
echo "Skipping $file, already exists at destination."
else
echo "Transferring $file from $dir..."
globus transfer $source_endpoint_id:$dir/$file $destination_endpoint_id:$local_destination$file
fi
done
done
done

# For MA-HISTORICAL, have to download PRECC and PRECL insrtead of PRECT
for i in {1..3}
do
echo "Listing files in /MA-HISTORICAL.00$i/atm/proc/tseries/month_1..."
files=$(globus ls $source_endpoint_id:/MA-HISTORICAL.00$i/atm/proc/tseries/month_1 | grep -E "\.PRECL\.|\.\PRECC\." | grep -E "\.h0\.")
for file in $files
do
if globus ls $destination_endpoint_id:$local_destination | grep -q "^$file$"; then
echo "Skipping $file, already exists at destination."
else
echo "Transferring $file from /MA-HISTORICAL.00$i/atm/proc/tseries/month_1..."
globus transfer $source_endpoint_id:/MA-HISTORICAL.00$i/atm/proc/tseries/month_1/$file $destination_endpoint_id:$local_destination$file
fi
done
done
# Only download additional variables if not only downloading tas
if [ "$download_tas" = false ]; then
# For MA-BASELINE.002 and MA-BASELINE.003, have to download PRECC and PRECL instead of PRECT
for i in {2..3}
do
echo "Listing files in /MA-BASELINE.00$i/atm/proc/tseries/month_1... with file pattern \.PRECL\.|\.\PRECC\."
files=$(globus ls $source_endpoint_id:/MA-BASELINE.00$i/atm/proc/tseries/month_1 | grep -E "\.PRECL\.|\.\PRECC\." | grep -E "\.h0\.")
for file in $files
do
if globus ls $destination_endpoint_id:$local_destination | grep -q "^$file$"; then
echo "Skipping $file, already exists at destination."
else
echo "Transferring $file from /MA-BASELINE.00$i/atm/proc/tseries/month_1..."
globus transfer $source_endpoint_id:/MA-BASELINE.00$i/atm/proc/tseries/month_1/$file $destination_endpoint_id:$local_destination$file
fi
done
done

# Download daily TREFHT and PRECT files for base_dirs
for dir in "${base_dirs[@]}"
do
# Replace month with day
dir=$(echo $dir | sed 's/month_1/day_1/g')
echo "Listing files in $dir..."
files=$(globus ls $source_endpoint_id:$dir | grep -E "\.TREFHT\.|\.PRECT" | grep -E "\.h1\.")
for file in $files
# For MA-HISTORICAL, have to download PRECC and PRECL insrtead of PRECT
for i in {1..3}
do
echo "Listing files in /MA-HISTORICAL.00$i/atm/proc/tseries/month_1... with file pattern \.PRECL\.|\.\PRECC\."
files=$(globus ls $source_endpoint_id:/MA-HISTORICAL.00$i/atm/proc/tseries/month_1 | grep -E "\.PRECL\.|\.\PRECC\." | grep -E "\.h0\.")
for file in $files
do
if globus ls $destination_endpoint_id:$local_destination | grep -q "^$file$"; then
echo "Skipping $file, already exists at destination."
else
echo "Transferring $file from /MA-HISTORICAL.00$i/atm/proc/tseries/month_1..."
globus transfer $source_endpoint_id:/MA-HISTORICAL.00$i/atm/proc/tseries/month_1/$file $destination_endpoint_id:$local_destination$file
fi
done
done
fi
fi

# Download daily data as long as user didn't only request monthly data
if [ "$download_monthly" = false ]; then
# Define the file pattern based on the user's selection
file_pattern=""
if [ "$download_tas" = true ]; then
file_pattern="\.TREFHT\."
fi
if [ "$download_pr" = true ]; then
file_pattern="\.PRECT"
fi
if [ "$download_tas" = false ] && [ "$download_pr" = false ]; then
file_pattern="\.TREFHT\.|\.PRECT"
fi

# Download daily TREFHT and/or PRECT files for base_dirs
for dir in "${base_dirs[@]}"
do
if globus ls $destination_endpoint_id:$local_destination | grep -q "^$file$"; then
echo "Skipping $file, already exists at destination."
else
echo "Transferring $file from $dir..."
globus transfer $source_endpoint_id:$dir/$file $destination_endpoint_id:$local_destination$file
fi
# Replace month with day
dir=$(echo $dir | sed 's/month_1/day_1/g')
echo "Listing files in $dir... with file pattern $file_pattern"
files=$(globus ls $source_endpoint_id:$dir | grep -E "$file_pattern" | grep -E "\.h1\.")
for file in $files
do
if globus ls $destination_endpoint_id:$local_destination | grep -q "^$file$"; then
echo "Skipping $file, already exists at destination."
else
echo "Transferring $file from $dir..."
globus transfer $source_endpoint_id:$dir/$file $destination_endpoint_id:$local_destination$file
fi
done
done
done

# Download daily PRECT from ARISE-HISTORICAL
echo "Listing files in /ARISE-HISTORICAL"
files=$(globus ls $source_endpoint_id:/ARISE-HISTORICAL | grep -E "\.PRECT\." | grep -E "\.h1\.")
for file in $files
do
if globus ls $destination_endpoint_id:$local_destination | grep -q "^$file$"; then
echo "Skipping $file, already exists at destination."
else
echo "Transferring $file from /ARISE-HISTORICAL..."
globus transfer $source_endpoint_id:/ARISE-HISTORICAL/$file $destination_endpoint_id:$local_destination$file
# Only download if not just downloading tas
if [ "$download_tas" = false ]; then
# Download daily PRECT from ARISE-HISTORICAL
echo "Listing files in /ARISE-HISTORICAL"
files=$(globus ls $source_endpoint_id:/ARISE-HISTORICAL | grep -E "\.PRECT\." | grep -E "\.h1\.")
for file in $files
do
if globus ls $destination_endpoint_id:$local_destination | grep -q "^$file$"; then
echo "Skipping $file, already exists at destination."
else
echo "Transferring $file from /ARISE-HISTORICAL..."
globus transfer $source_endpoint_id:/ARISE-HISTORICAL/$file $destination_endpoint_id:$local_destination$file
fi
done
fi
done
fi
Loading