Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/Data_Transfer/Globus/.pages.yml
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ nav:
- Add_Your_Computer_To_Globus.md
- Data_Transfer_using_Globus.md
- Data_Transfer_Between_Personal_Endpoints.md
- Data_Transfer_to_Freezer.md
- Signing_Up_To_Globus_Subscription.md
- Bookmarks.md
- Share_Collections.md
Expand Down
25 changes: 14 additions & 11 deletions docs/Data_Transfer/Globus/Data_Transfer_to_Freezer.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,36 +3,35 @@ tags:
- globus
- freezer
- file transfer
search:
exclude: true
---

!!! note
This service is still in the testing phase
We are currently trialing the transfer of data to and from Freezer using Globus.
We currently have a new Globus Collection to Freezer called: `{{ globus_collection_freezer}}`.
You will need to authenticate using your Freezer (S3) credentials. Please let us know if you would like some assistance or are having any difficulties with this service.

We are currently trialing the transfer of data to and from Freezer using Globus. We currently have a new Globus Collection to Freezer called: `NeSI Freezer`. You will need to authenticate using your Freezer (S3) credentials. Please let us know if you would like some assistance or are having any difficulties with this service.
## Requirements

You will need to have a Globus account to access Globus via Freezer. Please see the page on [first time Globus set up](First_Time_Setup.md) for information on getting a Globus account.

## Setting up Freezer Credentials

1. Go to the File Manager tab of [your Globus page](https://app.globus.org/file-manager?two_pane=true) in the left hand menu bar.
Under the `Collection` field, search for and select the `NeSI Freezer` collection, then click the blue `Continue` button.
Under the `Collection` field, search for and select the {{ globus_collection_freezer}} collection, then click the blue `Continue` button.
![Globus_Freezer_Collection.png](../../assets/images/Globus_Freezer_Collection.png)


2. You will need to authenticate with an identity from NeSI Keycloak. Click on `Use my user_id@iam.nesi.org.nz identity` text.
![Globus_Transfer_5.png](../../assets/images/Globus_Transfer_5.png)


3. In the next window, click `Allow`.
![Globus_Freezer_Manage_Data.png](../../assets/images/Globus_Freezer_Manage_Data.png)

!!! info
Steps 4 & 5 may not be required

4. To set up your credentials, please click `Continue`
4. To set up your credentials, please click `Continue`. You will be shown a Globus page requiring you to sign in to Mahuika.
![Globus_Freezer_Require_Credentials.png](../../assets/images/Globus_Freezer_Require_Credentials.png)

5. Fill in your Username and Secret Key. Please let us know if you have lost your Freezer Secret Key. We can <a href="mailto:support@nesi.org.nz?subject=Reset%20Freezer%20Secret%20Key">reset your Freezer key</a>, but you will also need to reset your Freezer config on Mahuika.
5. Fill in your Mahuika Username and Secret Key. Please let us know if you have lost your Freezer Secret Key. We can <a href="mailto:support@nesi.org.nz?subject=Reset%20Freezer%20Secret%20Key"> reset your Freezer key</a>, but you will also need to reset your Freezer config on Mahuika.

In the following sections, please enter:

Expand All @@ -47,10 +46,14 @@ We are currently trialing the transfer of data to and from Freezer using Globus.

## Freezer Endpoint

1. Go to the File Manager on the left hand menu and search for the collection `NeSI Freezer` .
1. Go to the File Manager on the left hand menu and search for the collection `{{ globus_collection_freezer}}`.
![Globus_Freezer_EP.png](../../assets/images/Globus_Freezer_EP.png)

2. Under 'Path', type in your Freezer bucket e.g., `nesi99991-12345` and press <kbd>Enter</kbd>. you should now see the contents of your bucket.
![Globus_Freezer_Load_Bucket.png](../../assets/images/Globus_Freezer_Load_Bucket.png)

If you initiate a transfer of offline data (only on tape - Glacier) from Freezer, the data will be automatically staged from tape before the transfer begins.

## Organizing Files Using Globus

We do NOT recommend organising or moving files or directories on Freezer using the Globus interface. Please use the s3cmd tool in these instances. Please see our [Freezer Guide](../../Storage/Long_Term_Storage/Freezer_Guide.md) for more information.
6 changes: 3 additions & 3 deletions docs/Data_Transfer/Globus/Data_Transfer_using_Globus.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ To move files and folders between `collections` (such as between Mahuika and ano
`{{ globus_collection_storage}}` Globus endpoint.


5. You will be shown a Globus page requiring you to sign in to our REANNZ user portal. Click on `Use my username@iam.nesi.org.nz identity` text.
5. You will be shown a globus page requiring you to sign in to Mahuika. Click on `Use my username@iam.nesi.org.nz identity` text.
![Globus_Transfer_5.png](../../assets/images/Globus_Transfer_5.png)

6. Once you have logged in, you will be sent back to the Globus transfer page. By default you will be placed to your
Expand Down Expand Up @@ -85,10 +85,10 @@ any person who has a Globus account.
| {{ globus_collection_storage }} | REANNZ Globus mapped collection, located at Tamaki Data Centre | File-sharing and transferring files to/from Mahuika HPC and storage facilities. | [First Time Setup](./First_Time_Setup.md) | {% include "partials/support_request.html" %} |
| [University of Otago - HCS](https://app.globus.org/file-manager?origin_id=108e72ac-c509-4cd0-940f-b7e3aa543007), [University of Otago - CHC HCS](https://app.globus.org/file-manager?origin_id=eeb5308a-2471-4696-9571-dd2092e041f9) | Endpoint for the High Capacity Research Storage Cluster, Dunedin Campus, University of Otago and Endpoint for the High Capacity Research Storage Cluster, Christchurch Campus, University of Otago | Primary endpoint for Otago Dunedin; uses local service accounts or Globus sharing. | [Access Form](https://www.otago.ac.nz/its/forms/hcs-high-speed-data-transfer-service-access-form) | [university@otago.ac.nz](mailto:university@otago.ac.nz) |
| University of Auckland Research Data Collection | Endpoint provides access to UoA research data. | Transferring files between UoA research drives and Mahuika | Apply by email | [researchdata@auckland.ac.nz](mailto:researchdata@auckland.ac.nz) |
| AgResearch DTN01 | A Globus endpoint attached to AgResearch's institutional Linux storage platform | Sharing large datasets with external collaborators and moving large datasets between NeSI's facility and AgResearch's internal storage platform | Apply by email | [servicedesk@agresearch.co.nz](mailto:servicedesk@agresearch.co.nz) |
| AgResearch DTN01 | A Globus endpoint attached to AgResearch's institutional Linux storage platform | Sharing large datasets with external collaborators and moving large datasets between Mahuika and AgResearch's internal storage platform | Apply by email | [servicedesk@agresearch.co.nz](mailto:servicedesk@agresearch.co.nz) |
| PFR Globus Connect Server | Endpoint provides access to Plant Food Research data | Generally for internal users, but also for sharing large datasets with collaborators | Contact the Plant and Food person you are wanting to share data with. | |
| MWLR PN-DTN-username | Customised endpoints for users to transfer data between MWLR and Mahuika, or to share data with third-party collaborators | Generally for internal users, but also for sharing large datasets with collaborators | Contact the MWLR person you are wanting to share data with. | [IToperations@landcareresearch.co.nz](mailto:IToperations@landcareresearch.co.nz) |
| Scion Data | Endpoint provides access to Scion research data | Sharing large datasets with external collaborators and moving large datasets between NeSI's facility and Scion's internal storage platform | Contact the Scion person you are wanting to share data with. | |
| Scion Data | Endpoint provides access to Scion research data | Sharing large datasets with external collaborators and moving large datasets between Mahuika and Scion's internal storage platform | Contact the Scion person you are wanting to share data with. | |
| ESR Endpoint | Endpoint provides access to ESR data | Generally for internal users, but also for sharing large datasets with collaborators | Contact the ESR person you are wanting to share data with. | |

!!! tip "How to establish a new node"
Expand Down
2 changes: 1 addition & 1 deletion docs/Data_Transfer/Globus/First_Time_Setup.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ You can learn about setting up your personal Globus endpoint on your computer on

## Setting up your Globus account

1. Go to [globus.org](https://www.globus.org) and click the `LOG IN` button.
1. Go to [globus.org](https://www.globus.org), or go to [transfer.reannz.co.nz](transfer.reannz.co.nz) and click the `LOG IN` button.
![Globus\_main\_page.png](../../assets/images/Globus_main_page.png)
2. Look up your institution in the drop-down box, or sign in using your GitHub, Google, or
ORCID ID, then click continue. You will be directed to your Tuakiri login screen or
Expand Down
20 changes: 11 additions & 9 deletions docs/Storage/Long_Term_Storage/Freezer_Guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -237,16 +237,17 @@ s3cmd restore --recursive s3://<freezer-bucket>/your_directory/data_folder/ --re

Example to get or download the directory `data_folder` and all contained objects/files/folders:

1. Create the `data_folder` you want to retrieve in file, and change directory into `data_folder`.
```sh
mkdir -p data_folder
cd data_folder
```
1. Create the `data_folder` you want to retrieve in file, and change directory into `data_folder`.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

low

What in Neptune's name does 'retrieve in file' mean? Are ye tryin' to retrieve the directory locally or on the filesystem? Clarify yer instructions so even a cabin boy can understand 'em!

References
  1. All feedback must be left as a 17th century pirate, slightly snarky and passive aggressive. (link)


2. Retrieve the data from Freezer
```sh
s3cmd get --recursive s3://<freezer-bucket>/your_directory/data_folder/
```
```sh
mkdir -p data_folder
cd data_folder
```

2. Retrieve the data from Freezer
```sh
s3cmd get --recursive s3://<freezer-bucket>/your_directory/data_folder/
```

This will place the all files and subdirectories in the above `data_folder` into your current directory.

Expand All @@ -256,6 +257,7 @@ After retrieving your tarball, you can extract its contents with the following c

```sh
tar -xzvf <name of tarball>.tar.gz
```

## s3cmd reference

Expand Down
Binary file modified docs/assets/images/Globus_Freezer_EP.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified docs/assets/images/Globus_Freezer_Require_Credentials.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading