diff --git a/docs/Data_Transfer/Globus/.pages.yml b/docs/Data_Transfer/Globus/.pages.yml index 55faab303..0e87d6f66 100644 --- a/docs/Data_Transfer/Globus/.pages.yml +++ b/docs/Data_Transfer/Globus/.pages.yml @@ -5,6 +5,7 @@ nav: - Add_Your_Computer_To_Globus.md - Data_Transfer_using_Globus.md - Data_Transfer_Between_Personal_Endpoints.md + - Data_Transfer_to_Freezer.md - Signing_Up_To_Globus_Subscription.md - Bookmarks.md - Share_Collections.md diff --git a/docs/Data_Transfer/Globus/Data_Transfer_to_Freezer.md b/docs/Data_Transfer/Globus/Data_Transfer_to_Freezer.md index 9e7a44ec3..295efe5e3 100644 --- a/docs/Data_Transfer/Globus/Data_Transfer_to_Freezer.md +++ b/docs/Data_Transfer/Globus/Data_Transfer_to_Freezer.md @@ -3,36 +3,35 @@ tags: - globus - freezer - file transfer -search: - exclude: true --- -!!! note - This service is still in the testing phase +We are currently trialing the transfer of data to and from Freezer using Globus. +We currently have a new Globus Collection to Freezer called: `{{ globus_collection_freezer}}`. +You will need to authenticate using your Freezer (S3) credentials. Please let us know if you would like some assistance or are having any difficulties with this service. -We are currently trialing the transfer of data to and from Freezer using Globus. We currently have a new Globus Collection to Freezer called: `NeSI Freezer`. You will need to authenticate using your Freezer (S3) credentials. Please let us know if you would like some assistance or are having any difficulties with this service. +## Requirements + +You will need to have a Globus account to access Globus via Freezer. Please see the page on [first time Globus set up](First_Time_Setup.md) for information on getting a Globus account. ## Setting up Freezer Credentials 1. Go to the File Manager tab of [your Globus page](https://app.globus.org/file-manager?two_pane=true) in the left hand menu bar. - Under the `Collection` field, search for and select the `NeSI Freezer` collection, then click the blue `Continue` button. + Under the `Collection` field, search for and select the {{ globus_collection_freezer}} collection, then click the blue `Continue` button. ![Globus_Freezer_Collection.png](../../assets/images/Globus_Freezer_Collection.png) - 2. You will need to authenticate with an identity from NeSI Keycloak. Click on `Use my user_id@iam.nesi.org.nz identity` text. ![Globus_Transfer_5.png](../../assets/images/Globus_Transfer_5.png) - 3. In the next window, click `Allow`. ![Globus_Freezer_Manage_Data.png](../../assets/images/Globus_Freezer_Manage_Data.png) !!! info Steps 4 & 5 may not be required -4. To set up your credentials, please click `Continue` +4. To set up your credentials, please click `Continue`. You will be shown a Globus page requiring you to sign in to Mahuika. ![Globus_Freezer_Require_Credentials.png](../../assets/images/Globus_Freezer_Require_Credentials.png) -5. Fill in your Username and Secret Key. Please let us know if you have lost your Freezer Secret Key. We can reset your Freezer key, but you will also need to reset your Freezer config on Mahuika. +5. Fill in your Mahuika Username and Secret Key. Please let us know if you have lost your Freezer Secret Key. We can reset your Freezer key, but you will also need to reset your Freezer config on Mahuika. In the following sections, please enter: @@ -47,10 +46,14 @@ We are currently trialing the transfer of data to and from Freezer using Globus. ## Freezer Endpoint -1. Go to the File Manager on the left hand menu and search for the collection `NeSI Freezer` . +1. Go to the File Manager on the left hand menu and search for the collection `{{ globus_collection_freezer}}`. ![Globus_Freezer_EP.png](../../assets/images/Globus_Freezer_EP.png) 2. Under 'Path', type in your Freezer bucket e.g., `nesi99991-12345` and press Enter. you should now see the contents of your bucket. ![Globus_Freezer_Load_Bucket.png](../../assets/images/Globus_Freezer_Load_Bucket.png) If you initiate a transfer of offline data (only on tape - Glacier) from Freezer, the data will be automatically staged from tape before the transfer begins. + +## Organizing Files Using Globus + +We do NOT recommend organising or moving files or directories on Freezer using the Globus interface. Please use the s3cmd tool in these instances. Please see our [Freezer Guide](../../Storage/Long_Term_Storage/Freezer_Guide.md) for more information. diff --git a/docs/Data_Transfer/Globus/Data_Transfer_using_Globus.md b/docs/Data_Transfer/Globus/Data_Transfer_using_Globus.md index c99e42ab6..cd0a4bade 100644 --- a/docs/Data_Transfer/Globus/Data_Transfer_using_Globus.md +++ b/docs/Data_Transfer/Globus/Data_Transfer_using_Globus.md @@ -28,7 +28,7 @@ To move files and folders between `collections` (such as between Mahuika and ano `{{ globus_collection_storage}}` Globus endpoint. -5. You will be shown a Globus page requiring you to sign in to our REANNZ user portal. Click on `Use my username@iam.nesi.org.nz identity` text. +5. You will be shown a globus page requiring you to sign in to Mahuika. Click on `Use my username@iam.nesi.org.nz identity` text. ![Globus_Transfer_5.png](../../assets/images/Globus_Transfer_5.png) 6. Once you have logged in, you will be sent back to the Globus transfer page. By default you will be placed to your @@ -85,10 +85,10 @@ any person who has a Globus account. | {{ globus_collection_storage }} | REANNZ Globus mapped collection, located at Tamaki Data Centre | File-sharing and transferring files to/from Mahuika HPC and storage facilities. | [First Time Setup](./First_Time_Setup.md) | {% include "partials/support_request.html" %} | | [University of Otago - HCS](https://app.globus.org/file-manager?origin_id=108e72ac-c509-4cd0-940f-b7e3aa543007), [University of Otago - CHC HCS](https://app.globus.org/file-manager?origin_id=eeb5308a-2471-4696-9571-dd2092e041f9) | Endpoint for the High Capacity Research Storage Cluster, Dunedin Campus, University of Otago and Endpoint for the High Capacity Research Storage Cluster, Christchurch Campus, University of Otago | Primary endpoint for Otago Dunedin; uses local service accounts or Globus sharing. | [Access Form](https://www.otago.ac.nz/its/forms/hcs-high-speed-data-transfer-service-access-form) | [university@otago.ac.nz](mailto:university@otago.ac.nz) | | University of Auckland Research Data Collection | Endpoint provides access to UoA research data. | Transferring files between UoA research drives and Mahuika | Apply by email | [researchdata@auckland.ac.nz](mailto:researchdata@auckland.ac.nz) | -| AgResearch DTN01 | A Globus endpoint attached to AgResearch's institutional Linux storage platform | Sharing large datasets with external collaborators and moving large datasets between NeSI's facility and AgResearch's internal storage platform | Apply by email | [servicedesk@agresearch.co.nz](mailto:servicedesk@agresearch.co.nz) | +| AgResearch DTN01 | A Globus endpoint attached to AgResearch's institutional Linux storage platform | Sharing large datasets with external collaborators and moving large datasets between Mahuika and AgResearch's internal storage platform | Apply by email | [servicedesk@agresearch.co.nz](mailto:servicedesk@agresearch.co.nz) | | PFR Globus Connect Server | Endpoint provides access to Plant Food Research data | Generally for internal users, but also for sharing large datasets with collaborators | Contact the Plant and Food person you are wanting to share data with. | | | MWLR PN-DTN-username | Customised endpoints for users to transfer data between MWLR and Mahuika, or to share data with third-party collaborators | Generally for internal users, but also for sharing large datasets with collaborators | Contact the MWLR person you are wanting to share data with. | [IToperations@landcareresearch.co.nz](mailto:IToperations@landcareresearch.co.nz) | -| Scion Data | Endpoint provides access to Scion research data | Sharing large datasets with external collaborators and moving large datasets between NeSI's facility and Scion's internal storage platform | Contact the Scion person you are wanting to share data with. | | +| Scion Data | Endpoint provides access to Scion research data | Sharing large datasets with external collaborators and moving large datasets between Mahuika and Scion's internal storage platform | Contact the Scion person you are wanting to share data with. | | | ESR Endpoint | Endpoint provides access to ESR data | Generally for internal users, but also for sharing large datasets with collaborators | Contact the ESR person you are wanting to share data with. | | !!! tip "How to establish a new node" diff --git a/docs/Data_Transfer/Globus/First_Time_Setup.md b/docs/Data_Transfer/Globus/First_Time_Setup.md index c147a57fb..895609f7f 100644 --- a/docs/Data_Transfer/Globus/First_Time_Setup.md +++ b/docs/Data_Transfer/Globus/First_Time_Setup.md @@ -14,7 +14,7 @@ You can learn about setting up your personal Globus endpoint on your computer on ## Setting up your Globus account -1. Go to [globus.org](https://www.globus.org) and click the `LOG IN` button. +1. Go to [globus.org](https://www.globus.org), or go to [transfer.reannz.co.nz](transfer.reannz.co.nz) and click the `LOG IN` button. ![Globus\_main\_page.png](../../assets/images/Globus_main_page.png) 2. Look up your institution in the drop-down box, or sign in using your GitHub, Google, or ORCID ID, then click continue. You will be directed to your Tuakiri login screen or diff --git a/docs/Storage/Long_Term_Storage/Freezer_Guide.md b/docs/Storage/Long_Term_Storage/Freezer_Guide.md index bb89f3126..c00342f09 100644 --- a/docs/Storage/Long_Term_Storage/Freezer_Guide.md +++ b/docs/Storage/Long_Term_Storage/Freezer_Guide.md @@ -237,16 +237,17 @@ s3cmd restore --recursive s3:///your_directory/data_folder/ --re Example to get or download the directory `data_folder` and all contained objects/files/folders: - 1. Create the `data_folder` you want to retrieve in file, and change directory into `data_folder`. - ```sh - mkdir -p data_folder - cd data_folder - ``` +1. Create the `data_folder` you want to retrieve in file, and change directory into `data_folder`. - 2. Retrieve the data from Freezer - ```sh - s3cmd get --recursive s3:///your_directory/data_folder/ - ``` + ```sh + mkdir -p data_folder + cd data_folder + ``` + +2. Retrieve the data from Freezer + ```sh + s3cmd get --recursive s3:///your_directory/data_folder/ + ``` This will place the all files and subdirectories in the above `data_folder` into your current directory. @@ -256,6 +257,7 @@ After retrieving your tarball, you can extract its contents with the following c ```sh tar -xzvf .tar.gz +``` ## s3cmd reference diff --git a/docs/assets/images/Globus_Freezer_EP.png b/docs/assets/images/Globus_Freezer_EP.png index f2f11f8bc..071fe4fb4 100644 Binary files a/docs/assets/images/Globus_Freezer_EP.png and b/docs/assets/images/Globus_Freezer_EP.png differ diff --git a/docs/assets/images/Globus_Freezer_Require_Credentials.png b/docs/assets/images/Globus_Freezer_Require_Credentials.png index 7200ade48..0ef1341aa 100644 Binary files a/docs/assets/images/Globus_Freezer_Require_Credentials.png and b/docs/assets/images/Globus_Freezer_Require_Credentials.png differ