Using Globus with ARCC
Globus is a widely used data transfer tool by many academic and research instituions. This page descibes how to get started using Globus with ARCC Resources.
Assuption
This guide assumes that you already have the abiltiy to access ARCC resources.
Step 1: Navigate to the Globus website
The first step to using Globus is using your favorite web browser to go the website at Globus.org.
OR if you are more familiar with using the MedicineBow Open OnDemand Page, you can simply click the ‘Globus’ button on the Files app.
From here we click on the ‘Get Started’ Rocket icon on the top right of the page.
From here there should should be a page that has different links for different roles. This can be confusing, the best bet is to look for the ‘Move Data Now’ box and click the link that says “login to Globus”.
Step 2: Log in to Globus
Now since ARCC protects data by usernames and passwords you will have to login to Globus in order to transfer data that you have access to. The good news is that if you have an active UWyo account you don’t have to create a new account! You can look up your organization to login. For ARCC, you will have to search for ‘Wyoming’ to find the University of Wyoming.
Warning
ARCC resources are only accessible with full UWyo accounts. If you have an ARCC-only or external colaborator account, you won't be able to connect. See our page on creating guest shares if you have a collaborator that can share data with you.
After you find the University of Wyoming in the drop-down menu, please click the blue continue button. From there, you will see the very familiar Wyo Login screen that other University services use.
And now you may login with your regular UWyo username and password. If this is the first time you are logging into Globus, you will be prompted with a message about linking accounts. This is actually a very useful feature if you are using other resources outside of UWyo like the NCAR Wyoming Supercomputing Center’s Derecho or Argonne Leadership Computing Facility resources as well, but if this is your first time ever using Globus just click “Continue” on this page to move on.
You will now be asked to accept the Terms of Service and Privacy Policy before your account will be able to be used.
Next the app will ask for permission to make transfers, view search for data associated with your identities in your account, as well as manage your Globus Groups. Please click “Allow” to move forward.
That's it! You can now use Globus to transfer data.
Step 3: Searching for Collections
Now that all of the account setup is completed, you will be sent to the file manager screen. More on using the file manager will be covered in the next step. Since Globus is a major service that is used by many universities and national labs, you have to search for where your data are hosted by your institution. Globus does this with the concept of what they call ‘Collections’. To use a collection, there is another concept called ‘Endpoints’ that are the technical location for your data. Endpoints, particularly personal endpoints, will be covered on the final step.
Note
It is important to know that some institutions uses these terms interchangeably so you may need to know the name of the endpoint instead of the collection and vise versa.
To get started looking for your data, please click the ‘Search’ with the magnifying glass icon on the file manager that says ‘Collection’
At ARCC we have just a few collections that we manage. To begin searching for ARCC managed collections, Type ‘uwyo-ARCC’ into the search.
You may notice that there are a few results, some may or may not be still in use, the best practice is to use the collection for the service you know you are using, e.g. Medicine Bow. All examples in the following steps will be using the Medicine Bow collection.
Current ARCC Managed Collections: - Medicine Bow
Info
If you have an Alcova storage only or AI4WY project, the collection to use will be the Medicine Bow collection
If we select the ‘Medicine Bow’ collection, we are taken back to the File Manager window and can now navigate to wherever we have access to on the system.
Step 4: Using the File Manager
The File Manager is where the bulk of the work is done in Globus, there are a few things to be aware of before starting to manage or transfer data when using this application.
Managing Panes
First off, if transferring data between two systems, it is this author’s preference to change the panel view to two separate panels one for each system. On the top right of the window, you can see how to change the panel view, by default it only shows pane 1.
Click on the middle icon showing two panes to see both collections that you want to see in the File Manager.
Now you can see the collection that you just logged into and another pane where you can search for a new collection from another system, at a different institution or your personal or work computer (more on this later).
File Management
Now that we have access to a collection, we can manage files right here using the web application if we wish. With options to share, transfer, create folders, rename and even download right here as long as we have permission on the system. using the options in the middle of the panes. You can see that the options will either be blue or grayed out. Blue means you can perform that operation and grayed out means that you must select something first.
For simplicity, this page will only cover data transfers, to do that we need to navigate to a place that we have permission to alter data such as our HPC project directory on MedicineBow.
Navigation
There are two ways to navigate directories in Globus. 1) using the path bar under the collection or 2) clicking on the displayed folder in the pane. If you know the path you need to go to using the direct path is the fastest way because if there are directories that have many folders in them, such as ‘/project’ it can take a while for the pane to display them all when clicking.
Note
Please notice that on this endpoint the root directory is called ‘cluster’ you will need to begin your path search with that folder name instead of something more familiar to you such as /home or /gscratch
This endpoint is unique since it serves as the collection for two separate ARCC services HPC and storage-only projects. (MedicineBow, AI4WY, and Alcova)
- When navigating to an MedicineBow HPC project, use the sub-folder ‘medbow’
- For AI4WY projects, go to 'ai4wy'
- Storage only projects use the sub-folder ‘alcova’
For the following examples, we will use the medbow sub-folder.
Transferring Data
Now that we can navigate the system on the collection we have access to, we can now transfer data. For demonstration purposes only, we will use the Globus Tutorial collection to transfer data into our project on MedicineBow. In the second pane, please search for ‘Globus Tutorial’ and select ‘Globus Tutorial Collection 1’.
This is a wide open collection that anyone with a Globus account can access. Now that we have accessed to two different systems we can transfer data between them.
You may notice that the Globus Tutorial is empty, that is because the administrators of this collection routinely clean it up. For our purposes, we will create a folder on the tutorial collection and transfer it to our HPC project. As you can see above on the left pane, it has already been navigated to a training project on MedicineBow that ARCC uses for workshops etc.
Create a folder with the name ‘Globus-test' in the tutorial endpoint using the ‘New Folder’ button in file management panel between the panes
Now all we have to do is select this folder and transfer it to our MedicineBow project. Once selected, there will be a check mark next to the folder and it will be highlighted in blue. Also notice that there is a button that says ‘start’ that changed colors once you select something from light blue to dark blue. This means this folder is ready to transfer.
All we need to do next is to simply click that button and it will transfer to the collection we have connected to in the other pane.
Once clicked, there will be a pop-up that says that the request was submitted successfully and the Activity icon on the left menu turns green. If there are any issues with a transfer it will be red or yellow instead of green.
You can click that ‘Activity’ icon to see the progress of any transfer and get performance statistics. Here we see that our transfer was successful and it took all of 4 seconds to complete.
Additionally, Globus sends an automated email to the email account you have associated with your account with this info.
Now if we go back to the file manager and click “refresh list” on the collection we just transferred to, we can see the folder in the list of other folders and files.
There are many more functions that Globus can provide in transfers such as encryption and others, but should only be used when needed.
Final Thoughts and Summary
Globus is one of the most widely used data transfer tools in research computing and if collaborating with others at different universities, can make sharing data much easier than traditional command line options.
In summary, this page covers:
- How to get to the Globus web application from both directly on their website as well as from MedicineBow Open OnDemand
- How to login with your University of Wyoming account (not available for ARCC-only accounts or external collaborators)
- Searching for ARCC collections
- Using the File Manager to transfer data and monitor them