Batch Connect - OSC Galaxy

An interactive app designed for OSC and UCR HPCC OnDemand that launches Galaxy within a batch job.

Prerequisites

This Batch Connect app requires the following software be installed on the compute nodes that the batch job is intended to run on (NOT the OnDemand node):

• Lmod 6.0.1+ or any other module restore and module load <modules> based CLI used to load appropriate environments within the batch job before launching Galaxy.

Install Galaxy and Bootstrap tools

The install process runs on any compute node

Use git to clone this app and checkout the desired branch/version you want to use:

git clone <repo>
cd <dir>
git checkout <tag/branch>

Install Galaxy and dependencies

sh install-galaxy-part1.sh

You will not need to do anything beyond this as all necessary assets are installed. You will also not need to restart this app as it isn't a Passenger app.

To update the app you would:

cd <dir>
git fetch
git checkout <tag/branch>

Again, you do not need to restart the app as it isn't a Passenger app.

To configure preset tools for Galaxy from the usegalaxy-tools repo, you will need at least two SSH sessions on the same compute node. One session will initialize the necessary files and directory to bootstrap a Galaxy instance and the other session will connect to that Galaxy session in order to install the tools from the usegalaxy-tools repo onto the bootstrap Galaxy instance.

In one session run the following:

sh install-galaxy-part2-p1.sh

In the second session run the following:

sh install-galaxy-part2-p2.sh

The reason why two sessions are required is that on our systems, Galaxy can not be launched in daemon mode under the directory where environment modules are stored. To work around this, we just launch Galaxy normally in the foreground and then use Ephemeris to automate the process of configuring and installing preset tools.

Contributing

1. Fork it ( https://github.com/OSC/bc_osc_galaxy/fork )
2. Create your feature branch (git checkout -b my-new-feature)
3. Commit your changes (git commit -am 'Add some feature')
4. Push to the branch (git push origin my-new-feature)
5. Create a new Pull Request

Developer Notes

• In order to allow remote user authentication to work for Galaxy instances launched via OnDemand, the source file for remote user authentication (remoteuser.py) was modified to include custom code developed by UCR HPCC. The installation script takes care of replacing the source file with the modified file. For more information regarding the changes done to remoteuser.py, please refer to the following section
• Other custom scripts and configuration files were developed to allow for a user to submit jobs via slurm and access the environment module system. These scripts and configuration files don't modify existing Galaxy code, instead they interface with existing Galaxy systems intended for modification by system administrators. For more information regarding the implementation, please refer to the following section.
• Custom webhook was developed to allow users to set job options for jobs submitted via slurm. The webhook is only active when a user launches a Galaxy session with the selected runner set as cluster. Currently the options set via this webhook are saved to a Galaxy user's sqlite database, allowing for the options to be used when queuing jobs via Galaxy. The configuration files and scripts for the webhook can be found under custom-scripts/slurm/. For more information regarding the implementation, please refer to the following section.
• Modification to configuration files were made to allow for Galaxy instances to interact with the environment module system on UCR's HPC cluster. Initialization script was developed for preset tools from the usegalaxy-tools repo to be avabile for all Galaxy instances launched via OnDemand. For more information regarding the implementation, please refer to the following section

Overview

Galaxy interface app runs on UCR HPC systems. The users can install, manage and run tools and workflows.

Changes

• Added install-galaxy-part1.sh to install Galaxy 25.0.3
• Added install-galaxy-part2-p1.sh and install-galaxy-part2-p2.sh to configure preset tools
• Galaxy config files are generated by before.sh.yml instead
• Update README
• Job Configuration. Currently, the jobs submitted via Galaxy will either run on the same node as the Galaxy instance(local) or via slurm(cluster)
• Added custom webhook to masthead panel to allow users to change slurm job settings for jobs submitted via Galaxy through slurm
• Remote user authentication. Achieved by modifying source remouteuser.py file
• Added configuration and tool files to interact with environment module system
• Get Data from external data sources has been resolved

Known Issues

See the following page for listed known issues not stated here

Database:

Data files are stored in the user’s dataroot (default to ~/.galaxy/ configured in Galaxy.yml

mcuay001@bluejay:~/bigdata_operations$ ls .galaxy/
cache   container_cache  dependencies  import-dir                 object_store_cache  shed_tool_data  tools              universe.sqlite
config  control.sqlite   gravity       install_tool_sheds.sqlite  results.sqlite      tmp             tool_search_index

Authentication

Previously, galaxy.yml took in the user email address as the user authentication in the single-user mode. In this mode, any user on the cluster who knew the web address where the Galaxy instance was being hosted was able to access said Galaxy instance, without being the user who launched the Galaxy session via OnDemand in the first place. This is not a bug with Galaxy, this is an intentional feature as part of single-user mode as stated in the Galaxy configuration manual.

The benefit to using single-user mode is that a user can access a Galaxy instance without needing to authenticate, thus allowing for Galaxy to act as an application and not a web server, which OnDemand applications are configured as. Remote user mode is similar in that it bypasses the need for a user to login via a tradiational login portal but instead adds some authentication via HTTP headers, for more information please reference the Galaxy configuration manual on the topic. Since Galaxy at its core is supposed to run as a stand alone web server, modifications to the remoteuser.py file had to be made to authenticate users via this mode through OnDemand.

The modification to remoteuser.py is the addition of one function named verify_user, which is implemented here from lines 103 to 115. The function simply checks if the remote user header passed via HTTP by nginx is the same user listed as the admin user on the galaxy config file generated before each Galaxy instance launched via OnDemand. Modification of the remoteuser.py source code isn't the most ideal solution, however other authentication options provided by Galaxy either didn't support the remote user mode as desired or would've required extensive research and effort to implement. Further authentication methods can be configured as described here, and maybe later on they can be configured as a replacement to the current solution implemented.

Select Job Runner

The users select the tool runner before starting the app. The developer adds destinations to job config file and assigns the user-selected runner to default. Job runner field:

Job Runners

1.) local: Run tools locally

• Tool jobs won't be queued and will run immediately
• The number of concurrent jobs is limited, the maximum is the number of cores.
• When the session ends, the unfinished jobs will end too.

2.) cluster: Submit tool jobs to the cluster

• When the session ends, the unfinished jobs will continue to run.
• Unlimited number of concurrent jobs
• Galaxy can only submit jobs to the partitions the user who launched Galaxy as has access too.

Job runners are set and controlled by two files, the first being a configuration file generated at runtime via before.sh.yml and the second being a python scriptcustom_destinations.py found under the custom-scripts/ directory. The job runner selected via OnDemand is set in the configuration file here, with the local option corresponding to the local runner in the configuration file and the cluster option corresponding to the slurm runner. The options for each runner are set in the destination block and are basically the same for both runners, all we're doing here is setting the path where the environment modules are, the number of cores to utilize for local jobs, and loading in the workspace module to define where temporary files generated should be placed.

The default runner is dynamic_cores_time, which dynamically controls job settings and destinations during runtime. Destinations in Galaxy just define what runner a job should use. This runner is custom and defined as a function in the custom_destinations.py file. The main reason why we're using a custom runner is mainly to configure job options for jobs submitted via slurm, options such as the partition to use, amount of cores, etc. If a dynamic job runner is not used, then these options must be defined in the configuration file generated via before.sh.yml and once Galaxy is running, this file can't be updated dynamically. For more information regarding job runners and custom destinations, refer to the Galaxy manual on the topic here

Slurm Webhook

Because job options for jobs submitted via slurm are not something a Galaxy user can modify directly, a custom webhook was developed to allow users to set these options via Galaxy. An example of how the webhook works on the user side can be found here. The webhook files can be found under the custom-scripts/slurm directory and is composed of 4 files, those being config.yml, script.js, styles.css, and __init__.py — with script.js and __init__.py being the most important as these two are main files that handle receiving the job options from a Galaxy user and updating the Galaxy session in the backend with the job options for custom_destinations.py to use.

The file script.js creates the menu and text fields the user interacts with and calls __init__.py, passing in the information the user supplied in JSON format here. The file __init__.py then updates a user object's preferences field in dictionary format, with the key being 'slurm' and the value being the JSON data received. There is also an if statement that will simply return the partitions a user has access to, this is just to populate the 'Partitions' field on the menu created by script.js. For more information regarding webhooks, please refer to the Galaxy manual on the topic here.

Modules and Preset Tools

The Galaxy environment module on the cluster comes configured with tools from the usegalaxy-tools repo, specifically the tools listed under the usegalaxy.org directory in the repo. These are the same tools installed and configured on Galaxy's public accessible instance https://usegalaxy.org. These tools are configured using the yaml.lock files from the repo, the reason being that these files organize tools by panel sections properly which the yaml files in the repo don't. The tools are installed by through install-galaxy-part2-p2.sh, with following line downloading the tool configuration files and updating the necessary Galaxy configuration files and the final line downloads the actual binary for a specific tool via conda.