Galaxy Tool Management with Ephemeris

Overview

question Questions
  • How to install, update, and maintain Galaxy tools?

  • How to extract a list of tools from a workflow or Galaxy instance?

objectives Objectives
  • Learn about Ephemeris

  • Extract a list of tools from a workflow

  • Install these tools on a given Galaxy

time Time estimation: 45 minutes

Supporting Materials

last_modification Last modification: Feb 25, 2021

Overview

This tutorial will introduce you to one of Galaxy’s associated projects - Ephemeris. Ephemeris is a small Python library and set of scripts for managing the bootstrapping of Galaxy plugins - tools, index data, and workflows. It aims to help automate, and limit the quantity of manual actions admins have to do in order to maintain a Galaxy instance.

Agenda

  1. Background
  2. Requirements
  3. Extracting Tools
  4. Installing Tools
  5. Tool Testing
  6. Obtaining a Tool List
  7. Production Best Practices

Background

You are an administrator of a Galaxy server. A colleague has approached you with a request to run a specific Galaxy workflow on their data. In order to enable this workflow for your users, you will have to:

  • identify what tools are required for the workflow
  • install these tools and their dependencies on your Galaxy instance.

Requirements

To run this tutorial, you will need to install Ephemeris. You would normally install it on your workstation, but during training courses we recommend to install it on the same virtual machine used for the Galaxy server.

tip Installing Ephemeris in a Python virtual environment

  1. Install virtualenv if it is not already available. On Ubuntu this can be done with sudo apt install virtualenv
  2. Create a virtual environment just for ephemeris, activate it and install ephemeris inside it:
    virtualenv -p python3 ~/ephemeris_venv
    . ~/ephemeris_venv/bin/activate
    pip install ephemeris
    

Extracting Tools

A common request you will experience as an administrator is “I want to run this workflow”. Since this is such a common workflow, Galaxy has a built in way to accomplish it. We can use Ephemeris to extract a list of tools from a Galaxy workflow document, and then use Ephemeris to install these tools and specific versions into your Galaxy.

However, Galaxy workflow files are complex JSON documents, and the process of mapping the tool IDs to a ToolShed repository and revision is not trivial. Workflow files contain tool IDs which look like:

toolshed.g2.bx.psu.edu/repos/devteam/fastqc/fastqc/0.71
toolshed.g2.bx.psu.edu/repos/bgruening/trim_galore/trim_galore/0.4.3.1
toolshed.g2.bx.psu.edu/repos/iuc/multiqc/multiqc/1.6
toolshed.g2.bx.psu.edu/repos/devteam/bowtie2/bowtie2/2.3.4.2
toolshed.g2.bx.psu.edu/repos/devteam/samtools_stats/samtools_stats/2.0.1
toolshed.g2.bx.psu.edu/repos/devteam/bamtools_filter/bamFilter/2.4.1
toolshed.g2.bx.psu.edu/repos/devteam/samtools_stats/samtools_stats/2.0.1

In order to actually install these tools, we need to convert each tool ID into a ToolShed repository name and revision. For example, FastQC version 0.71 corresponds to revision ff9530579d1f in the ToolShed.

- name: fastqc
  owner: devteam
  revisions:
  - ff9530579d1f
  tool_panel_section_label: Tools from workflows
  tool_shed_url: https://toolshed.g2.bx.psu.edu

Ephemeris can take care of this process. Let’s practice this on a real worfklow.

hands_on Hands-on: Extracting a list of tools from a workflow

  1. Download the mapping workflow:
    wget https://training.galaxyproject.org/archive/2021-03-01/topics/sequence-analysis/tutorials/mapping/workflows/mapping.ga
    
  2. Use the Ephemeris workflow-to-tools command to extract the tool list from this workflow into a file named workflow_tools.yml.

    question Question

    What did your command look like?

    solution Solution

  3. Inspect the workflow_tools.yml file, which contains a tool list in YAML format.

Installing Tools

Now that you have extracted a list of tools, let’s install these on your Galaxy instance. In order to accomplish this, you will need:

  • The URL of your Galaxy server
  • The API key for your account, which must be an admin

tip Getting your admin API key

Galaxy admin accounts are specified as a comma-separated email list in the admin_users directive of galaxy.yml . If you have set up your Galaxy server using the Galaxy Installation with Ansible tutorial, this is set to admin@example.org.

  1. In your browser, open your Galaxy homepage
  2. Log in using the admin email, or register a new account with it if it is the first time you use it
  3. Go to User -> Preferences in the top menu bar, then click on Manage API key
  4. If there is no current API key available, click on Create a new key to generate it
  5. Copy your API key to somewhere convenient, you will need it throughout this tutorial

There are two ways to install tools, depending on how you specify the tools to install:

hands_on Hands-on: Installing a single tool

  1. Use the Ephemeris shed-tools command to install the tool pilon, owned by iuc into a section named Assembly

    question Question

    What did your command look like?

    solution Solution

    Use your Galaxy URL and API key in the example command below:

tip Certificate issues

If your Galaxy instance is served via the HTTPS protocol (as it should be!), ephemeris will use the requests Python library to encrypt the communication with Galaxy. Therefore, if your Galaxy uses a self-signed SSL certificate, shed-tools may fail with a CERTIFICATE_VERIFY_FAILED error.

Under Ubuntu, you can allow the use of the unrecognized certificate as follows:

  1. Get hold of the Certificate Authority (CA) certificate used to sign your Galaxy SSL certificate. For a Galaxy Admin Training course, this is usually the Fake LE Root X1 certificate.
  2. Copy the CA certificate file into /usr/local/share/ca-certificates/ with a .crt extension.
  3. Run update-ca-certificates as root.
  4. Execute export REQUESTS_CA_BUNDLE=/etc/ssl/certs/ca-certificates.crt, as explained in requests docs.

Now you should be able to execute successfully the shed-tools commands.

This provides an easy way to do a one-off installation of a tool, but is not very convenient if you want to install many tools. For that, you can install from a YAML file:

hands_on Hands-on: Installing tools from a tool list

  1. (optional) Watch the installation proceed by running journalctl -f -u galaxy in a separate remote shell.

  2. Use the Ephemeris shed-tools command to install all of the tools from the workflow_tools.yml file on your Galaxy.

    question Question

    What did your command look like?

    solution Solution

    Use your Galaxy URL and API key in the example command below:

  3. Open your Galaxy’s admin interface and check that the tools have been installed.

  4. Using the UI import the workflow file that you used, mapping.ga .

Occasionally the tool installation may fail due to network issues; if it does, just re-run the shed-tools installation process until it succeeds. This is a known issue the developers are working on.

tip Tip: Opening a split screen in byobu

Shift-F2: Create a horizontal split

Shift-Left/Right/Up/Down: Move focus among splits

Ctrl-F6: Close split in focus

Ctrl-D: (Linux, Mac users) Close split in focus

There are more byobu commands described in this gist

tip Can I install tools without restarting?

Yes. The default tool config (config/tool_conf.xml.sample, copy to config/tool_conf.xml to modify) has an option, monitor="true" set in the root <toolbox> tag. This instructs Galaxy to watch the tool files referenced in that config and load or reload them as necessary. It will also add any tools you have added.

tip Can I install tools without a ToolShed?

Yes. The galaxy_local_tools option for the galaxyproject.galaxy Ansible role can be used to install local tools, or you can manage them in another way that fits your workflow better. UseGalaxy.eu, for example, maintains a repository of tools that are not installed from the ToolShed to aid their local developers. This is deployed to the server using the git module, rather than the Galaxy Ansible role.

Tool Testing

Having the tools installed is a good first step, but your users will expect that they actually work as well. You can use Ephemeris to automatically test all of the installed tools.

hands_on Hands-on: Test the installed tools

  1. Use the Ephemeris shed-tools command to test the pilon tool on your Galaxy.

    question Question

    What did your command look like?

    solution Solution

    Use your Galaxy URL and API key in the example command below:

This can give you some more confidence that things are working correctly. Oftentimes, users provide workflows for biological domains that we are not familiar with, so knowing how we can test these tools is impossible for us as admins. Leveraging the built-in tool test cases can give you reassurance that things are functional before you inform your users of the new tools.

Obtaining a Tool List

Sometimes a user might ask you to install all the tools they were previously using on another Galaxy instance. Ephemeris can produce a tool_list.yaml file for all the tools installed on a server.

hands_on Hands-on: Obtain UseGalaxy.eu’s tool list

  1. Use the Ephemeris get-tool-list command to obtain the full set of tools installed on UseGalaxy.eu

    question Question

    What did your command look like?

    solution Solution

    This command does not require authentication and can be used to obtain the tool list from any public Galaxy server:

We will not install all the tools from the EU Galaxy server as that server likely has more tools than any other Galaxy instance, but it is useful as an example of how you can use Ephemeris to facilitate the mirroring of another Galaxy instance.

tip Non-shed tools

The output of get-tool-list only includes ToolShed tools, not local non-TS tools.

Production Best Practices

The servers which are part of the usegalaxy.* network use Ephemeris extensively to manage their large tool sets.

Interestingly, UseGalaxy.eu and UseGalaxy.org.au have different approaches:

  • AU maintains a separate YAML file per tool panel section, which allows to easily identify where tools should be added or found to add new revisions. They follow a cycle of adding the tool to their YAML file, triggering installation, and then updating the YAML files from the current server status.
  • EU maintains YAML files roughly per domain, it is not as clear of an ordering. They maintain a YAML file where humans add the tools which should be installed in a given category, and lock files are automatically generated from these with the latest revision if it is missing. They follow a cycle of updating the lock files with the latest available revisions of tools, and then installing from these lock files any missing revisions. They use a Jenkins server to automatically run tool installation weekly.
  • Together, usegalaxy.* are working on a collaborative approach at galaxyproject/usegalaxy-tools but this is not consumption ready yet.

If running ephemeris directly is not your preference, there is also an Ansible role and a sample playbook that can help automate some tasks.

tip What if environments are not working

It sometimes happens in Galaxy, that one environment isn’t working anymore. It mostly happens from the start when it does happen. You can remove the environment on disk, or use the “Manage Dependencies” interface, select the environment, and delete it. Then re-install the dependency through the same Manage Dependencies interface.

tip Can you install multiple tools simultaneously?

Previous experience with this is not good, there was a lot of unsafe code that would do things simultaneously that would destroy config files. But now it’s conda, so you may destroy conda environments. A solution for this is like how UseGalaxy.eu does it, where we keep the full list of tools we want to install, and then a CI server (Jenkins) installs them. In this way we can enforce that only a single install process is running at any time.

tip Certificate Issues (GAT Only)

If a student is running shed-tools on the VM, then it should work without certificate issues, because we installed the Fake LE X1 CA certificate, meaning that to your VM, the certificate chain is valid. We cannot (and would not recommend) setting that certificate on your local machine. That is the first way that comes to mind, that running an ephemeris command could generate that error.

tip Uninstalling tools

While there is a function to accomplish this in BioBlend, it has not been included in Ephemeris yet. If you’re looking for a way to contribute to Galaxy, this would be great :)

keypoints Key points

  • Ephemeris and automation help with the tool management on Galaxy

  • There are tool management best practices you can learn from

  • Do not manage your Galaxy tools manually

Feedback

Did you use this material as an instructor? Feel free to give us feedback on how it went.

Click here to load Google feedback frame

Citing this Tutorial

  1. Martin Čech, Helena Rasche, Nicola Soranzo, 2021 Galaxy Tool Management with Ephemeris (Galaxy Training Materials). /archive/2021-03-01/topics/admin/tutorials/tool-management/tutorial.html Online; accessed Thu Jan 09 2025
  2. Batut et al., 2018 Community-Driven Data Analysis Training for Biology Cell Systems 10.1016/j.cels.2018.05.012

details BibTeX

congratulations Congratulations on successfully completing this tutorial!