Galaxy Tool Management with Ephemeris
Overview
question Questionsobjectives Objectives
How to install, update, and maintain Galaxy tools?
How to extract a list of tools from a workflow or Galaxy instance?
Learn about Ephemeris
Extract a list of tools from a workflow
Install these tools on a given Galaxy
time Time estimation: 45 minutes
Supporting Materials
last_modification Last modification: Feb 25, 2021
Overview
This tutorial will introduce you to one of Galaxy’s associated projects - Ephemeris. Ephemeris is a small Python library and set of scripts for managing the bootstrapping of Galaxy plugins - tools, index data, and workflows. It aims to help automate, and limit the quantity of manual actions admins have to do in order to maintain a Galaxy instance.
Agenda
Background
You are an administrator of a Galaxy server. A colleague has approached you with a request to run a specific Galaxy workflow on their data. In order to enable this workflow for your users, you will have to:
- identify what tools are required for the workflow
- install these tools and their dependencies on your Galaxy instance.
Requirements
To run this tutorial, you will need to install Ephemeris. You would normally install it on your workstation, but during training courses we recommend to install it on the same virtual machine used for the Galaxy server.
tip Installing Ephemeris in a Python virtual environment
Extracting Tools
A common request you will experience as an administrator is “I want to run this workflow”. Since this is such a common workflow, Galaxy has a built in way to accomplish it. We can use Ephemeris to extract a list of tools from a Galaxy workflow document, and then use Ephemeris to install these tools and specific versions into your Galaxy.
However, Galaxy workflow files are complex JSON documents, and the process of mapping the tool IDs to a ToolShed repository and revision is not trivial. Workflow files contain tool IDs which look like:
toolshed.g2.bx.psu.edu/repos/devteam/fastqc/fastqc/0.71
toolshed.g2.bx.psu.edu/repos/bgruening/trim_galore/trim_galore/0.4.3.1
toolshed.g2.bx.psu.edu/repos/iuc/multiqc/multiqc/1.6
toolshed.g2.bx.psu.edu/repos/devteam/bowtie2/bowtie2/2.3.4.2
toolshed.g2.bx.psu.edu/repos/devteam/samtools_stats/samtools_stats/2.0.1
toolshed.g2.bx.psu.edu/repos/devteam/bamtools_filter/bamFilter/2.4.1
toolshed.g2.bx.psu.edu/repos/devteam/samtools_stats/samtools_stats/2.0.1
In order to actually install these tools, we need to convert each tool ID into a ToolShed repository name and revision. For example, FastQC version 0.71 corresponds to revision ff9530579d1f
in the ToolShed.
- name: fastqc
owner: devteam
revisions:
- ff9530579d1f
tool_panel_section_label: Tools from workflows
tool_shed_url: https://toolshed.g2.bx.psu.edu
Ephemeris can take care of this process. Let’s practice this on a real worfklow.
hands_on Hands-on: Extracting a list of tools from a workflow
- Download the mapping workflow:
wget https://training.galaxyproject.org/archive/2021-03-01/topics/sequence-analysis/tutorials/mapping/workflows/mapping.ga
Use the Ephemeris
workflow-to-tools
command to extract the tool list from this workflow into a file namedworkflow_tools.yml
.question Question
What did your command look like?
solution Solution
- Inspect the
workflow_tools.yml
file, which contains a tool list in YAML format.
Installing Tools
Now that you have extracted a list of tools, let’s install these on your Galaxy instance. In order to accomplish this, you will need:
- The URL of your Galaxy server
- The API key for your account, which must be an admin
tip Getting your admin API key
There are two ways to install tools, depending on how you specify the tools to install:
hands_on Hands-on: Installing a single tool
Use the Ephemeris
shed-tools
command to install the toolpilon
, owned byiuc
into a section namedAssembly
question Question
What did your command look like?
solution Solution
tip Certificate issues
This provides an easy way to do a one-off installation of a tool, but is not very convenient if you want to install many tools. For that, you can install from a YAML file:
hands_on Hands-on: Installing tools from a tool list
(optional) Watch the installation proceed by running
journalctl -f -u galaxy
in a separate remote shell.Use the Ephemeris
shed-tools
command to install all of the tools from theworkflow_tools.yml
file on your Galaxy.question Question
What did your command look like?
solution Solution
Open your Galaxy’s admin interface and check that the tools have been installed.
Using the UI import the workflow file that you used, mapping.ga .
Occasionally the tool installation may fail due to network issues; if it does, just re-run the shed-tools
installation process until it succeeds. This is a known issue the developers are working on.
tip Tip: Opening a split screen in byobu
tip Can I install tools without restarting?
tip Can I install tools without a ToolShed?
Tool Testing
Having the tools installed is a good first step, but your users will expect that they actually work as well. You can use Ephemeris to automatically test all of the installed tools.
hands_on Hands-on: Test the installed tools
Use the Ephemeris
shed-tools
command to test thepilon
tool on your Galaxy.question Question
What did your command look like?
solution Solution
This can give you some more confidence that things are working correctly. Oftentimes, users provide workflows for biological domains that we are not familiar with, so knowing how we can test these tools is impossible for us as admins. Leveraging the built-in tool test cases can give you reassurance that things are functional before you inform your users of the new tools.
Obtaining a Tool List
Sometimes a user might ask you to install all the tools they were previously using on another Galaxy instance. Ephemeris can produce a tool_list.yaml
file for all the tools installed on a server.
hands_on Hands-on: Obtain UseGalaxy.eu’s tool list
Use the Ephemeris
get-tool-list
command to obtain the full set of tools installed on UseGalaxy.euquestion Question
What did your command look like?
solution Solution
We will not install all the tools from the EU Galaxy server as that server likely has more tools than any other Galaxy instance, but it is useful as an example of how you can use Ephemeris to facilitate the mirroring of another Galaxy instance.
tip Non-shed tools
Production Best Practices
The servers which are part of the usegalaxy.*
network use Ephemeris extensively to manage their large tool sets.
Interestingly, UseGalaxy.eu and UseGalaxy.org.au have different approaches:
- AU maintains a separate YAML file per tool panel section, which allows to easily identify where tools should be added or found to add new revisions. They follow a cycle of adding the tool to their YAML file, triggering installation, and then updating the YAML files from the current server status.
- EU maintains YAML files roughly per domain, it is not as clear of an ordering. They maintain a YAML file where humans add the tools which should be installed in a given category, and lock files are automatically generated from these with the latest revision if it is missing. They follow a cycle of updating the lock files with the latest available revisions of tools, and then installing from these lock files any missing revisions. They use a Jenkins server to automatically run tool installation weekly.
- Together,
usegalaxy.*
are working on a collaborative approach at galaxyproject/usegalaxy-tools but this is not consumption ready yet.
If running ephemeris directly is not your preference, there is also an Ansible role and a sample playbook that can help automate some tasks.
tip What if environments are not working
tip Can you install multiple tools simultaneously?
tip Certificate Issues (GAT Only)
tip Uninstalling tools
keypoints Key points
Ephemeris and automation help with the tool management on Galaxy
There are tool management best practices you can learn from
Do not manage your Galaxy tools manually
Feedback
Did you use this material as an instructor? Feel free to give us feedback on how it went.
Citing this Tutorial
- Martin Čech, Helena Rasche, Nicola Soranzo, 2021 Galaxy Tool Management with Ephemeris (Galaxy Training Materials). /archive/2021-03-01/topics/admin/tutorials/tool-management/tutorial.html Online; accessed Thu Jan 09 2025
- Batut et al., 2018 Community-Driven Data Analysis Training for Biology Cell Systems 10.1016/j.cels.2018.05.012
details BibTeX