View markdown source on GitHub

Tool Shed: sharing Galaxy tools

Contributors

AvatarAnthony Bretaudeau AvatarLoraine Guéguen AvatarStéphanie Legras AvatarBérénice Batut

Questions

Objectives

last_modification Last modification: Jul 9, 2021

1. What is a Tool Shed?

toolshed logo with a shed over the words galaxy tool shed


same logo = Galaxy App Store


same logo again = Galaxy App Store


same logo again = Galaxy App Store


Available Tool Sheds


Tool Shed interface

toolshed interface with categories, description, and number of repositories called out with arrows


Example of tool

view of a repository in the toolshed showing a lot of metadata


2. Installing tools from a Tool Shed


Connect your Galaxy to a Tool Shed


Install a tool from the Tool Shed

arrow to galaxy main toolshed button in older versions of galaxy


Install a tool from the Tool Shed


Install a tool from the Tool Shed


Install a tool from the Tool Shed

trinity button showing a dropdown with preview and install


Install a tool from the Tool Shed


Install a tool from the Tool Shed


Manage installed tools


Manage installed tools


Tips: How to install workflows


How to install a workflow?


How to install a workflow?

galaxy workflow list showing no workflows


How to install a workflow?

the installed WF is found in the manage installed tools menu


How to install a workflow?

contents of the repository section lists fastqc, a link to click on


How to install a workflow?

clicking on that brings you to a view of the workflow, and you can click import to galaxy on the top right


How to install a workflow?

it appears in the workflow menu


3. Tool Shed repository types


3.1 Tool with dependencies


Tool with dependencies

tool folder with python script, xml, and test-data is put in an archive and uploaded to the TS.


3.2 Tool suite


Tool suite

multiple wrappers in a single tool archive


Tool suite

phylogeny pipeline containing 4 tools


Tool suite

[…]
No need to write it yourself, Planemo does it for you

---

###  Tool suite

![phylogeny suite gets uploaded to the TS](../../images/toolsuite3.png)

---

### 3.3 Workflow

---


###  Workflow
![the four tools from the phylogeny pipeline are connected as a workflow](../../images/workflow.png)
- A succession of tools to perform a complete analysis

---

###  Workflow

- In galaxy

![same workflow with galaxy ui](../../images/galaxyWorkflowEx.png)

---

###  Workflow

![extract workflow menu from galaxy](../../images/shareWorkflow.png)

- You get a file: Galaxy-Workflow-NAME.ga

---

###  Workflow

- You have your file Galaxy-Workflow-NAME.ga
- How to automatically install the tools used in the workflow?
  - -> `repository_dependencies.xml`
```xml
<?xml version="1.0"?>
<repositories description="Phylogeny workflow">
   <repository name="suite_phylogeny" owner="galaxy" />
</repositories>

Workflow

phylogeny workflow is uploaded to the TS


Workflow

the workflow points at the tool suite which points at tools


Tips

What about tools working with *.loc files or new datatypes


Tools with *.loc files or new datatypes

Their content or the file themselves are put to the right location on tool set up.


Tools with *.loc files or new datatypes

toolshed interface showing some files

tool_data_table_conf.xml.sample

<tables>
	<!-- Locations of indexes in the Bowtie2 mapper format -->
	<table name="bowtie2_indexes" comment_char="#">
		<columns>value, dbkey, name, path</columns>
		<file path="tool-data/bowtie2_indices.loc" />
	</table>
</tables>

On install, the content of the file and the right path to the *.loc file is added to shed_tool_data_table_conf.xml


4. Publishing with Planemo


warning There are automated ways to publish to a ToolShed using GitHub and Continuous Integration

Check out these slides, “Publishing tools” section


Configure the target Tool Shed

planemo config_init
## Planemo Global Configuration File.
## Everything in this file is completely optional - these values can all be
## configured via command line options for the corresponding commands.
## [...]
sheds:
     toolshed:
           #key: "<TODO>"
           #email: "<TODO>"
           #password: "<TODO>"
     testtoolshed:
           #key: "<TODO>"
           #email: "<TODO>"
           #password: "<TODO>"
     local:
           #key: "<TODO>"
           #email: "<TODO>"
           #password: "<TODO>"

Configure the target Tool Shed

## Planemo Global Configuration File.
## Everything in this file is completely optional - these values can all be
## configured via command line options for the corresponding commands.
## [...]
sheds:
     toolshed:
           key: "xxxxxxxxxmytoolshedAPIkeyxxxxxxx"
           #email: "<TODO>"
           #password: "<TODO>"
     testtoolshed:
           #key: "<TODO>"
           #email: "<TODO>"
           #password: "<TODO>"
     local:
           #key: "<TODO>"
           #email: "<TODO>"
           #password: "<TODO>"

Configure repository

planemo shed_init --name="tandem_repeats_finder_2"
                     --owner="gandres"
                     --description="Tandem Repeats Finder description"
                     --long_description="A long long description."
                     --category="Sequence Analysis"
                     [--remote_repository_url=<URL to .shed.yml on github>]
                     [--homepage_url=<Homepage for tool.>]
categories: [Sequence Analysis]
description: Tandem Repeats Finder description
long_description: A long long description.
<span id="tandem_repeats_finder_2"><i class="fas fa-link" aria-hidden="true"></i> tandem_repeats_finder_2</span>
owner: gandres

Configure repository

Naming conventions

Repository Type Recommended Name Examples
Data Managers data_manager_$name data_manager_bowtie2
Packages package_$name_$version package_aragorn_1_2_36
Tool Suites suite_$name suite_samtools
Tools $name stringle, bowtie2

</small> Planemo will determine the repository type based on these conventions


Configure repository

[...]
auto_tool_repositories:
    name_template: ""
    description_template: " (from the Trinity tool suite)"
suite:
    name: "suite_trinity"
    description: Trinity tools to assemble transcript sequences from Illumina RNA-Seq data.

Check

planemo shed_lint --tools
Linting repository […]/tandem_repeats_finder
Applying linter expansion... CHECK
.. INFO: Included files all found.
Applying linter tool_dependencies_xsd... CHECK
.. INFO: tool_dependencies.xml found and appears to be valid XML
Applying linter tool_dependencies_actions... CHECK
.. INFO: Parsed tool dependencies.
Applying linter repository_dependencies... CHECK
.. INFO: No repository_dependencies.xml, skipping.
Applying linter shed_yaml... CHECK
.. INFO: .shed.yml found and appears to be valid YAML.
Applying linter readme... CHECK
.. INFO: No README found skipping.
+Linting tool […]/tandem_repeats_finder/tandem_repeats_finder_wrapper.xml
[…]

Create repository

planemo shed_create --shed_target testtoolshed

Create repository

planemo shed_create --shed_target testtoolshed
Repository created
cd '[…]/tandem_repeats_finder' && git rev-parse HEAD
Repository tandem_repeats_finder_2 updated successfully.

an image of a toolshed metadata from the planemo uploaded data


Update repository

planemo shed_diff --shed_target testtoolshed

wget -q --recursive -O - 'https://testtoolshed.g2.bx.psu.edu/repository/download?repository_id=6306166e4da9a8bf&changeset_revision=default&file_type=gz' | tar -xzf - -C /tmp/tool_shed_diff_DMcOrR/_testtoolshed_ --strip-components 1
mkdir "/tmp/tool_shed_diff_DMcOrR/_local_"; tar -xzf "/tmp/tmp5ousfg" -C "/tmp/tool_shed_diff_DMcOrR/_local_"; rm -rf /tmp/tmp5ousfg
cd "/tmp/tool_shed_diff_DMcOrR"; diff -r _local_ _testtoolshed_
diff -r _local_/tandem_repeats_finder.xml _testtoolshed_/tandem_repeats_finder.xml
41d40
<     <param name="fileout_label2" type="text" value="Unused" label="none" help="Output name for files" />

Update repository

planemo shed_update --check_diff --shed_target testtoolshed

wget -q --recursive -O - 'https://testtoolshed.g2.bx.psu.edu/repository/download?repository_id=6306166e4da9a8bf&changeset_revision=default&file_type=gz' | tar -xzf - -C /tmp/tool_shed_diff_DMcOrR/_testtoolshed_ --strip-components 1
mkdir "/tmp/tool_shed_diff_DMcOrR/_local_"; tar -xzf "/tmp/tmp5ousfg" -C "/tmp/tool_shed_diff_DMcOrR/_local_"; rm -rf /tmp/tmp5ousfg
cd "/tmp/tool_shed_diff_DMcOrR"; diff -r _local_ _testtoolshed_
diff -r _local_/tandem_repeats_finder.xml _testtoolshed_/tandem_repeats_finder.xml
41d40
<     <param name="fileout_label2" type="text" value="Unused" label="none" help="Output name for files" />
Repository tandem_repeats_finder updated successfully.
Repository metadata updated.

Update repository

planemo shed_update --check_diff --shed_target testtoolshed

tool shed info with revision circled


planemo logo

Publishing to the Tool Shed

planemo shed_init creates a repo, shed_lint checks, and shed_create/shed_update uploads to the TS


planemo logo again

Command-line utilities to assist in building and publishing Galaxy tools.


and again

larger graph with local development, toolshed, conda, and galaxy and various lines between them.


planemo logo again

Checklist (3/3)


References


hands_on Hands-on

picture indicating exercise


Hands-on

The goal

Publish your tool (seqtk_seq) in your local toolshed with Planemo


hands_on Hands-on

Step 1: Start and configure your toolshed


hands_on Hands-on

Step 2: Use planemo to publish your tool


hands_on Hands-on

Step 2: Use planemo to publish your tool

planemo config_init
[...]  local:
#key: "<TODO>"
email: "galaxy@localhost.fr"
password: "azerty"
planemo shed_init --name=seqtk_seq --owner=galaxy --description="common transformation of FASTA/Q" --category="Fasta Manipulation"
planemo shed_lint .
planemo shed_create --shed_target local

hands_on Hands-on

Step 3: Update your tool and then use planemo to update your tool on the Tool Shed


hands_on Hands-on

Step 3: Update your tool and then use planemo to update your tool on the Tool Shed

planemo shed_diff --shed_target local
planemo shed_update --check_diff --shed_target local

Key Points

Thank you!

This material is the result of a collaborative work. Thanks to the Galaxy Training Network and all the contributors! Galaxy Training Network This material is licensed under the Creative Commons Attribution 4.0 International License.