View markdown source on GitHub

Reference Genomes in Galaxy

Contributors

Daniel Blankenberg, Simon Gladman
last_modification Last modification: Apr 6, 2021

Overview

.large[


Built in Data

List_of_data.png


Data, what data?

.large[


Overview

.large[


Data schematics in Galaxy

schematic


Using reference data in a tool

bwa.xml

<conditional name="reference_source">
      <param name="reference_source_selector" type="select" label="Will you select a reference genome from your history or use a built-in index?" help="Built-ins were indexed using default options. See 'Indexes' section of help below">
        <option value="cached">Use a built-in genome index</option>
        <option value="history">Use a genome from history and build index</option>
      </param>
      <when value="cached">
        <param name="ref_file" type="select" label="Using reference genome" help="Select genome from the list">
          <options from_data_table="bwa_mem_indexes">
            <filter type="sort_by" column="2" />
            <validator type="no_options" message="No indexes are available" />
          </options>
          <validator type="no_options" message="A built-in reference genome is not available for the build associated with the selected input file"/>
        </param>
      </when>
      <when value="history">


Where are the data tables?

tool_data_table_conf.xml

(Usually located in galaxy/config/)

  <tables>
    <!-- Locations of indexes in the BWA mapper format -->
    <table name="bwa_mem_indexes" comment_char="#" allow_duplicate_entries="False">
      <columns>value, dbkey, name, path</columns>
      <file path="tool-data/bwa_index.loc" />
    </table>
  </tables>

“loc” files - Short for location!

bwa_index.loc

#
#<unique_build_id>   <dbkey>   <display_name>   <file_path>
#
bosTau7 bosTau7 Cow (bosTau7)   /genomes/bosTau7/bwa_mem_index/bosTau7/bosTau7.fa
ce10    ce10    C. elegans (ce10)       /genomes/ce10/bwa_mem_index/ce10/ce10.fa
danRer7 danRer7 Zebrafish (danRer7)     /genomes/danRer7/bwa_mem_index/danRer7/danRer7.fa
dm3     dm3     D. melanogaster Apr. 2006 (BDGP R5/dm3) (dm3)   /genomes/dm3/bwa_mem_index/dm3/dm3.fa
hg19    hg19    Human (hg19)    /genomes/hg19/bwa_mem_index/hg19/hg19.fa
hg38    hg38    Human (hg38)    /genomes/hg38/bwa_mem_index/hg38/hg38.fa
mm10    mm10    Mouse (mm10)    /genomes/mm10/bwa_mem_index/mm10/mm10.fa

Overview

.large[


Some Problems!

.large[


Typical conversation

.middle[ref-problem-1.png]


Typical conversation

.middle[ref-problem-2.png]


Typical conversation

.middle[ref-problem-3.png]


Typical conversation

.middle[ref-problem-4.png]


Other concerns

.large[


Overview

.large[


Data Managers

.large[


Data Managers

.large[


Special class of Galaxy tool

Looks just like a normal Galaxy tool!

Data-manager-ui.png


What does it do?

The output of the data manager is a JSON description of the new data table entry

data_table_JSON.png

This gets turned into a new data table entry

data_table_entry.png

The index files themselves get placed in the appropriate location.


Data Managers Admin

.large[


Data Managers Admin

.large[


Resources / further reading

.large[

https://galaxyproject.org/admin/tools/data-managers/ ]


Exercise Time!


Overview

.large[


There’s a lot of reference data

.large[ (and it’s hard to keep up with) ] ref_data_prob_flow.png


CernVM-FS to the rescue


.widen_image[ cvmfs_server_distribution.png ]


CVM-FS Global Structure

.widen_image[ cvmfs_global_structure.png ]


Exercise #2:

.large[ Connect our instances to CVMFS for reference data ]


Thank you!

This material is the result of a collaborative work. Thanks to the Galaxy Training Network and all the contributors! page logo This material is licensed under the Creative Commons Attribution 4.0 International License.