View markdown source on GitHub

Docker and Galaxy

Contributors

Bérénice Batut, Björn Grüning

Questions

Objectives

last_modification Last modification: Jul 26, 2021

Preparing Admin Training VMs

If you’re falling along on Galaxy Admin Training VMs, you will need to setup Docker.

You can modify your Ansible setup to configure this as described at bit.ly/gxadmin-docker-playbook.

You will also need to stop Galaxy running on the VM so that you can run Galaxy in a container.

sudo supervisorctl stop galaxy

Speaker Notes

TODO: Remove this slide after the training, it isn’t general.


Preparation

$ docker pull bgruening/galaxy-stable:18.05
$ git clone https://github.com/bgruening/docker-galaxy-stable.git

Speaker Notes

Here you will learn the internals of the Docker Galaxy Image. We will show you tips and tricks on how to run the Galaxy Docker Image successfully in production, how to manage updates and how to bind the container to a cluster scheduler. Moreover, you will learn how to create your own Galaxy flavour mixing a variety of different tools and visualisations


Why Docker?
What is it?

Image of a blue whale with containers on its back.


Deployment issues of bioinformatics tools

Impact on usability and reproducibility


Deployment issue

Galaxy is a box with many dependencies like node, java, nginx, python, uwsgi, anaconda. On the right knime is another box with java and eclipse dependencies. Lines show the multiplicity of tools and dependencies trying to run on multiple environments with windows/osx/linux desktops, servers, and the cloud.


Deployment issue

Matrix from Hell

Galaxy and knime form the y axis of a table with a lot of question marks about which environments support them.


Transport Pre 1960

Icons for multiple types of goods shown multiple methods for transport and storing.


Transport Pre 1960

Matrix from Hell

A similar table full of question marks between goods and shipment methods is shown.


Intermodal shipping container

A blue shipping container is shown solving the transport problem by being a one size container to fit all sorts of goods, on all sorts of transports.

Speaker Notes

A standard container


Docker

Docker logo on a blue shipping container solves the galaxy/knime deployment problem across environments.

Speaker Notes

Docker = An engine that encapsulated any tools as a


A Docker container?

A schematic with the kernal below and two containers is shown, each container has different stacks.

Lightweight and Open

Speaker Notes


Virtual machines vs Containers

A cartoon comparing containers vs VMs. On the left the VMs are presented as a layer of server, host os, and hypervisor (type 2) followed by several tall pillars of guest OS, bin/libs and finally their respective apps. Everything is siloed. On the right is containers where the layers are server, host os, and docker engine. Above those are very short pillars of each different app, and when duplicate apps are run, the bin/libs are shown as shared. Text reads: containers are isolated but share OS and where appropriate, bins/libraries.

Containers more portable and efficient

Speaker Notes

Similar resource isolation and allocation benefits

but a different architectural approach


How to use Docker?

A cartoon of a whale with shipping containers atop it.


The client

A command line is explained, docker is the program you're using, run is the subcommand that creates and runs a container, and hello-world is the image you want to load.

Speaker Notes

Docker Engine = the core Docker technology on your computer

Client: to talk with Docker

docker --help


Containers? Images?

A single container labelled galaxy with postgres, nginx, and python inside is shown pointing to three other galaxy containers.

Speaker Notes

Image:

Container


How to get images?

Images are stored in an image registry which are downloaded with docker pull onto a docker host, where you can run multiple instances of galaxy.

Speaker Notes

Docker Registry:

to stores and distribute Docker images

Example: docker hub, quay.io


docker pull

$ docker pull hello-world
Using default tag: latest
latest: Pulling from library/hello-world
c04b14da8d14: Pull complete
Digest: sha256:0256e8a36e2070f7bf2d0b0763dbabdd67798512411de4cdcf9431a1feb60fd9
Status: Downloaded newer image for hello-world:latest

Creation of containers

Images are stored in an image registry which are downloaded with docker pull onto a docker host, where you can run multiple instances of galaxy.


docker run

$ docker run hello-world

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker Hub account:
 https://hub.docker.com

For more examples and ideas, visit:
 https://docs.docker.com/engine/userguide/

docker run

$ docker run docker/whalesay cowsay Galaxy
Unable to find image 'docker/whalesay:latest' locally
latest: Pulling from docker/whalesay
e190868d63f8: Pull complete
909cd34c6fd7: Pull complete
0b9bfabab7c1: Pull complete
a3ed95caeb02: Pull complete
00bf65475aba: Pull complete
c57b6bcc83e3: Pull complete
8978f6879e2f: Pull complete
8eed3712d2cf: Pull complete
Digest: sha256:178598e51a26abbc958b8a2e48825c90bc22e641de3d31e18aaf55f3258ba93b
Status: Downloaded newer image for docker/whalesay:latest
 ________
< Galaxy >
 --------
    \
     \
      \
                    ##        .
              ## ## ##       ==
           ## ## ## ##      ===
       /""""""""""""""""___/ ===
  ~~~ {~~ ~~~~ ~~~ ~~~~ ~~ ~ /  ===- ~~~
       \______ o          __/
        \    \        __/
          \____\______/

Speaker Notes


docker run

$ docker run --help

Usage:  docker run [OPTIONS] IMAGE [COMMAND] [ARG...]

Run a command in a new container

  -a, --attach=[]                 Attach to STDIN, STDOUT or STDERR
  --add-host=[]                   Add a custom host-to-IP mapping (host:ip)
  --cpu-shares                    CPU shares (relative weight)
  ...
  -d, --detach                    Run container in background and print container ID
  -e, --env=[]                    Set environment variables
  --entrypoint                    Overwrite the default ENTRYPOINT of the image
  --env-file=[]                   Read in a file of environment variables
  ...
  -h, --hostname                  Container host name
  -i, --interactive               Keep STDIN open even if not attached
  --name                          Assign a name to the container
  --net=default                   Connect a container to a network
  ...
  -P, --publish-all               Publish all exposed ports to random ports
  -p, --publish=[]                Publish a container's port(s) to the host
  --privileged                    Give extended privileges to this container
  --rm                            Automatically remove the container when it exits
  -t, --tty                       Allocate a pseudo-TTY
  -v, --volume=[]                 Bind mount a volume
  ...

Run an interactive container

$ docker run -t -i docker/whalesay
root@7de97f8dd5eb:/cowsay#
root@7de97f8dd5eb:/cowsay# cowsay Galaxy
 ________
< Galaxy >
 --------
    \
     \
      \
                    ##        .
              ## ## ##       ==
           ## ## ## ##      ===
       /""""""""""""""""___/ ===
  ~~~ {~~ ~~~~ ~~~ ~~~~ ~~ ~ /  ===- ~~~
       \______ o          __/
        \    \        __/
          \____\______/
root@7de97f8dd5eb:/cowsay#

Control during startup

$ docker run -i -t
    -p 8080:80 -p 8021:21 -p 9002:9002
    --privileged=true
    -e "NONUSE=reports"
    -e "GALAXY_CONFIG_ADMIN_USERS=albert@einstein.gov"
    -e "GALAXY_CONFIG_MASTER_API_KEY=83D4jaba7330aDKHkakjGa937"
    -e "GALAXY_CONFIG_BRAND='My own Galaxy flavour'"
    -e "GALAXY_LOGGING=full"

    bgruening/galaxy-stable:18.05

Speaker Notes

possibility to add many different parameters to control the container when starting it


Management of data

$ mkdir data
$ docker run docker/whalesay cowsay Galaxy > data/cowsay
$ more data/cowsay
 ________
< Galaxy >
 --------
    \
     \
      \
                    ##        .
              ## ## ##       ==
           ## ## ## ##      ===
       /""""""""""""""""___/ ===
  ~~~ {~~ ~~~~ ~~~ ~~~~ ~~ ~ /  ===- ~~~
       \______ o          __/
        \    \        __/
          \____\______/
$ docker run -t -i docker/whalesay
root@f4fa8ed32ef8:/cowsay# ls
ChangeLog  INSTALL  LICENSE  MANIFEST  README  Wrap.pm.diff  cows  cowsay  cowsay.1  install.pl  install.sh  pgp_public_key.txt
root@f4fa8ed32ef8:/cowsay# cowsay Hello Galaxy > cowsay2

Can we access the cowsay file inside the container?
And the cowsay2 file outside the container?


Management of data

A container is shown with an arrow pointing to data and a big x over it.

Speaker Notes

A container is closed


Management of data

The same as before, but inside the container it wants to create a new file in that same folder but it cannot.

Speaker Notes

A container is closed


Data volume

A data volume with two files is shown as a folder, now this is moutned inside two containers.

Speaker Notes

Data volume

Data volumes designed to persist data, independent of the container’s life cycle:


Data volume

Another command line explained, now docker run -v host_dir:container_dir ..., where host_dir is the absolute path or data volume name, and container_dir is where it should be mounted.

Speaker Notes

How to get a data volume?


Data volume

$ ls data/
cowsay_Galaxy
$ docker run -t -i -v $PWD/data:/data docker/whalesay
root@f4fa8ed32ef8:/cowsay# ls /data
cowsay_Galaxy
root@f4fa8ed32ef8:/cowsay# cowsay Galaxy2 > /data/cowsay_Galaxy2
root@f4fa8ed32ef8:/cowsay# ls /data
cowsay_Galaxy  cowsay_Galaxy2
root@f4fa8ed32ef8:/cowsay# exit
$ ls data/
cowsay_Galaxy   cowsay_Galaxy2

Execution of commands
inside a running container

The schematic of a docker host with three galaxies is running, docker exec points to one of the galaxy containers.

Speaker Notes

Run a command in a running container


docker exec

$ docker run -d docker/whalesay /bin/sh -c "while true; do sleep 1; done"
7179e85085ef14634f8b50f908a255707743dec0a5d1fd7fb3cd9036334d5177
$ docker exec 7179e85085ef14634f8b50f908a255707743dec0a5d1fd7fb3cd9036334d5177 \
cowsay Galaxy
 ________
< Galaxy >
 --------
    \
     \
      \
                    ##        .
              ## ## ##       ==
           ## ## ## ##      ===
       /""""""""""""""""___/ ===
  ~~~ {~~ ~~~~ ~~~ ~~~~ ~~ ~ /  ===- ~~~
       \______ o          __/
        \    \        __/
          \____\______/

Stop/Start containers

Again the same graphic, but now one of the galaxy containers becomes stopped through the docker stop command and turns grey. Docker start brings it back to running.

Note: Sending SIGTERM and then SIGKILL after a grace period


docker stop & docker start

$ docker stop 7179e85085ef14634f8b50f908a255707743dec0a5d1fd7fb3cd9036334d5177
7179e85085ef14634f8b50f908a255707743dec0a5d1fd7fb3cd9036334d5177
$
$ docker exec 7179e85085ef14634f8b50f908a255707743dec0a5d1fd7fb3cd9036334d5177 \
cowsay Galaxy
Error response from daemon: Container 7179e85085ef14634f8b50f908a255707743dec0a5d1fd7fb3cd9036334d5177 \
is not running
$
$ docker start 7179e85085ef14634f8b50f908a255707743dec0a5d1fd7fb3cd9036334d5177
7179e85085ef14634f8b50f908a255707743dec0a5d1fd7fb3cd9036334d5177
$
$ docker exec 7179e85085ef14634f8b50f908a255707743dec0a5d1fd7fb3cd9036334d5177 \
cowsay Galaxy
 ________
< Galaxy >
 --------
    \
     \
      \
                    ##        .
              ## ## ##       ==
           ## ## ## ##      ===
       /""""""""""""""""___/ ===
  ~~~ {~~ ~~~~ ~~~ ~~~~ ~~ ~ /  ===- ~~~
       \______ o          __/
        \    \        __/
          \____\______/

View all containers

docker ps shows a box around the previous graphic and shows running and stopped container.


docker ps

$ docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS               NAMES
7179e85085ef        docker/whalesay     "/bin/sh -c 'while tr"   12 minutes ago      Up 2 seconds                            agitated_lovelace
$
$ docker ps -a
CONTAINER ID        IMAGE                  COMMAND                  CREATED             STATUS                        PORTS               NAMES
7de97f8dd5eb        docker/whalesay        "/bin/bash"              4 minutes ago       Exited (0) About a minute ago                       trusting_swanson
9218bbee9c48        docker/whalesay        "/bin/bash"              5 minutes ago       Exited (0) 4 minutes ago                            evil_swirles
7179e85085ef        docker/whalesay        "/bin/sh -c 'while tr"   13 minutes ago      Up 55 seconds                                       agitated_lovelace
ad275579c454        ubuntu                 "/bin/sh -c 'while tr"   15 minutes ago      Exited (137) 13 minutes ago                         condescending_mestorf
66179c4d16da        ubuntu                 "/bin/bash"              About an hour ago   Exited (130) 15 minutes ago                         determined_pasteur
27386c8b69b3        ubuntu                 "/bin/sh"                About an hour ago   Exited (0) About an hour ago                        lonely_ramanujan
4cfefa19e6fa        docker/whalesay        "/bin/bash"              About an hour ago   Exited (0) About an hour ago                        thirsty_chandrasekhar
82687eb94ab9        docker/whalesay        "cowsay Galaxy"          2 hours ago         Exited (0) 2 hours ago                              fervent_babbage
6dbabb9384ad        tmp-bioconda-builder   "/usr/local/bin/tini "   8 days ago          Exited (0) 7 days ago                               tender_bhaskara
5d6f09b94727        tmp-bioconda-builder   "/usr/local/bin/tini "   8 days ago          Exited (0) 8 days ago                               jolly_brattain
4e6f38b4c34c        tmp-bioconda-builder   "/usr/local/bin/tini "   8 days ago          Created                                             angry_colden
b3e6c7412a75        tmp-bioconda-builder   "/usr/local/bin/tini "   8 days ago          Created                                             desperate_visvesvaraya
1ec56c9e37f8        tmp-bioconda-builder   "/usr/local/bin/tini "   8 days ago          Created                                             hopeful_khorana
2b129d00eb10        tmp-bioconda-builder   "/usr/local/bin/tini "   8 days ago          Created                                             gigantic_ptolemy
da45ab698f58        fb77c13d04c0           "/usr/local/bin/tini "   13 days ago         Exited (0) 13 days ago                              jovial_yalow
48dc3ed4e173        fb77c13d04c0           "/usr/local/bin/tini "   13 days ago         Created                                             focused_ritchie
e9195b6512dd        a2107450fdf2           "/usr/local/bin/tini "   2 weeks ago         Created                                             thirsty_bardeen

Note: Name of containers and id


Creation of a new image

A dockerfile built with docker build builds a new container on the local docker host.

Speaker Notes

Need a Dockerfile. I will describe it later. Here it is just to give you a broad overview of the principles


View all images

Docker image shows a list of all the images on the docker host that can be run.


Push your image on a registry

Docker push is shown pushing the locally built container to the image registry.


hands_on Hands on! (1)

Play with quay.io/biocontainers/samtools:1.3--1

  1. Get the image
  2. Launch samtools container interactively
  3. Print the help page for samtools
  4. Launch a samtools container in detach mode
  5. Check if it exists and find its name
  6. Stop it and restart it
  7. Print the help page using this container

hands_on Hands on! (2)

Play with quay.io/biocontainers/samtools:1.3--1

  1. Create a local samtool_dir directory
  2. Launch an interactive container with a volume pointing at the samtool_dir directory
  3. Add a toy sam file into the local samtool_dir directory
  4. Check if the existence of the file from inside the container
  5. Visualize the content of the toy file with samtools view commands
  6. Generate stats of the toy file into a toy_stat file into the local samtool_dir directory

Speaker Notes

data volume is the way to make your tool connecting


How to containerize your tools?

Cartoon of a docker container

Speaker Notes

Now I am sure all of you want to containerize his tool :)


How to create your image?

Cartoon showing a dockerfile building a container with docker build.


A Dockerfile?

A text document that contains all the commands a user could call on the command line to assemble an image

# Comment
INSTRUCTION arguments

.footnote[Best practices]

Speaker Notes

The paper describing the recipe for you container

Review of the commands


Dockerfile & Layers

dockerfile with steps like run and add and set user and command are shown inverted as the layers of an image, each building on the last.


But Don’t Do it Manually, Use Biocontainers

Tool Dependencies and Containers


How to integrate Galaxy inside Docker Container?


Galaxy Docker Container

A Docker launching a Galaxy instance and

docker run -i -t -p 8080:80 bgruening/galaxy-stable:18.05

Visit your machine on port 8080 to see Galaxy running.

https://github.com/bgruening/docker-galaxy-stable


hands_on Hands on!

Launch a Galaxy Docker Container and try to


Galaxy + SLURM =

https://github.com/bgruening/docker-galaxy-stable/tree/dev/docs/Running_jobs_outside_of_the_container.md


Docker flavours

Schematic of multiple docker containers and how they layer upon one another. ubuntu forms the base, and requirements builds on that, then galaxy-stable on top of that.


Galactic Flavours


Create your own Flavour

# Galaxy - My own flavour
#
# VERSION       0.1

FROM bgruening/galaxy-stable:18.05

MAINTAINER Björn A. Grüning, bjoern.gruening@gmail.com

ENV GALAXY_CONFIG_BRAND deepTools

# Adding the tool definitions to the container
ADD my_tool_list.yml $GALAXY_ROOT/my_tool_list.yml

# Install deepTools
RUN install-tools $GALAXY_ROOT/my_tool_list.yml

The tools file

galaxy_instance: http://localhost:8080
tools:
- name: fastqc
  owner: devteam
  tool_panel_section_id: cshl_library_information
  tool_shed_url: https://toolshed.g2.bx.psu.edu
  #revisions:

hands_on Hands on!

Create a Galaxy Flavour


Galaxy Flavor Generator

Website used to build custom flavours of galaxy docker containers

https://github.com/galaxyFlavorsGenerators/galaxyFlavorGenerator


Key Points

Thank you!

This material is the result of a collaborative work. Thanks to the Galaxy Training Network and all the contributors! page logo This material is licensed under the Creative Commons Attribution 4.0 International License.