View markdown source on GitHub

Galaxy and Celery

Contributors

Author(s) orcid logoMira Kuntz avatar Mira Kuntz
Editor(s) orcid logoHelena Rasche avatar Helena Rasche
last_modification Published: Jul 16, 2022
last_modification Last Updated: Apr 16, 2023

Can you eat it?

.pull-left[ Celery is an asynchronous distributed task queue.

It consists of:

It’s written in Python, multiple other languages are supported. ]

.pull-right[ celery logo, a cartoon of a piece of celery ]


So many features

.pull-left[

.pull-right[ celery logo, a cartoon of a piece of celery, now with the text celery next to it ]


How does Celery improve Galaxy?


The Problem

The Galaxy Server should respond quickly to every request.
While this is easily possible for small instances with few users,
when scaling to thousands of users and millions of jobs,
without Celery, Gunicorn and the job handlers spend much of their time on I/O bound side tasks,
for example packing a zip for history export.

This leads to slow responses and scheduling.


.pull-left[ The Solution

Queue asynchronous tasks with Celery
on a different node or even a whole cluster…

]

.pull-right[ galaxy logo next to celery logo, with two purple hearts ]


A workflow diagram is shown with logos and arrows. Galaxy on the left sends tasks to rabbit MQ. Celery fetches tasks from Rabbit MQ and proceses them. Then celery sends results to the backend database. Finally Galaxy fetches those same results back from the backend.


How does the magic work?


What is Celery used for


.pull-left[

What do you need to enable Celery?

.pull-right[ Collection of three logos, Galaxy, RabbitMQ, and celery ]


.pull-left[ ]

.pull-right[

Where to get Celery?


.pull-left[ screenshot of galaxy documentation page, linked in next section ] .pull-right[

How to set the Galaxy Config


.pull-top[

How to monitor Celery

In the Flower dashboard you can monitor your workers live.
The screenshot below shows all registered workers and their status. ]

.pull-bottom[ screenshot of the flower dashboard showing a bunch of workers ]


.pull-top[

How to monitor Celery Tasks

In the ‘Tasks’ tab, you can click on each individual task and see all its details, like
args, timestamp, worker, result and stack trace if it errored. ]

.pull-bottom[ screenshot of a simple flower task ]


Take Home Message


Thank you!

This material is the result of a collaborative work. Thanks to the Galaxy Training Network and all the contributors! page logo Tutorial Content is licensed under Creative Commons Attribution 4.0 International License.