View markdown source on GitHub

Galaxy and Celery

Contributors

Author(s) AvatarMira Kuntz
Editor(s) orcid logoAvatarHelena Rasche
last_modification Last modification: Nov 28, 2022

Can you eat it

.pull-left[ Celery is an asynchronous distributed task queue.

It consists of:

It’s written in Python but supports multiple other languages via webhooks. ]

.pull-right[ celery logo, a cartoon of a piece of celery ]


So many features

.pull-left[

.pull-right[ celery logo, a cartoon of a piece of celery, now with the text celery next to it ]


How Celery Can Improve Galaxy


.pull-left[ The Problem

The Galaxy Server should respond quickly to every request. Gunicorn Workers should not do everything.

The Solution

Queue asynchronous tasks with Celery (happens currently internal on the same node)

The Improvement

Send the tasks to an external Celery-Cluster ]

.pull-right[ galaxy logo next to celery logo, with two purple hearts ]


A workflow diagram is shown with logos and arrows. Galaxy on the left sends tasks to rabbit MQ. Celery fetches tasks from Rabbit MQ and proceses them. Then celery sends results to the backend database. Finally Galaxy fetches those same results back from the backend.


What is Celery Used For


.pull-left [

What we need

.pull-right[ Collection of three logos, Galaxy, RabbitMQ, and celery ]


.pull-left[ screenshot of galaxy documentation page, linked in next section ] .pull-right[


.pull-left[ Screenshots of Galaxy, RabbitMQ, and Flower ]

.pull-right[

What’s currently easy


To Be Completed


Thank you!

This material is the result of a collaborative work. Thanks to the Galaxy Training Network and all the contributors! page logo This material is licensed under the Creative Commons Attribution 4.0 International License.