View markdown source on GitHub

Running Jobs on Remote Resources with Pulsar

Contributors

AvatarNate Coraor AvatarSimon Gladman AvatarMarius van den Beek AvatarHelena Rasche

Questions

Objectives

Requirements

last_modification Last modification: Apr 6, 2021

What are heterogenous compute resources?

Differences in:

Galaxy expects:


Example - Australia

australia_locations.png


Partial solution - CLI job runner

SSH to remote, submit jobs with CLI sbatch, qsub, etc.

Still depends on shared FS


Pulsar

pulsar_logo.png

Galaxy’s remote job management system


Pulsar - Architecture


Pulsar - Architecture

pulsar_schematic.png


Pulsar Transports - RESTful

Pulsar server listens over HTTP(S)

Pulsar client (Galaxy) initiates connections to Pulsar server

Good for:


Pulsar Transports - AMQP

Pulsar server and client connect to AMQP server

Good for:


Pulsar Transports - Embedded

Galaxy runs Pulsar server internally

Good for:


Pulsar - Job file staging

Pulsar can be configured to push or pull when using RESTful:

Pulsar can use libcurl for more robust transfers with resume capability

AMQP is pull-only because Pulsar does not run HTTP server


Pulsar - Dependency management

Pulsar does not provide Tool Shed tool dependency management. But:


Pulsar - Job management

Pulsar “managers” provide job running interfaces:


Pulsar Australia

pulsar_australia.png


Resources


Key Points

Thank you!

This material is the result of a collaborative work. Thanks to the Galaxy Training Network and all the contributors! page logo This material is licensed under the Creative Commons Attribution 4.0 International License.