Running Jobs on Remote Resources with Pulsar

name: inverse
layout: true
class: center, middle, inverse

</span></div>

</span></div>

---

# Running Jobs on Remote Resources with Pulsar

<div class="contributors-line">
		
	
<ul class="text-list">
			
			<li>
				<a href="/training-material/hall-of-fame/natefoo/" class="contributor-badge contributor-natefoo"><img src="/training-material/assets/images/orcid.png" alt="orcid logo" width="36" height="36"/><img src="https://avatars.githubusercontent.com/natefoo?s=36" alt="Nate Coraor avatar" width="36" class="avatar" />
    Nate Coraor</a>
			<li>
				<a href="/training-material/hall-of-fame/slugger70/" class="contributor-badge contributor-slugger70"><img src="/training-material/assets/images/orcid.png" alt="orcid logo" width="36" height="36"/><img src="https://avatars.githubusercontent.com/slugger70?s=36" alt="Simon Gladman avatar" width="36" class="avatar" />
    Simon Gladman</a>
			<li>
				<a href="/training-material/hall-of-fame/mvdbeek/" class="contributor-badge contributor-mvdbeek"><img src="https://avatars.githubusercontent.com/mvdbeek?s=36" alt="Marius van den Beek avatar" width="36" class="avatar" />
    Marius van den Beek</a>
			<li>
				<a href="/training-material/hall-of-fame/hexylena/" class="contributor-badge contributor-hexylena"><img src="/training-material/assets/images/orcid.png" alt="orcid logo" width="36" height="36"/><img src="https://avatars.githubusercontent.com/hexylena?s=36" alt="Helena Rasche avatar" width="36" class="avatar" />
    Helena Rasche</a></li>
</ul>

</div>

<div class="footnote" style="bottom: 8em;">
  <i class="far fa-calendar" aria-hidden="true"></i><span class="visually-hidden">last_modification</span> Updated:   
  <i class="fas fa-fingerprint" aria-hidden="true"></i><span class="visually-hidden">purl</span><abbr title="Persistent URL">PURL</abbr>: <a href="https://gxy.io/GTN:S00020">gxy.io/GTN:S00020</a>
</div>

<div class="footnote" style="bottom: 5em;">

<i class="fas fa-file-alt" aria-hidden="true"></i><span class="visually-hidden">text-document</span><a href="slides-plain.html"> Plain-text slides</a> |

<a href="/training-material/topics/admin/tutorials/pulsar/recordings/" class="btn btn-default dropdown-toggle topic-icon" data-toggle="dropdown" aria-expanded="false" title="Latest recordings of this material in the GTN Video Library">
        <i class="fas fa-video" aria-hidden="true"></i><span class="visually-hidden">video</span> Recordings
  </a>

<li><a class="dropdown-item" href="/training-material/topics/admin/tutorials/pulsar/recordings/index.html#lecture-recording-15-february-2021" title="View the recording for this tutorial">
                <i class="fas fa-video" aria-hidden="true"></i><span class="visually-hidden">video</span> Lecture (February 2021) - 14m</a>
    </li>
      
    
    <li><a class="dropdown-item" href="/training-material/topics/admin/tutorials/pulsar/recordings/" title="View all recordings for this tutorial">
                <i class="fas fa-video" aria-hidden="true"></i><span class="visually-hidden">video</span> View All</a>
    </li>

</ul>

</div>

<div class="footnote" style="bottom: 2em;">
    <strong>Tip: </strong>press <kbd>P</kbd> to view the presenter notes
    | <i class="fa fa-arrows" aria-hidden="true"></i><span class="visually-hidden">arrow-keys</span> Use arrow keys to move between slides

</div>

???
Presenter notes contain extra information which might be useful if you intend to use these slides for teaching.

Press `P` again to switch presenter notes off

Press `C` to create a new window where the same presentation will be displayed.
This window is linked to the main window. Changing slides on one will cause the
slide to change on the other.

Useful when presenting.

---

## Requirements

Before diving into this slide deck, we recommend you to have a look at:

- [Galaxy Server administration](/training-material/topics/admin)
    - Ansible: [<i class="fab fa-slideshare" aria-hidden="true"></i><span class="visually-hidden">slides</span> slides](/training-material/topics/admin/tutorials/ansible/slides.html) - [<i class="fas fa-laptop" aria-hidden="true"></i><span class="visually-hidden">tutorial</span> hands-on](/training-material/topics/admin/tutorials/ansible/tutorial.html)
    - Galaxy Installation with Ansible: [<i class="fab fa-slideshare" aria-hidden="true"></i><span class="visually-hidden">slides</span> slides](/training-material/topics/admin/tutorials/ansible-galaxy/slides.html) - [<i class="fas fa-laptop" aria-hidden="true"></i><span class="visually-hidden">tutorial</span> hands-on](/training-material/topics/admin/tutorials/ansible-galaxy/tutorial.html)

- A server/VM on which to deploy Pulsar

---

### <i class="far fa-question-circle" aria-hidden="true"></i><span class="visually-hidden">question</span> Questions

- How does pulsar work?

- How can I deploy it?

---

### <i class="fas fa-bullseye" aria-hidden="true"></i><span class="visually-hidden">objectives</span> Objectives

- Have an understanding of what Pulsar is and how it works

- Install and configure a Pulsar server on a remote linux machine

- Be able to get Galaxy to send jobs to a remote Pulsar server

---

# What are heterogenous compute resources?

Differences in:
- Operating system or version
- Users/groups
- Data accessibility
- Administrative control
- Physical Location (i.e. Cities)

Galaxy expects:
- One OS, version (dependencies)
- Shared filesystem w/ fixed paths

---
# Example - Australia

![australia_locations.png](../../images/australia_locations.png)

---
# Partial solution - CLI job runner

SSH to remote, submit jobs with CLI `sbatch`, `qsub`, etc.

Still depends on shared FS

---
# Pulsar

![pulsar_logo.png](../../images/pulsar_logo.png)

Galaxy's remote job management system

* Can run jobs on any(?) OS including Windows

* Multiple modes of operation for every environment

---
# Pulsar - Architecture

* Pulsar server runs on remote resource (e.g. cluster head node)

* Galaxy Pulsar job runner is Pulsar client

* Communication is via HTTP or AMQP, language is JSON

* File transport is dependent on communication method

---
# Pulsar - Architecture

![pulsar_schematic.png](../../images/pulsar_schematic.png)

---
# Pulsar Transports - RESTful

Pulsar server listens over HTTP(S)

Pulsar client (Galaxy) initiates connections to Pulsar server

Good for:
- Environments where firewall, open ports are not concerns
- No external dependencies (AMQP server)

---
# Pulsar Transports - AMQP

Pulsar server and client connect to AMQP server

Good for:
- Firewalled/NATted remote compute
- Networks w/ bad connectivity

---
# Pulsar Transports - Embedded

Galaxy runs Pulsar server internally

Good for:
- Manipulating paths
- Copying input datasets from non-shared filesystem

---
# Pulsar - Job file staging

Pulsar can be configured to *push* or *pull* when using RESTful:
- Push
  - Galaxy sends job inputs, metadata to Pulsar over HTTP
  - Upon completion signal from Pulsar, Galaxy pulls from Pulsar over HTTP
- Pull
  - Upon setup signal, Pulsar pulls job inputs, metadata from Galaxy over HTTP
  - Upon completion, Pulsar pushes to Galaxy over HTTP

Pulsar can use libcurl for more robust transfers with resume capability

AMQP is pull-only because Pulsar does not run HTTP server

---
# Pulsar - Dependency management

Pulsar does not provide Tool Shed tool dependency management. But:
- It has a similar dependency resolver config to Galaxy
- It can auto-install **conda** dependencies
- It can use containers too!

---
# Pulsar - Job management

Pulsar "managers" provide job running interfaces:
- `queued_python`: Run locally on the Pulsar server
- `queued_drmaa`: Run on a cluster with DRMAA
- `queued_cli`: Run on a cluster with local `qsub`, `sbatch`, etc.
- `queued_condor`: Run on HTCondor

---
# Pulsar Australia

![pulsar_australia.png](../../images/pulsar_australia.png)

---

# Resources

* Pulsar Read-the-docs
    * [https://pulsar.readthedocs.io/en/latest/index.html](https://pulsar.readthedocs.io/en/latest/index.html)
* Pulsar on galaxyproject.org
    * [https://galaxyproject.org/admin/config/pulsar/](https://galaxyproject.org/admin/config/pulsar/)
* Pulsar Github
    * [https://github.com/galaxyproject/pulsar](https://github.com/galaxyproject/pulsar)
* Pulsar Ansible
    * [https://github.com/galaxyproject/ansible-pulsar](https://github.com/galaxyproject/ansible-pulsar)

---
### <i class="fas fa-key" aria-hidden="true"></i><span class="visually-hidden">keypoints</span> Key points

- Pulsar allows you to easily add geographically distributed compute resources into your Galaxy instance

- It also works well in situations where the compute resources cannot share storage pools.

---

## Thank You!

This material is the result of a collaborative work. Thanks to the [Galaxy Training Network](https://training.galaxyproject.org) and all the contributors!

<div class="contributors-line">
		
<table class="contributions">
	
	<tr>
		<td><abbr title="These people wrote the bulk of the tutorial, they may have done the analysis, built the workflow, and wrote the text themselves.">Author(s)</abbr></td>
		<td>
			<a href="/training-material/hall-of-fame/natefoo/" class="contributor-badge contributor-natefoo"><img src="/training-material/assets/images/orcid.png" alt="orcid logo" width="36" height="36"/><img src="https://avatars.githubusercontent.com/natefoo?s=36" alt="Nate Coraor avatar" width="36" class="avatar" />
    Nate Coraor</a><a href="/training-material/hall-of-fame/slugger70/" class="contributor-badge contributor-slugger70"><img src="/training-material/assets/images/orcid.png" alt="orcid logo" width="36" height="36"/><img src="https://avatars.githubusercontent.com/slugger70?s=36" alt="Simon Gladman avatar" width="36" class="avatar" />
    Simon Gladman</a><a href="/training-material/hall-of-fame/mvdbeek/" class="contributor-badge contributor-mvdbeek"><img src="https://avatars.githubusercontent.com/mvdbeek?s=36" alt="Marius van den Beek avatar" width="36" class="avatar" />
    Marius van den Beek</a><a href="/training-material/hall-of-fame/hexylena/" class="contributor-badge contributor-hexylena"><img src="/training-material/assets/images/orcid.png" alt="orcid logo" width="36" height="36"/><img src="https://avatars.githubusercontent.com/hexylena?s=36" alt="Helena Rasche avatar" width="36" class="avatar" />
    Helena Rasche</a>
		</td>
	</tr>

<tr class="reviewers">
		<td><abbr title="These people reviewed this material for accuracy and correctness">Reviewers</abbr></td>
		<td>
			<a href="/training-material/hall-of-fame/hexylena/" class="contributor-badge contributor-badge-small contributor-hexylena"><img src="https://avatars.githubusercontent.com/hexylena?s=36" alt="Helena Rasche avatar" width="36" class="avatar" /></a><a href="/training-material/hall-of-fame/natefoo/" class="contributor-badge contributor-badge-small contributor-natefoo"><img src="https://avatars.githubusercontent.com/natefoo?s=36" alt="Nate Coraor avatar" width="36" class="avatar" /></a><a href="/training-material/hall-of-fame/nsoranzo/" class="contributor-badge contributor-badge-small contributor-nsoranzo"><img src="https://avatars.githubusercontent.com/nsoranzo?s=36" alt="Nicola Soranzo avatar" width="36" class="avatar" /></a><a href="/training-material/hall-of-fame/martenson/" class="contributor-badge contributor-badge-small contributor-martenson"><img src="https://avatars.githubusercontent.com/martenson?s=36" alt="Martin Čech avatar" width="36" class="avatar" /></a><a href="/training-material/hall-of-fame/slugger70/" class="contributor-badge contributor-badge-small contributor-slugger70"><img src="https://avatars.githubusercontent.com/slugger70?s=36" alt="Simon Gladman avatar" width="36" class="avatar" /></a><a href="/training-material/hall-of-fame/shiltemann/" class="contributor-badge contributor-badge-small contributor-shiltemann"><img src="https://avatars.githubusercontent.com/shiltemann?s=36" alt="Saskia Hiltemann avatar" width="36" class="avatar" /></a></td>
	</tr>

</table>

</div>

</div>

Tutorial Content is licensed under <a rel="license" href="http://creativecommons.org/licenses/by/4.0/">Creative Commons Attribution 4.0 International License</a>.<br/>