View markdown source on GitHub

uWSGI

Contributors

Nate Coraor, Simon Gladman
last_modification Last modification: Apr 6, 2021

What is uWSGI?

…and the rest of the kitchen sink

Speaker Notes


Why do we need uWSGI?

Speaker Notes


What is uWSGI used for?

.left[In a production Galaxy server:]

Speaker Notes


Why uWSGI?

Because Python can’t multithread.

Python GIL

.footnote[Image credit: Dariusz Fryta]

Speaker Notes


Why uWSGI?

Speaker Notes


uWSGI

Speaker Notes


uWSGI Configuration File

uWSGI configuration is performed in the uwsgi key of galaxy.yml.

galaxy.yml.sample is never consulted.

.left[If using run.sh:]

Galaxy servers deployed with Ansible do not use run.sh and should fully specify their uWSGI configuration.

Speaker Notes


uWSGI default command line

Without a config file, the uWSGI command line is:

$ /home/nate/galaxy/.venv/bin/python3 .venv/bin/uwsgi \
    --module 'galaxy.webapps.galaxy.buildapp:uwsgi_app()' \
    --virtualenv /home/nate/galaxy/.venv --pythonpath lib \
    --threads 4 --http localhost:8080 \
    --static-map /static=/home/nate/galaxy/static --die-on-term \
    --hook-master-start 'unix_signal:2 gracefully_kill_them_all' \
    --hook-master-start 'unix_signal:15 gracefully_kill_them_all' \
    --enable-threads --py-call-osafterfork

Speaker Notes


uWSGI command line

.left[Behind the scenes, run.sh calls scripts/get_uwsgi_args.py, which:]

You can take “full control” over this process by skipping run.sh and simply calling uwsgi directly, e.g. uwsgi --yaml config/galaxy.yml.

Speaker Notes


uWSGI communication

A proxy server is not required, but its use is strongly encouraged for performance reasons.

Speaker Notes


YAML config

uWSGI natively supports YAML configs (and INI, and PasteDeploy INI, and XML, and JSON, and …)

WARNING: “uWSGI YAML” is not real YAML!

.pull-left[ Real YAML

uwsgi:
    # quoting forces string
    socket: '127.0.0.1:8001'
    # proper YAML list
    mule:
      - lib/galaxy/main.py
      - lib/galaxy/main.py

]

.pull-right[ uWSGI “YAML”

uwsgi:
    # quote chars read literally
    socket: '127.0.0.1:8001'
    # a uWSGI YAML "list":
    #  repeat keys
    mule: lib/galaxy/main.py
    mule: lib/galaxy/main.py

]

Speaker Notes


YAML config

Speaker Notes


Configuration Schema

.center[[config_schema.yml][config-schema] contains possible options and their types.]

.left[config_schema.yml is the canonical source for config option documentation, from which:]

Speaker Notes


uWSGI

Speaker Notes


uWSGI Wheel

Galaxy’s uWSGI is not built like standard pip install uwsgi

A bit of technical minutiae that might help debugging

Speaker Notes


uWSGI Wheel

Speaker Notes

Speaker Notes


uWSGI Wheel

The uWSGI wheel is built differently than when built from source with

pip install uwsgi

pyuwsgi wheel: the only way to precompile uWSGI in a way that does not require libpythonX.Y.so or ship CPython

Speaker Notes


uWSGI

Speaker Notes


class: top

Processes and Job Handling

uWSGI starts up in a single “master” process and then fork()s a configured number of anonymous web worker processes to serve web requests.

By default, web workers also handle Galaxy job (tool execution) preparation and completion.

Speaker Notes

Processing Galaxy job (tool execution) preparation and finishing is somewhat resource intensive and affects web responsiveness.

Production Galaxy servers traditionally start additional dedicated Galaxy “webless” (do not serve web requests) job handler processes.

Speaker Notes

uWSGI provides a useful feature for running Galaxy job handlers: uWSGI Mules.

Speaker Notes


uWSGI Mules

Mules are processes fork()ed from the uWSGI master after the application has been loaded and web workers have fork()ed.

Mules can continue to run the same code or can load and run arbitrary code.

Mules can receive messages from the web proceses.

Mules can be pooled in to Farms, and messages can be sent to the farm to be handled by any mule in that farm.

Speaker Notes


Mule Advantages

.pull-left.reduce90[ Webless handlers

.pull-right.reduce90[ uWSGI Mule handlers

Speaker Notes


Job Handler Mule Configuration

Adding job handler mules is performed by simply instructing uWSGI to start them in galaxy.yml:

uwsgi:
    mule: lib/galaxy/main.py
    mule: lib/galaxy/main.py
    farm: job-handlers:1,2

That’s it! This Galaxy instance will now start and use two job handler mules.

Speaker Notes


The full startup picture

Speaker Notes


class: top

uWSGI Start/Run and Job Handling Lifecycle

  1. Master process loads Galaxy application.

Speaker Notes

  1. Master fork()s web worker processes and begins serving web requests.

Speaker Notes

  1. Master fork()s job handler mules.

Speaker Notes

  1. Mules reload Galaxy application as job handlers.

Speaker Notes

  1. The first mule to fully initialize grabs a lock on the farm message queue.

Speaker Notes

  1. All additional mules wait on the lock.

Speaker Notes

  1. A web worker receives a request for a new job.

Speaker Notes

  1. The web worker creates a job record in the database but leaves the handler field null.

Speaker Notes

  1. The web worker uses uWSGI’s farm_msg() function to notify the job-handlers farm that a new job is ready to run.

Speaker Notes

  1. The mule with the lock receives the message, assigns itself, and gives up the lock.

Speaker Notes

  1. Another mule acquires the lock and waits for the next message.

Speaker Notes

  1. Repeat

Speaker Notes


Advanced: Job Handler Assignment Methods

Mules can only be used if web workers and job handlers run on the same host.

The database can be used like a message queue to get the benefits of mules using webless job handlers.

See Job Handler Assignment Methods for details.

Mules are preferred in scenarios where webless handlers are not needed.

Speaker Notes


Advanced: Transparent Restart

Provides uninterrupted restart capability (clients do not notice restarts).

See Transparent Restart and uWSGI Zerg Mode documentation for details.

Speaker Notes


Advanced: Zerg Mode + Job Handler Mules

Combining both advanced modes was long thought to be impossible.

However, a proof of concept has emerged!

Watch this space.

Speaker Notes


Thank you!

This material is the result of a collaborative work. Thanks to the Galaxy Training Network and all the contributors! page logo This material is licensed under the Creative Commons Attribution 4.0 International License.