name: inverse layout: true class: center, middle, inverse
--- # uWSGI .footnote[Tip: press `P` to view the presenter notes] ??? Presenter notes contain extra information which might be useful if you intend to use these slides for teaching. Press `P` again to switch presenter notes off Press `C` to create a new window where the same presentation will be displayed. This window is linked to the main window. Changing slides on one will cause the slide to change on the other. Useful when presenting. --- # What is uWSGI? - Python WSGI (Web Server Gateway Interface) application framework - Written in C - Forking application server - Optional HTTP server - Process manager ...and the rest of the kitchen sink --- # Why do we need uWSGI? - Galaxy is a Python web application without a web server - Implements standard Python WSGI - Some other application does the web serving - Technically we could use any HTTP/WSGI interface - Traditionally [Python Paste](https://github.com/cdent/paste/) --- # What is uWSGI used for? .left[In a production Galaxy server:] - To start, manage, and scale web workers - Optionally, to start, manage, and scale job handlers --- # Why uWSGI? Because Python can't multithread. ![Python GIL](../../images/gil.png) .footnote[Image credit: [Dariusz Fryta](http://www.tivix.com/blog/lets-go-python/)] ??? Python can't multithread, but it can *multiprocess*. --- # Why uWSGI? - Fork many Galaxy server processes - Isolate job functions from web functions - Built in load balancing - Speak native high performance uWSGI protocol to nginx - Uninterrupted restarting - Can do anything you can imagine: [uWSGI configuration options](http://uwsgi-docs.readthedocs.io/en/latest/Options.html) --- ## uWSGI - **Configuration** - uWSGI wheel - Job handler mules --- ## uWSGI Configuration File uWSGI configuration is performed in the `uwsgi` key of `galaxy.yml`. `galaxy.yml.sample` is never consulted. .left[If using `run.sh`:] - If no config exists, all necessary defaults are generated as command line arguments to uWSGI - If a config exists, it is parsed, and any missing required options are passed as command line arguments to uWSGI **Galaxy servers deployed with Ansible do not use `run.sh` and should fully specify their uWSGI configuration.** --- ## uWSGI default command line Without a config file, the uWSGI command line is: ```sh-session $ /home/nate/galaxy/.venv/bin/python3 .venv/bin/uwsgi \ --module 'galaxy.webapps.galaxy.buildapp:uwsgi_app()' \ --virtualenv /home/nate/galaxy/.venv --pythonpath lib \ --threads 4 --http localhost:8080 \ --static-map /static=/home/nate/galaxy/static --die-on-term \ --hook-master-start 'unix_signal:2 gracefully_kill_them_all' \ --hook-master-start 'unix_signal:15 gracefully_kill_them_all' \ --enable-threads --py-call-osafterfork ``` --- ## uWSGI command line .left[Behind the scenes, `run.sh` calls `scripts/get_uwsgi_args.py`, which:] - locates your config file (if any), - parses it to determine whether you have set any of the default (or conflicting) options, - determines the flags needed to run Galaxy. You can take "full control" over this process by skipping `run.sh` and simply calling `uwsgi` directly, e.g. `uwsgi --yaml config/galaxy.yml`. --- # uWSGI communication - uWSGI can serve HTTP directly using the `--http` option - uWSGI speaks a native protocol (which nginx also speaks) using `--socket` A proxy server is not *required,* but its use is strongly encouraged for performance reasons. --- ## YAML config uWSGI natively supports YAML configs (and INI, and PasteDeploy INI, and XML, and JSON, and ...) **WARNING:** "uWSGI YAML" is not real YAML! -- .pull-left[ Real YAML ```yaml uwsgi: # quoting forces string socket: '127.0.0.1:8001' # proper YAML list mule: - lib/galaxy/main.py - lib/galaxy/main.py ``` ] .pull-right[ uWSGI "YAML" ```yaml uwsgi: # quote chars read literally socket: '127.0.0.1:8001' # a uWSGI YAML "list": # repeat keys mule: lib/galaxy/main.py mule: lib/galaxy/main.py ``` ] --- ## YAML config - "uWSGI YAML" only applies to the `uwsgi` section of `galaxy.yml` - The `galaxy` section must use real YAML - uWSGI can be compiled against libyaml for real YAML support - Galaxy Ansible role can write *both* uWSGI (default), real YAML - `galaxy.ini` is still usable but deprecated --- ## Configuration Schema .center[`[config_schema.yml][config-schema]` contains possible options **and their types**.] .left[config_schema.yml is the canonical source for config option documentation, from which:] - .left[`[galaxy.yml.sample][config-sample]` is generated] - [Galaxy configuration options documentation][config-docs] is generated [config-schema]: https://github.com/galaxyproject/galaxy/blob/release_20.01/lib/galaxy/webapps/galaxy/config_schema.yml [config-sample]: https://github.com/galaxyproject/galaxy/blob/release_20.01/lib/galaxy/config/sample/galaxy.yml.sample [config-docs]: https://docs.galaxyproject.org/en/master/admin/config.html --- ## uWSGI - Configuration - **uWSGI wheel** - Job handler mules --- ## uWSGI Wheel Galaxy's uWSGI is not built like standard `pip install uwsgi` A bit of technical minutiae that might help debugging --- ## uWSGI Wheel - The challenge: - Unlike Galaxy's other framework dependencies, uWSGI is not a Python library - uWSGI embeds the CPython interpreter - Essentially uWSGI *is* Galaxy's CPython -- - The goal: Provide a no-compilation-required, no-dependencies installation method for Galaxy that includes uWSGI that: - Doesn't embed statically linked CPython - Doesn't require system `libpythonX.Y.so` - Uses system CPython --- ## uWSGI Wheel The uWSGI wheel is built differently than when built from source with `pip install uwsgi` - From source by pip: - Built as a single `uwsgi` ELF binary embedding the CPython interpreter - As the pyuwsgi wheel: - Built as a Python C extension into an ELF shared library `pyuwsgi.so` - Loaded by a stub `uwsgi` Python script by the CPython interpreter pyuwsgi wheel: the only way to precompile uWSGI in a way that does not require `libpythonX.Y.so` or ship CPython --- ## uWSGI - Configuration - uWSGI wheel - **Job handler mules** --- class: top ## Processes and Job Handling uWSGI starts up in a single "master" process and then `fork()`s a configured number of anonymous *web worker* processes to **serve web requests**. By default, web workers also handle Galaxy job (tool execution) preparation and completion. -- Processing Galaxy job (tool execution) preparation and finishing is somewhat resource intensive and affects web responsiveness. Production Galaxy servers traditionally start additional dedicated Galaxy "webless" (do not serve web requests) *job handler* processes. -- uWSGI provides a useful feature for running Galaxy job handlers: [uWSGI Mules](https://uwsgi-docs.readthedocs.io/en/latest/Mules.html). --- ## uWSGI Mules **Mules** are processes `fork()`ed from the uWSGI master after the application has been loaded and web workers have `fork()`ed. Mules can continue to run the same code or can load and run arbitrary code. Mules can receive messages from the web proceses. Mules can be pooled in to *Farms*, and messages can be sent to the farm to be handled by any mule in that farm. --- ## Mule Advantages .pull-left.reduce90[ Webless handlers - Config is complex - Must manage handler processes externally (e.g. w/ systemd) - Handler assignment: - At random from configured handlers - No regard as to handler health - Notification via database - Can be spread across multiple hosts ] .pull-right.reduce90[ uWSGI Mule handlers - Config is trivial - Handler processes managed automatically by uWSGI master - Handler assignment: - "Grabbed" by healthy mule - IPC (messaging) from web worker - Mules run on same host as web workers ] --- ## Job Handler Mule Configuration Adding job handler mules is performed by simply instructing uWSGI to start them in `galaxy.yml`: ```yaml uwsgi: mule: lib/galaxy/main.py mule: lib/galaxy/main.py farm: job-handlers:1,2 ``` That's it! This Galaxy instance will now start and use two job handler mules. --- class: top ## uWSGI Start/Run and Job Handling Lifecycle 1. Master process loads Galaxy application. -- 2. Master `fork()`s web worker processes and begins serving web requests. -- 3. Master `fork()`s job handler mules. -- 4. Mules reload Galaxy application as job handlers. -- 5. The first mule to fully initialize grabs a lock on the farm message queue. -- 6. All additional mules wait on the lock. -- 7. A web worker receives a request for a new job. -- 8. The web worker creates a `job` record in the database but leaves the `handler` field `null`. -- 9. The web worker uses uWSGI's `farm_msg()` function to notify the `job-handlers` farm that a new job is ready to run. -- 10. The mule with the lock receives the message, assigns itself, and gives up the lock. -- 11. Another mule acquires the lock and waits for the next message. -- 12. Job handling continues as normal. --- ## Advanced: Job Handler Assignment Methods **Mules can only be used if web workers and job handlers run on the same host.** The database can be used like a message queue to get the benefits of mules using webless job handlers. See [Job Handler Assignment Methods](https://docs.galaxyproject.org/en/master/admin/scaling.html#job-handler-assignment-methods) for details. Mules are preferred in scenarios where webless handlers are not needed. --- ## Advanced: Transparent Restart Provides uninterrupted restart capability (clients do not notice restarts). See [Transparent Restart](https://docs.galaxyproject.org/en/master/admin/scaling.html#transparent-restart-zerg-mode) and [uWSGI Zerg Mode documentation](https://uwsgi-docs.readthedocs.io/en/latest/Zerg.html) for details. --- ## Advanced: Zerg Mode + Job Handler Mules Combining both advanced modes was long thought to be impossible. However, a proof of concept has emerged! [Watch this space](https://gist.github.com/natefoo/6e45ae363ef434f38a935ff1b05b4b6e). --- ## Thank you! This material is the result of a collaborative work. Thanks to the [Galaxy Training Network](https://wiki.galaxyproject.org/Teach/GTN) and all the contributors!
This material is licensed under the
Creative Commons Attribution 4.0 International License
. .footnote[Found a typo? Something is wrong in this tutorial?
Edit it on [GitHub](https://github.com/galaxyproject/training-material/tree/master/topics/admin/tutorials/uwsgi/slides.html)]