Deploying a Beacon v1 in Galaxy

Author(s) orcid logoHelena Rasche avatar Helena Rasche
Editor(s) orcid logoSaskia Hiltemann avatar Saskia Hiltemann
Reviewers Nate Coraor avatarHelena Rasche avatarMarius van den Beek avatarSaskia Hiltemann avatar
Overview
Creative Commons License: CC-BY Questions:

  • What is a Beacon?

  • How do I deploy it?

  • Is v1 the same as v2?

Objectives:

  • Deploy a Beacon

Requirements:
Time estimation: 30 minutes
Supporting Materials:
Published: Apr 16, 2023
Last modification: Dec 18, 2024
License: Tutorial Content is licensed under Creative Commons Attribution 4.0 International License. The GTN Framework is licensed under MIT
PURL: https://gxy.io/GTN:T00325
Revision: 5

This tutorial will guide you through setting up a GA4GH Beacon!

The Beacon Project is developed under a Global Alliance for Genomics and Health (GA4GH) Iniciative for the federated discovery of genomic data in biomedical research and clinical applications.

Warning: Beacon v1

This deploys an older Beacon v1 which was a simpler system. The Beacon v1 is more or less deprecated, with users being pushed to Beacon v2 which gives much richer answers, and offers better querying syntax.

Agenda
  1. Installing and Configuring
  2. Setting up the Importer

Installing and Configuring

Hands-on: Setting up a Beacon with Ansible

  1. Setup the hosts

    --- a/hosts
+++ b/hosts
@@ -9,3 +9,12 @@ gat-0.eu.galaxy.training ansible_connection=local ansible_user=ubuntu
    
 [sentryservers]
 gat-0.eu.training.galaxyproject.eu ansible_connection=local ansible_user=ubuntu
+
+[beacon]
+[beacon:children]
+beacon_import
+beacon_server
+[beacon_server]
+gat-0.eu.galaxy.training ansible_connection=local ansible_user=ubuntu
+[beacon_import]
+gat-0.eu.galaxy.training ansible_connection=local ansible_user=ubuntu

    Here we use some of the more advanced features of Ansible’s Inventory system. We declare a host group called ‘beacon’ with no hosts of its own.

    Then we declare that this beacon group has two children: beacon_import, and beacon_server. We can then define host groups for those two entries with as many different hosts as we need. This makes it very easy to scale up configuration.

    Here we will be using that feature to declare some ‘beacon variables’, which will be shared between the beacon_server and beacon_importer. Because they’re children of ‘beacon’, they’ll inherit any group variables defined for group_vars/beacon.yml.

  2. Setup the requirements

    --- a/requirements.yml
+++ b/requirements.yml
@@ -60,3 +60,8 @@
 # Our FTP Server
 - src: galaxyproject.proftpd
   version: 0.3.1
+# Beacon support
+- name: paprikant.beacon
+  src: https://github.com/Paprikant/ansible-role-beacon
+- name: paprikant.beacon-importer
+  src: https://github.com/Paprikant/ansible-role-beacon_importer

  3. Install the role with:

    Input: Bash
    ansible-galaxy install -p roles -r requirements.yml

  4. Install the docker role as well.

    Input: Bash
    ansible-galaxy collection install community.docker

    Ansible over time added the feature of ‘collections’ in addition to roles. It’s a nice and convenient method for grouping roles together. Unfortunately this means we need to update a significant portion of the GAT learning path to use this, which we have not done yet, as it will take a bit of time to do. Collections come with the unpleasant additional aspect that they will be installed into a different path than roles, so we’ll need to add a collections folder as well, or maybe deps/collections and deps/roles folders. When this happens we’ll have to additionally modify the ansible.cfg to look for collections in this additional path.

  5. Create the vars file

    --- /dev/null
+++ b/group_vars/beacon.yml
@@ -0,0 +1,31 @@
+---
+postgres_data_dir: /data/beacon/postgresql/data
+postgres_init_dir: /data/beacon/postgresql/init
+bp_external_binding: 5050 # The default
+postgres_user: "{{ beacon_db_user }}"
+postgres_pass: "{{ beacon_db_password }}"
+postgres_external_binding: "{{ beacon_db_port }}"
+# Database Configuration
+beacon_db_user: beacon
+beacon_db_host: "{{ groups['beacon_server'][0] }}"
+beacon_db_password: "{{ vault_beacon_db_password }}"
+beacon_db_port: 9001
+#galaxy_api_key: This we will set in secrets.
+# Information about your beacon (consider filling this out.
+beacon_info_title: GA4GH Beacon
+beacon_info_beacon_id: your.galaxy.beacon
+beacon_info_description: Beacon service hosting datasets from all over the Galaxy
+beacon_info_url: https://{{ groups['beacon_server'][0] }}/beacon/
+beacon_info_service_group: galaxy-eu
+beacon_info_org_id: usegalaxy.aq
+beacon_info_org_name: Some Galaxy
+beacon_info_org_description: Galaxy community
+beacon_info_org_address: 123 Main Street, ZA
+beacon_info_org_welcome_url: https://galaxyproject.org/
+beacon_info_org_contact_url: https://galaxyproject.org/
+beacon_info_org_logo_url: https://galaxyproject.org/images/galaxy-logos/galaxy_project_logo_square.png
+beacon_info_org_info: More information about the organisation than just the description can go here.
+# Script Configuration
+galaxy_api_url: "https://{{ groups['galaxyservers'][0] }}"
+script_user: beacon
+script_dir: /home/beacon/script

    Here we again use some advanced features of Ansible’s inventory system. Ansible knows the name of every hostname in the inventory. Now that we want to point the beacon configuration, either at the database which should be on beacon_server, or at Galaxy in galaxyservers, we ask the groups variable for what the inventory looks like. We use [0] to pull out the first hostname we find for both of those groups.

  6. Edit your group_vars/secret.yml and define some random passwords:

    • The API key for your account, which must be an admin
    • A password for the beacon database

    Galaxy admin accounts are specified as a comma-separated email list in the admin_users directive of galaxy.yml . If you have set up your Galaxy server using the Galaxy Installation with Ansible tutorial, this is set to admin@example.org.

    1. In your browser, open your Galaxy homepage
    2. Log in, or register a new account, if it’s the first time you’re logging in
    3. Go to User -> Preferences in the top menu bar, then click on Manage API key
    4. If there is no current API key available, click on Create a new key to generate it
    5. Copy your API key to somewhere convenient, you will need it throughout this tutorial

    Input: Bash
    ansible-vault edit group_vars/secret.yml

    --- a/group_vars/secret.yml
+++ b/group_vars/secret.yml
@@ -1,13 +1,15 @@
+galaxy_api_key: your-galaxy-api-key
+vault_beacon_db_password: some-super-secret-password

  7. Add the beacon-server playbook

    --- /dev/null
+++ b/beacon-server.yml
@@ -0,0 +1,9 @@
+---
+- name: Beacon Server
+  hosts: beacon_server
+  become: true
+  become_user: root
+  vars_files:
+    - group_vars/secret.yml
+  roles:
+    - paprikant.beacon

  8. Run the playbook

    Input: Bash
    ansible-playbook beacon-server.yml

    TODO: Check that it works

Setting up the Importer

Now that our beacon is running, we need to get data from Galaxy to the Beacon

Hands-on: Setting up the Beacon Importer

  1. Add the beacon-import playbook

    --- /dev/null
+++ b/beacon-import.yml
@@ -0,0 +1,9 @@
+---
+- name: Beacon Importer
+  hosts: beacon_import
+  become: true
+  become_user: root
+  vars_files:
+    - group_vars/secret.yml
+  roles:
+    - paprikant.beacon-importer

  2. Run the playbook

    Input: Bash
    ansible-playbook beacon-import.yml

  3. Add the nginx routes

    --- a/templates/nginx/galaxy.j2
+++ b/templates/nginx/galaxy.j2
@@ -117,4 +117,14 @@ server {
    
 	{{ tiaas_nginx_routes }}
    
+	location /beacon {
+		proxy_pass http://{{ groups['beacon_server'][0] }}:5050;
+		proxy_http_version 1.1;
+		proxy_set_header Upgrade $http_upgrade;
+		proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
+		proxy_set_header X-Real-IP $remote_addr;
+		proxy_set_header Connection "upgrade";
+		proxy_set_header Host $host;
+	}
+
 }

  4. Run the playbook

    Input: Bash
    ansible-playbook galaxy.yml

Congratulations, you’ve set up a Beacon v1 for Galaxy! Go check it out at /beacon/ on your server.

Comment: Got lost along the way?

If you missed any steps, you can compare against the reference files, or see what changed since the previous tutorial.

If you’re using git to track your progress, remember to add your changes and commit with a good commit message!

Comment: Galaxy Admin Training Path

The yearly Galaxy Admin Training follows a specific ordering of tutorials. Use this timeline to help keep track of where you are in Galaxy Admin Training.

  1. Step 1
    ansible-galaxy
  2. Step 2
    backup-cleanup
  3. Step 3
    customization
  4. Step 4
    tus
  5. Step 5
    cvmfs
  6. Step 6
    apptainer
  7. Step 7
    tool-management
  8. Step 8
    reference-genomes
  9. Step 9
    data-library
  10. Step 10
    dev/bioblend-api
  11. Step 11
    connect-to-compute-cluster
  12. Step 12
    job-destinations
  13. Step 13
    pulsar
  14. Step 14
    celery
  15. Step 15
    gxadmin
  16. Step 16
    reports
  17. Step 17
    monitoring
  18. Step 18
    tiaas
  19. Step 19
    sentry
  20. Step 20
    ftp
  21. Step 21
    beacon