GTN 'Pangeo 101 for everyone - Xarray'

climate-pangeo/main-workflow

Author(s)
Anne Fouilloux
Version
1
Last updated
Feb 18, 2022
License
MIT
Tags
pangeo
climate
copernicus
GTN

Features
Tutorial
Pangeo ecosystem 101 for everyone - Introduction to Xarray Galaxy Tools
Workflow Testing
Tests: ❌
Results: Not yet automated
FAIRness PURL
https://gxy.io/GTN:W00026
RO-Crate logo with flask Download Workflow RO-Crate Workflowhub cloud with gears logo View on (Dev) WorkflowHub
Launch in Tutorial Mode
Download 
flowchart TD
  0["ℹ️ Input Parameter\nplot first time e.g. index=0"];
  style 0 fill:#ded,stroke:#393,stroke-width:4px;
  1["ℹ️ Input Dataset\nCAMS-PM2_5-20211222.netcdf"];
  style 1 stroke:#2c3143,stroke-width:4px;
  2["ℹ️ Input Parameter\nplot forecast time 2 days 12:00 UTC"];
  style 2 fill:#ded,stroke:#393,stroke-width:4px;
  3["ℹ️ Input Parameter\ntime=5"];
  style 3 fill:#ded,stroke:#393,stroke-width:4px;
  4["ℹ️ Input Parameter\ntime=6"];
  style 4 fill:#ded,stroke:#393,stroke-width:4px;
  5["ℹ️ Input Parameter\ntime=3"];
  style 5 fill:#ded,stroke:#393,stroke-width:4px;
  6["ℹ️ Input Parameter\ntime=2"];
  style 6 fill:#ded,stroke:#393,stroke-width:4px;
  7["ℹ️ Input Parameter\ntime=7"];
  style 7 fill:#ded,stroke:#393,stroke-width:4px;
  8["ℹ️ Input Parameter\ntime=1"];
  style 8 fill:#ded,stroke:#393,stroke-width:4px;
  9["ℹ️ Input Parameter\ntime=4"];
  style 9 fill:#ded,stroke:#393,stroke-width:4px;
  10["ℹ️ Input Parameter\ntime=0"];
  style 10 fill:#ded,stroke:#393,stroke-width:4px;
  11["NetCDF xarray Coordinate Info"];
  1 -->|output| 11;
  12["NetCDF xarray Metadata Info"];
  1 -->|output| 12;
  13["extract longitude"];
  11 -->|output_dir| 13;
  14["extract latitude"];
  11 -->|output_dir| 14;
  15["Extract time from coordinate info"];
  11 -->|output_dir| 15;
  3b4968d7-8984-4e66-87a9-a3ab8fc12ca4["Output\ninput dataset(s) (extracted element)"];
  15 --> 3b4968d7-8984-4e66-87a9-a3ab8fc12ca4;
  style 3b4968d7-8984-4e66-87a9-a3ab8fc12ca4 stroke:#2c3143,stroke-width:4px;
  16["NetCDF xarray operations"];
  1 -->|output| 16;
  14 -->|output| 16;
  13 -->|output| 16;
  12 -->|output| 16;
  17["NetCDF xarray operations"];
  1 -->|output| 17;
  15 -->|output| 17;
  12 -->|output| 17;
  18["NetCDF xarray operations"];
  1 -->|output| 18;
  15 -->|output| 18;
  14 -->|output| 18;
  13 -->|output| 18;
  12 -->|output| 18;
  19["PM2.5 CAMS Europe 0 days 00:00 UTC reference time"];
  15 -->|output| 19;
  0 -->|output| 19;
  1 -->|output| 19;
  12 -->|output| 19;
  42483031-21e2-4d88-8898-4a9aa9bb10e5["Output\nPM2.5 reference date Europe"];
  19 --> 42483031-21e2-4d88-8898-4a9aa9bb10e5;
  style 42483031-21e2-4d88-8898-4a9aa9bb10e5 stroke:#2c3143,stroke-width:4px;
  20["PM2.5 CAMS forecast 2 days 12:00 UTC"];
  15 -->|output| 20;
  2 -->|output| 20;
  1 -->|output| 20;
  12 -->|output| 20;
  07ba8141-bad3-4a61-833d-3b3e82b53fdb["Output\nPM2.5 fc 2 days 12:00 UTC Europe"];
  20 --> 07ba8141-bad3-4a61-833d-3b3e82b53fdb;
  style 07ba8141-bad3-4a61-833d-3b3e82b53fdb stroke:#2c3143,stroke-width:4px;
  21["NetCDF xarray Selection"];
  16 -->|output_netcdf| 21;
  12 -->|output| 21;
  22["NetCDF xarray Coordinate Info"];
  17 -->|output_netcdf| 22;
  23["NetCDF xarray Coordinate Info"];
  18 -->|output_netcdf| 23;
  24["NetCDF xarray Metadata Info"];
  18 -->|output_netcdf| 24;
  25["Column Regex Find And Replace"];
  21 -->|simpleoutput| 25;
  26["PM2.5 over Naples 4 days forecast"];
  21 -->|simpleoutput| 26;
  8984beb4-e593-42f8-8de5-0e92631061c7["Output\n1D plot PM2.5 over Naples (4 days forecast)"];
  26 --> 8984beb4-e593-42f8-8de5-0e92631061c7;
  style 8984beb4-e593-42f8-8de5-0e92631061c7 stroke:#2c3143,stroke-width:4px;
  27["Extract times from coordinates"];
  22 -->|output_dir| 27;
  28["Extract time"];
  23 -->|output_dir| 28;
  29[" stripes PM2.5 over Naples 4 days forecast"];
  25 -->|out_file1| 29;
  92cad576-78b5-4e07-9a07-6f05ff8f4a56["Output\nPM2.5 stripes Naples for 4 days forecast from reference date"];
  29 --> 92cad576-78b5-4e07-9a07-6f05ff8f4a56;
  style 92cad576-78b5-4e07-9a07-6f05ff8f4a56 stroke:#2c3143,stroke-width:4px;
  30["NetCDF xarray map plotting"];
  27 -->|output| 30;
  0 -->|output| 30;
  18 -->|output_netcdf| 30;
  24 -->|output| 30;
  31["NetCDF xarray map plotting"];
  28 -->|output| 31;
  6 -->|output| 31;
  18 -->|output_netcdf| 31;
  24 -->|output| 31;
  32["NetCDF xarray map plotting"];
  28 -->|output| 32;
  8 -->|output| 32;
  18 -->|output_netcdf| 32;
  24 -->|output| 32;
  33["NetCDF xarray map plotting"];
  28 -->|output| 33;
  5 -->|output| 33;
  18 -->|output_netcdf| 33;
  24 -->|output| 33;
  34["NetCDF xarray map plotting"];
  28 -->|output| 34;
  9 -->|output| 34;
  18 -->|output_netcdf| 34;
  24 -->|output| 34;
  35["NetCDF xarray map plotting"];
  28 -->|output| 35;
  4 -->|output| 35;
  18 -->|output_netcdf| 35;
  24 -->|output| 35;
  36["NetCDF xarray map plotting"];
  28 -->|output| 36;
  7 -->|output| 36;
  18 -->|output_netcdf| 36;
  24 -->|output| 36;
  37["NetCDF xarray map plotting"];
  28 -->|output| 37;
  3 -->|output| 37;
  18 -->|output_netcdf| 37;
  24 -->|output| 37;
  38["NetCDF xarray map plotting"];
  28 -->|output| 38;
  3 -->|output| 38;
  18 -->|output_netcdf| 38;
  24 -->|output| 38;
  39["NetCDF xarray map plotting"];
  28 -->|output| 39;
  4 -->|output| 39;
  18 -->|output_netcdf| 39;
  24 -->|output| 39;
  40["NetCDF xarray map plotting"];
  28 -->|output| 40;
  10 -->|output| 40;
  18 -->|output_netcdf| 40;
  24 -->|output| 40;
  41["NetCDF xarray map plotting"];
  28 -->|output| 41;
  5 -->|output| 41;
  18 -->|output_netcdf| 41;
  24 -->|output| 41;
  42["NetCDF xarray map plotting"];
  28 -->|output| 42;
  6 -->|output| 42;
  18 -->|output_netcdf| 42;
  24 -->|output| 42;
  43["NetCDF xarray map plotting"];
  28 -->|output| 43;
  10 -->|output| 43;
  18 -->|output_netcdf| 43;
  24 -->|output| 43;
  44["NetCDF xarray map plotting"];
  28 -->|output| 44;
  7 -->|output| 44;
  18 -->|output_netcdf| 44;
  24 -->|output| 44;
  45["NetCDF xarray map plotting"];
  28 -->|output| 45;
  9 -->|output| 45;
  18 -->|output_netcdf| 45;
  24 -->|output| 45;
  46["NetCDF xarray map plotting"];
  28 -->|output| 46;
  8 -->|output| 46;
  18 -->|output_netcdf| 46;
  24 -->|output| 46;
  47["Merge collections"];
  30 -->|output_dir| 47;
  32 -->|output_dir| 47;
  31 -->|output_dir| 47;
  33 -->|output_dir| 47;
  34 -->|output_dir| 47;
  37 -->|output_dir| 47;
  35 -->|output_dir| 47;
  36 -->|output_dir| 47;
  48["Merge collections"];
  43 -->|output_dir| 48;
  46 -->|output_dir| 48;
  42 -->|output_dir| 48;
  41 -->|output_dir| 48;
  45 -->|output_dir| 48;
  38 -->|output_dir| 48;
  39 -->|output_dir| 48;
  44 -->|output_dir| 48;
  49["PM2.5 forecast 10:00 - 17:00 UTC > 30 um/m3"];
  47 -->|output| 49;
  1c16eeb7-dfe1-40de-9be7-3aa2a8568308["Output\nPM2.5 (values > 30 um/m3)  fc 10:00 - 17:00 UTC over Italy"];
  49 --> 1c16eeb7-dfe1-40de-9be7-3aa2a8568308;
  style 1c16eeb7-dfe1-40de-9be7-3aa2a8568308 stroke:#2c3143,stroke-width:4px;
  50["PM2.5 forecast 10:00 - 17:00 UTC "];
  48 -->|output| 50;
  070fe227-b533-4c77-bed0-4d005b174920["Output\nPM2.5 fc 10:00 - 17:00 UTC over Italy"];
  50 --> 070fe227-b533-4c77-bed0-4d005b174920;
  style 070fe227-b533-4c77-bed0-4d005b174920 stroke:#2c3143,stroke-width:4px;

Inputs

Input Label
Input parameter plot first time e.g. index=0
Input dataset CAMS-PM2_5-20211222.netcdf
Input parameter plot forecast time 2 days 12:00 UTC
Input parameter time=5
Input parameter time=6
Input parameter time=3
Input parameter time=2
Input parameter time=7
Input parameter time=1
Input parameter time=4
Input parameter time=0

Outputs

From Output Label
toolshed.g2.bx.psu.edu/repos/ecology/xarray_coords_info/xarray_coords_info/0.18.2+galaxy0 NetCDF xarray Coordinate Info
__EXTRACT_DATASET__ Extract dataset extract longitude
__EXTRACT_DATASET__ Extract dataset extract latitude
__EXTRACT_DATASET__ Extract dataset Extract time from coordinate info
toolshed.g2.bx.psu.edu/repos/ecology/xarray_netcdf2netcdf/xarray_netcdf2netcdf/0.18.2+galaxy0 NetCDF xarray operations
toolshed.g2.bx.psu.edu/repos/ecology/xarray_netcdf2netcdf/xarray_netcdf2netcdf/0.18.2+galaxy0 NetCDF xarray operations
toolshed.g2.bx.psu.edu/repos/ecology/xarray_netcdf2netcdf/xarray_netcdf2netcdf/0.18.2+galaxy0 NetCDF xarray operations
toolshed.g2.bx.psu.edu/repos/ecology/xarray_mapplot/xarray_mapplot/0.18.2+galaxy0 NetCDF xarray map plotting PM2.5 CAMS Europe 0 days 00:00 UTC (reference time)
toolshed.g2.bx.psu.edu/repos/ecology/xarray_mapplot/xarray_mapplot/0.18.2+galaxy0 NetCDF xarray map plotting PM2.5 CAMS forecast 2 days 12:00 UTC
toolshed.g2.bx.psu.edu/repos/iuc/ggplot2_point/ggplot2_point/2.2.1+galaxy2 Scatterplot with ggplot2 PM2.5 over Naples (4 days forecast)
__EXTRACT_DATASET__ Extract dataset Extract times from coordinates
__EXTRACT_DATASET__ Extract dataset Extract time
toolshed.g2.bx.psu.edu/repos/climate/climate_stripes/climate_stripes/1.0.1 climate stripes stripes PM2.5 over Naples (4 days forecast)
toolshed.g2.bx.psu.edu/repos/ecology/xarray_mapplot/xarray_mapplot/0.18.2+galaxy0 NetCDF xarray map plotting
toolshed.g2.bx.psu.edu/repos/bgruening/graphicsmagick_image_montage/graphicsmagick_image_montage/1.3.31+galaxy1 Image Montage PM2.5 forecast 10:00 - 17:00 UTC > 30 um/m3
toolshed.g2.bx.psu.edu/repos/bgruening/graphicsmagick_image_montage/graphicsmagick_image_montage/1.3.31+galaxy1 Image Montage PM2.5 forecast 10:00 - 17:00 UTC

Tools

Tool Links
__EXTRACT_DATASET__
__MERGE_COLLECTION__
toolshed.g2.bx.psu.edu/repos/bgruening/graphicsmagick_image_montage/graphicsmagick_image_montage/1.3.31+galaxy1 View in ToolShed
toolshed.g2.bx.psu.edu/repos/climate/climate_stripes/climate_stripes/1.0.1 View in ToolShed
toolshed.g2.bx.psu.edu/repos/ecology/xarray_coords_info/xarray_coords_info/0.18.2+galaxy0 View in ToolShed
toolshed.g2.bx.psu.edu/repos/ecology/xarray_mapplot/xarray_mapplot/0.18.2+galaxy0 View in ToolShed
toolshed.g2.bx.psu.edu/repos/ecology/xarray_metadata_info/xarray_metadata_info/0.15.1 View in ToolShed
toolshed.g2.bx.psu.edu/repos/ecology/xarray_netcdf2netcdf/xarray_netcdf2netcdf/0.18.2+galaxy0 View in ToolShed
toolshed.g2.bx.psu.edu/repos/ecology/xarray_select/xarray_select/0.15.1 View in ToolShed
toolshed.g2.bx.psu.edu/repos/galaxyp/regex_find_replace/regexColumn1/1.0.1 View in ToolShed
toolshed.g2.bx.psu.edu/repos/iuc/ggplot2_point/ggplot2_point/2.2.1+galaxy2 View in ToolShed

To use these workflows in Galaxy you can either click the links to download the workflows, or you can right-click and copy the link to the workflow which can be used in the Galaxy form to import workflows.

Importing into Galaxy

Below are the instructions for importing these workflows directly into your Galaxy server of choice to start using them!
Hands-on: Importing a workflow
  • Click on Workflow on the top menu bar of Galaxy. You will see a list of all your workflows.
  • Click on Import at the top-right of the screen
  • Provide your workflow
    • Option 1: Paste the URL of the workflow into the box labelled “Archived Workflow URL”
    • Option 2: Upload the workflow file in the box labelled “Archived Workflow File”
  • Click the Import workflow button

Below is a short video demonstrating how to import a workflow from GitHub using this procedure:

Video: Importing a workflow from URL

Version History

Version Commit Time Comments
2 bf3253c16 2021-12-28 19:46:28 Add Galaxy workflow for Pangeo tutorial
1 5851fc468 2021-12-23 09:04:41 template for Galaxy Pangeo Tutorial

For Admins

Installing the workflow tools

wget https://training.galaxyproject.org/training-material/topics/climate/tutorials/pangeo/workflows/main_workflow.ga -O workflow.ga
workflow-to-tools -w workflow.ga -o tools.yaml
shed-tools install -g GALAXY -a API_KEY -t tools.yaml
workflow-install -g GALAXY -a API_KEY -w workflow.ga --publish-workflows