Pipelines

A pipeline is a series of tasks used to process data. Pipelines are built and managed under the Analyses tab in Connected Multiomics. The combination of circular data nodes and rectangular task nodes displayed under the Analyses tab are herein collectively referred to as the task graph.

This user guide will cover the following topics:

• Create new pipeline

• Run a pipeline

• Download a pipeline

• Preview a pipeline

• Delete a pipeline

• Import a pipeline

Create new pipeline

If you want to re-use a series of analysis steps in a certain Study > Analysis, click to select the analysis to get into the Analyses page, create a pipeline by saving the series of the tasks on this page.

To save (create) a pipeline:

1. Click the Create new pipeline link at the bottom of the screen under the Analyses tab

   
2. In the dialog, name the pipeline and add an optional description. Make sure the Section name is set to Pipelines\

   
3. Left-click the rectangular task nodes you wish to save as part of the pipeline from the task graph. Selected tasks will have a thick black outline. You can't skip any tasks in the series.\

   
4. Click the Create pipeline button. This saves the pipeline and the screen will return to the Analyses tab. To check if the pipeline has been saved, select the circular data node preceding the first task in the saved pipeline and it will appear in the context sensitive menu on the right under Pipelines.

The settings for each of the tasks will be saved, including the advanced settings. However, when running a saved pipeline, you may be prompted to specify additional settings. Furthermore, the task version is also saved. In very rare cases, a pipeline may include a version of a task that is no longer supported. In this situation, the most recent version of the task will be applied to the data.

Note that it is not necessary to select all of the tasks from the task graph. It is possible to select a subset of the tasks, but the selected tasks must be consecutive.

Run a pipeline

Saved and imported pipelines can be applied to different data sets in other analyses. Prior to running a saved or imported pipeline, the data needs to be imported and the sample attributes need to be specified under the Metadata tab. Pipelines that include tasks requiring sample attributes (e.g. differential analysis) will not run unless the sample attributes have been specified beforehand. To run a pipeline:

1. Click on a circular data node under the Analysis tab and expand the Pipelines section from the menu on the right. The pop-up menu will only display saved pipelines that can be applied to the selected data node

   
2. Click on the pipeline name from the menu.

3. If further settings need to be specified for any of the tasks, you will be redirected to a task-specific page. For each task, specify the requested settings and click Next and follow the wizard. Once additional settings have been specified for each task, click Finish. all tasks will be queued and the jobs will run sequentially.

Download a pipeline

All pipelines are automatically made available to all users in the same workgroup, making it easy to share pipelines. It is also possible to download a pipeline as a backup and to share with others. To download a pipeline:

1. Click on a data node and expand the Pipelines section from the menu on the right. The pop-up menu will only display pipelines that can be applied to the data type of the selected data node

2. Click on the 3 dots next to the pipeline name and choose Download pipeline\

   
3. Choose a file location on your local machine and pipeline will be saved as a .pipeline file

The .pipeline file can then be shared, e.g. via email or USB stick, for use on other domain.

Preview a pipeline

It is possible to view an image of the full pipeline before running it. All saved and imported pipelines are available to check the tasks included in the pipeline. To preview a pipeline:

1. Click on a data node and expand the Pipelines section from the menu on the right. The pop-up menu will only display pipelines that can be applied to the data type of the selected data node

2. Click on the 3 dots next to the pipeline name and choose Preview pipeline\

   

An image of the pipeline will be displayed at the center of the screen. To close the image and return to the Analyses tab, click anywhere outside of the image.

Delete a pipeline

Only the user that created or imported a pipeline and administrative users have the ability to delete a pipeline. The delete option is not available to non-administrative users that that did not create or import a particular pipeline. To delete a pipeline:

1. Click on a data node and expand the Pipelines section from the menu on the right. The pop-up menu will only display pipelines that can be applied to the data type of the selected data node

2. Click on the 3 dots next to the pipeline name and choose Delete pipeline\

   

You will be asked if you are sure you want to delete the pipeline. Click Delete.

Import a pipeline

To import a .pipeline file from your local machine into Connected Multiomics:

1. Click the Import a pipeline link at the bottom of the screen under the Analyses tab

2. Click the +Choose file button in Pipeline file section

3. Browse to the location of the .pipeline file on your local machine and select it for upload. The selected file name will be displayed next to the Choose pipeline button

4. If you wish to give the pipeline a new name, type a new name into the Rename pipeline box and click Import.

You will be redirected back to the Analyses tab. When you select a circular data node, the newly imported pipeline will be available in the Pipelines section in the menu on the right, the pipeline will also be added in the pipeline management page.