Reference Data are reference genome sets which you use to help look for deviations and to compare your data against.

Creating Reference Data

Reference data properties are located at the main navigation level and consist of the following free text fields.

• Types

• Species

• Reference Sets

Once these are configured,

1. Go to your data in Projects > your_project > Data.

2. Select the data you want to use as reference data and Manage > Use as reference data.

3. Fill out the configuration screen

You can see the result at the main navigation level > Reference Data (outside of projects) or in Projects > your_project > Flow > Reference Data.

Linking Reference Data to your Project

To use a reference set from within a project, you have first to add it. Select Projects > your_project > Flow > Reference Data > Link. Then select a reference set to add to your project.

Copying Reference Data to other Regions

1. Navigate to Reference Data (Not from within a project, but outside of project context, so at the main navigation level).

2. Select the data set(s) you wish to add to another region and select Copy to another project.

3. Select a project located in the region where you want to add your reference data.

Creating a Pipeline with Reference Data

To create a pipeline with a reference data, use the mode. Projects > your_project > Flow > Pipelines > +Create > CWL Graphical. Use the reference data icon instead of regular input icon. On the right hand side use the Reference files submenu to specify the name, the format, and the filters. You can specify the options for an end-user to choose from and a default selection. You can select more than 1 file, but you can only select 1 at a time (so, repeat process to select multiple reference files). If you only select 1 reference file, that file will be the only one users can use with your pipeline. In the screenshot a reference data with two options is presented.

If your pipeline was built to give users the option of choosing among multiple input reference files, they will see the option to select among the reference files you configured, under Settings. After clicking the magnifying glass icon the user can select from provided options.

Flow provides tooling for building and running secondary analysis pipelines. The platform supports analysis workflows constructed using Common Workflow Language (CWL) and Nextflow. Each step of an analysis pipeline executes a containerized application using inputs passed into the pipeline or output from previous steps.

You can configure the following components in Illumina Connected Analytics Flow:

• Reference Data — Reference Data for Graphical CWL flows. See Reference Data.

• Pipelines — One or more tools configured to process input data and generate output files. See Pipelines.

• Analyses — Launched instance of a pipeline with selected input data. See .

Developing on the cloud incurs inherent runtime costs due to compute and storage used to execute workflows. Here are a few tips that can facilitate development.

• Leverage the cross-platform nature of these workflow languages. Both CWL and Nextflow can be run locally in addition to on ICA. When possible, testing should be performed locally before attempting to run in the cloud. For Nextflow, configuration files can be utilized to specify settings to be used either locally or on ICA. An example of advanced usage of a config would be applying the scratch directive to a set of process names (or labels) so that they use the higher performance local scratch storage attached to an instance instead of the shared network disk,

• When trying to test on the cloud, it's oftentimes beneficial to create scripts to automate the deployment and launching / monitoring process. This can be performed either using the or by creating your own scripts integrating with the REST API.

• For scenarios in which instances are terminated prematurely (for example, while using spot instances) without warning, you can implement scripts like the following to retry the job a certain number of times. Adding the following script to 'nextflow.config' enables five retries for each job, with increasing delays between each try.

  Note: Adding the retry script where it is not needed might introduce additional delays.

• When hardening a Nextflow to handle resource shortages (for example exit code 2147483647), an immediate retry will in most circumstances fail because the resources have not yet been made available. It is best practice to use which has an increasing backoff delay, allowing the system time to provide the necessary resources.

• When publishing your Nextflow pipeline, make sure your have defined a container such as 'public.ecr.aws/lts/ubuntu:22.04' and are not using the default container 'ubuntu:latest'.

• To limit potential costs, there is a timeout of 96 hours: if the analysis does not complete within four days, it will go to a 'Failed' state. This time begins to count as soon as the input data is being downloaded. This takes place during the ICA 'Requested' step of the analysis, before going to 'In Progress'. In case parallel tasks are executed, running time is counted once. As an example, let's assume the initial period before being picked up for execution is 10 minutes and consists of the request, queueing and initializing. Then, the data download takes 20 minutes. Next, a task runs on a single node for 25 minutes, followed by 10 minutes of queue time. Finally, three tasks execute simultaneously, each of them taking 25, 28, and 30 minutes, respectively. Upon completion, this is followed by uploading the outputs for one minute. The overall analysis time is then 20 + 25 + 10 + 30 (as the longest task out of three) + 1 = 86 minutes:

If there are no available resources or your project priority is low, the time before download commences will be substantially longer.

• By default, Nextflow will not generate the trace report. If you want to enable generating the report, add the section below to your userNextflow.config file.

1. Useful Links

A Pipeline is a series of Tools with connected inputs and outputs configured to execute in a specific order.

Linking Existing Pipelines

Linking a pipeline (Projects > your_project > Flow > Pipelines > Link) adds that pipeline to your project. This is not as a copy, but as the actual pipeline, so any changes to the pipeline are atomatically propagated to and from any project which has this pipeline linked.

You can link a pipeline if it is not already linked to your project and it is from your tenant or available in your bundle or activation code.

If you unlink a pipeline it removes the pipline from your project, but it remains part of the list of pipelines of your tenant, so it can be linked to other projects later on.

Create a Pipeline

Pipelines are created and stored within projects.

1. Navigate to Projects > your_project > Flow > Pipelines > +Create.

2. Select Nextflow (XML / JSON / ) , CWL Graphical or CWL code (XML / JSON / ) to create a new Pipeline.

3. Configure pipeline settings in the pipeline property tabs.

Pipeline Statuses

For pipeline authors sharing and distributing their pipelines, the draft, released, deprecated, and archived statuses provide a structured framework for managing pipeline availability, user communication, and transition planning. To change the pipeline status, select it at Projects > your_project > Pipelines > your_pipeline > change status.

Pipeline Properties

The following sections describe the properties that can be configured in each tab of the pipeline editor.

Depending on how you design the pipeline, the displayed tabs differ between the graphical and code definitions. For CWL you have a choice on how to define the pipeline, Nextflow is always defined in code mode.

Any additional source files related to your pipeline will be displayed here in alphabetical order.

See the following pages for language-specific details for defining pipelines:

Details

The details tab provides options for configuring basic information about the pipeline.

The following information becomes visible when viewing the pipeline details.

The clone action will be shown in the pipeline details at the top-right. Cloning a pipeline allows you to create modifications without impacting the original pipeline. When cloning a pipeline, you become the owner of the cloned pipeline. When you clone a pipeline, you must give it a unique name because no duplicate names are allowed within all projects of the tenant. So the name must be unique per tenant. It is possible that you see the same pipeline name twice when a pipeline linked from another tenant is cloned with that same name in your tenant. The name is then still unique per tenant, but you will see them both in your tenant.

When you clone a Nextflow pipeline, a verification of the configured Nextflow version is done to prevent the use of deprecated versions.

Documentation

The Documentation tab provides is the place where you explain how your pipeline works to users. The description appears in the tool repository but is excluded from exported CWL definitions. If no documentation has been provided, this tab will be empty.

Definition (Graphical)

When using graphical mode for the pipeline definition, the Definition tab provides options for configuring the pipeline using a visualization panel and a list of component menus.

XML Configuration / JSON Inputform Files (Code)

This page is used to specify all relevant information about the pipeline parameters.

Compute Resources

Compute Nodes

For each process defined by the workflow, ICA will launch a compute node to execute the process.

• For each compute type, the standard (default - AWS on-demand) or economy (AWS spot instance) tiers can be selected.

• When selecting an fpga instance type for running analyses on ICA, it is recommended to use the medium size. While the large size offers slight performance benefits, these do not proportionately justify the associated cost increase for most use cases.

• When no type is specified, the default type of compute node is

By default, compute nodes have no scratch space. This is an advanced setting and should only be used when absolutely necessary as it will incur additional costs and may offer only limited performance benefits because it is not local to the compute node.

For simplicity and better integration, consider using shared storage available at /ces. It is what is provided in the Small/Medium/Large+ compute types. This shared storage is used when writing files with relative paths.

Compute Types

Daemon sets and system processes consume approximately 1 CPU and 2 GB Memory from the base values shown in the table. Consumption will vary based on the activity of the pod.

Nextflow/CWL Files (Code)

Syntax highlighting is determined by the file type, but you can select alternative syntax highlighting with the drop-down selection list. The following formats are supported:

• DIFF (.diff)

• GROOVY (.groovy .nf)

• JAVASCRIPT (.js .javascript)

• JSON (.json)

Main.nf (Nextflow code)

The Nextflow project main script.

Nextflow.config (Nextflow code)

The Nextflow configuration settings.

Workflow.cwl (CWL code)

The Common Workflow Language main script.

Adding Files

Multiple files can be added by selecting the +Create option at the bottom of the screen to make pipelines more modular and manageable.

Metadata Model

See

Report

Here patterns for detecting report files in the analysis output can be defined. On opening an analysis result window of this pipeline, an additional tab will display these report files. The goal is to provide a pipeline-specific user-friendly representation of the analysis result.

To add a report select the + symbol on the left side. Provide your report with a unique name, a regular expression matching the report and optionally, select the format of the report. This must be the source format of the report data generated during the analysis.

Start a New Analysis

Use the following instructions to start a new analysis for a single pipeline.

1. Select Projects > your_project > Flow > Pipelines.

2. Select the pipeline or pipeline details of the pipeline you want to run.

3. Select Start Analysis.

Analysis Settings

The Start Analysis screen provides the configuration options for the analysis.

¹ When using the API, you can to be outside of the current project.

Aborting Analyses

You can abort a running analysis from either the analysis overview (Projects > your_project > Flow > Analyses > your_analysis > Manage > Abort) or from the analysis details (Projects > your_project > Flow > Analyses > your_analysis > Details tab > Abort).

View Analysis Results

You can view analysis results on the Analyses page or in the output folder on the Data page.

1. Select a project, and then select the Flow > Analyses page.

2. Select an analysis.

3. From the output files tab, expand the list if needed and select an output file.

ICA supports running pipelines defined using .

Compute Type

To specify a compute type for a CWL CommandLineTool, either define the ram and number of cores or use the resource type and size. The ICA Compute Type will automatically be determined based on coresMin/coresMax (CPU) and ramMin/ramMax (Memory) values using a "best fit" strategy to meet the minimum specified requirements (See the table to see to what the resources are mapped).

For example, take the following

Let's create the Nextflow Scatter Gather pipeline with a JSON input form.

Select Projects > your_project > Flow > Pipelines. From the Pipelines view, click the +Create > Nextflow > JSON based button to start creating a Nextflow pipeline.

In the Details tab, add values for the required Code (unique pipeline name) and Description fields. Nextflow Version and Storage size defaults to preassigned values.

Nextflow files

split.nf

First, we present the individual processes. Select Nextflow files > + Create and label the file split.nf. Copy and paste the following definition.

sort.nf

Next, select +Create and name the file sort.nf. Copy and paste the following definition.

merge.nf

Select +Create again and label the file merge.nf. Copy and paste the following definition.

main.nf

Edit the main.nf file by navigating to the Nextflow files > main.nf tab and copying and pasting the following definition.

Here, the operators flatten and collect are used to transform the emitting channels. The Flatten operator transforms a channel in such a way that every item of type Collection or Array is flattened so that each single entry is emitted separately by the resulting channel. The collect operator collects all the items emitted by a channel to a List and return the resulting object as a sole emission.

Inputform files

On the Inputform files tab, edit the inputForm.json to allow selection of a file.

inputForm.json

Click the Simulate button (at the bottom of the text editor) to preview the launch form fields.

The onSubmit.js and onRender.js can remain with their default scripts and are just shown here for reference.

onSubmit.js

onRender.js

Click the Save button to save the changes.

Pipelines defined using the "Code" mode require either an XML-based or JSON-based input form to define the fields shown on the launch view in the user interface (UI). The XML-based input form is defined in the "XML Configuration" tab of the pipeline editing view.

The input form XML must adhere to the input form schema.

Empty Form

During the creation of a Nextflow pipeline the user is given an empty form to fill out.

ICA supports running pipelines defined using . See for an example.

In order to run Nextflow pipelines, the following process-level attributes within the Nextflow definition must be considered.

System Information

An Analysis is the execution of a pipeline.

Starting Analyses

You can start an analysis from both the dedicated analysis screen or from the actual pipeline.

From Analyses

1. Navigate to Projects > Your_Project > Flow > Analyses.

2. Select Start.

3. Select a single Pipeline.

From Pipelines or Pipeline details

1. Navigate to Projects > <Your_Project> > Flow > Pipelines

2. Select the pipeline you want to run or open the pipeline details of the pipeline which you want to run.

3. Select Start Analysis.

Aborting Analyses

You can abort a running analysis from either the analysis overview (Projects > your_project > Flow > Analyses > your_analysis > Manage > Abort) or from the analysis details (Projects > your_project > Flow > Analyses > your_analysis > Details tab > Abort).

Rerunning Analyses

Once an analysis has been executed, you can rerun it with the same settings or choose to modify the parameters when rerunning. Modifying the parameters is possible on a per-analysis basis. When selecting multiple analyses at once, they will be executed with the original parameters. Draft pipelines are subject to updates and thus can result in a different outcome when rerunning. ICA will display a warning message to inform you of this when you try to rerun an analysis based on a draft pipeline.

When there is an XML configuration change on a a pipeline for which you want to rerun an analysis, ICA will display a warning and not fill out the parameters as it cannot guarantee their validity for the new XML. You will need to provide the input data and settings again to rerun the analysis.

Some restrictions apply when trying to rerun an analysis.

To rerun one or more analyses with te same settings:

1. Navigate to Projects > Your_Project > Flow > Analyses.

2. In the overview screen, select one or more analyses.

3. Select Manage > Rerun. The analyses will now be executed with the same parameters as their original run.

To rerun a single analysis with modified parameters:

1. Navigate to Projects > Your_Project > Flow > Analyses.

2. In the overview screen, open the details of the analysis you want to rerun by clicking on the analysis user reference.

3. Select Rerun. (at the top right)

Lifecycle

When an analysis is started, the availability of resources may impact the start time of the pipeline or specific steps after execution has started. Analyses are subject to delay when the system is under high load and the availability of resources is limited.

During analysis start, ICA runs a verification on the input files to see if they are available. When it encounters files that have not completed their upload or transfer, it will report "Data found for parameter [parameter_name], but status is Partial instead of Available". Wait for the file to be available and restart the analysis.

Analysis steps logs

During the execution of an analysis, logs are produced for each process involved in the analysis lifecyle. In the analysis details view, the Steps tab is used to view the steps in near real time as they're produced in the running processes. A grid layout is used for analyses with more than 50 steps, a tiled view for analyses with 50 steps or less, though you can choose to also use the grid layout for those by means of the tile/grid button on the top right of the analysis log tab. The steps tab also shows which resources were used as compute type in the different main analysis steps. (For child steps, these are displayed on the parent step)

There are system processes involved in the lifeycle for all analyses (ie. downloading inputs, uploading outputs, etc.) and there are processes which are pipeline-specific, such as processes which execute the pipeline steps. The below table describes the system processes. You can choose to display or hide these system processes with the Show technical steps

Additional log entries will show for the processes which execute the steps defined in the pipeline.

Each process shows as a distinct entry in the steps view with a Queue Date, Start Date, and End Date.

The time between the Start Date and the End Date is used to calculate the duration. The time of the duration is used to calculate the usage-based cost for the analysis. Because this is an active calculation, sorting on this field is not supported.

Each log entry in the Steps view contains a checkbox to view the stdout and stderr log files for the process. Clicking a checkbox adds the log as a tab to the log viewer where the log text is displayed and made available for download.

Analysis Cost

To see the price of an analysis in iCredits, look at Projects > your_project > Flow > Analyses > your_analysis > Details tab. The pricing section will show you the entitlement bundle, storage detail and price in iCredits once the analysis has succeeded, failed or been aborted.

Log Files

By default, the stdout and stderr files are located in the ica_logs subfolder within the analysis. This location can be changed by selecting a different in the current project at the start of the analysis. Do not use a folder which already contains log files as these will be overwritten. To set the log file location, you can also use the CreateAnalysisLogs section of the Create Analysis .

You can access the log files from the analysis details (projects > your_project > flow > analysis > your_analysis > details tab)

Log Streaming

Logs can also be streamed using websocket client tooling. The API to retrieve analysis step details returns websocket URLs for each step to stream the logs from stdout/stderr during the step's execution. Upon completion, the websocket URL is no longer available.

Analysis Output Mappings

By default, analysis outputs are directed to a new folder within the project where the analysis is launched. Analysis output mappings may be specified to redirect outputs to user-specified locations consisting of project and path. An output mapping consists of:

• the source path on the local disk of the analysis execution environment, relative to the working folder.

• the data type, either FILE or FOLDER

• the target project ID to direct outputs to; analysis launcher must have contributor access to the project.

You can jump from the Analysis Details to the individual files and folders by opening the output files tab on the detail view (Projects > your_project > Flow > Analyses > your_analysis > Output files tab > your_output_file) and selecting Open in data.

Tags

You can add and remove tags from your analyses.

1. Navigate to Projects > Your_Project > Flow > Analyses.

2. Select the analyses whose tags you want to change.

3. Select Manage > Manage tags.

Both system tags and customs tags exist. User tags are custom tags which you set to help identify and process information while technical tags are set by the system for processing. Both run-in and run-out tags are set on data to identify which analyses use the data. Connector tags determine data entry methods and reference data tags identify where data is used as reference data.

Hyperlinking

If you want to share a link to an analysis, you can copy and paste the URL from your browser when you have the analysis open. The syntax of the analysis link will be <hostURL>/ica/link/project/<projectUUID>/analysis/<analysisUUID>. Likewise, workflow sessions will use the syntax <hostURL>/ica/link/project/<projectUUID>/workflowSession/<workflowsessionUUID>. To prevent third parties from accessing data via the link when it is shared or forwarded, ICA will verify the access rights of every user when they open the link.

Restrictions

Input for analysis is limited to a total of 50,000 files (including multiple copies of the same file). Concurrency limits on analyses prevent resource hogging which could result in resource starvation for other tenants. Additional analyses will be queued and scheduled when currently running analyses complete and free up positions. The theoretical limit is 20, but this can be less in practice, depending on a number of external factors.

Troubleshooting

When your analysis fails, open the analysis details view (Projects > your_project> Flow > Analyses > your_analysis) and select display failed steps. This will give you the steps view filtered on those steps that had non-0 exit codes. If there is only one failed step which has logfiles, the stderr of that step will be displayed.

• Exit code 55 indicates analysis failure on economy instances due to an external event such as spot termination. You can retry the analysis.

• Exit code 56 indicates analysis failure due to pod disruption and deletion by Kubernetes' Pod Garbage Collector (PodGC) because the node it was running on no longer exists. You can retry the anlaysis.

In the InputForm.json, use the syntax for the individual components you want to as listed below. This is a listing of all the currently available components and not to be used "as is", but adapted to the inputs you need in your inputform. For more information on JSON schema syntax, please see the .

Introduction

Pipelines defined using the "Code" mode require an XML or JSON-based input form to define the fields shown on the launch view in the user interface (UI).

To create a JSON-based Nextflow (or CWL) pipeline, go to Projects > your_project > Flow > Pipelines > +Create > Nextflow (or CWL) > JSON-based.

Three files, located on the inputform files tab, work together for evaluating and presenting JSON-based input.

• inputForm.json contains the actual input form which is rendered when starting the pipeline run.

• onRender.js is triggered when a value is changed.

• onSubmit.js is triggered when starting a pipeline via the GUI or API.

Use + Create to add additional files and Simulate to test your inputForms.

Scripting execution supports crossfield validation of the values, hiding fields, making them required, .... based on value changes.

inputForm.json

The JSON schema allowing you to define the input parameters. See the page for syntax details.

Parameter types

Parameter Attributes

These attributes can be used to configure all parameter types.

Experimental Features

onSubmit.js

The onSubmit.js javascript function receives an input object which holds information about the chosen values of the input form and the pipeline and pipeline execution request parameters. This javascript function is not only triggered when submitting a new pipeline execution request in the user interface, but also when submitting one through the rest API..

Input parameters

Return values (taken from the response object)

AnalysisError

This is the object used for representing validation errors.

onRender.js

Receives an input object which contains information about the current state of the input form, the chosen values and the field value change that triggered the onrender call. It also contains pipeline information. Changed objects are present in the onRender return value object. Any object not present is considered to be unmodified. Changing the storage size in the start analysis screen triggers an onRender execution with storageSize as changed field.

Input Parameters

Return values (taken from the response object)

RenderMessage

This is the object used for representing validation errors and warnings. The attributes can be used with first letter lowercase (consistent with the input object attributes) or uppercase.