Illumina® Connected Analytics is a cloud-based software platform intended to be used to manage, analyze, and interpret large volumes of multi-omics data in a secure, scalable, and flexible environment. The versatility of the system allows the platform to be used for a broad range of applications.

When using the applications provided on the platform for diagnostic purposes, it is the responsibility of the user to determine regulatory requirements and to validate for intended use, as appropriate.

The platform is hosted in regions listed below.

The platform hosts a suite of RESTful HTTP-based application programming interfaces (APIs) to perform operations on data and analysis resources. A web application user-interface is hosted alongside the API to deliver an interactive visualization of the resources and enables additional functionality beyond automated analysis and data transfer. Storage and compute costs are presented via usage information in the account console, and a variety of compute resource options are specifiable for applications to fine tune efficiency.

Getting Started

The user documentation provides material for learning the basics of interacting with the platform including examples and tutorials. Start with the documentation to learn more.

Getting Help

Use the search bar on the top right to navigate through the help docs and find specific topics of interest.

If you have any questions, contact Illumina Technical Support by phone or email:

Illumina Technical Support | [email protected] | 1-800-809-4566

For customers outside the United States, Illumina regional Technical Support contact information can be found at www.illumina.com/company/contact-us.html.

To see the current ICA version you are logged in to, click your username found on the top right of the screen and then select About.

Other Illumina Products

To view a list of the products to which you have access, select the 9 dots symbol at the top right of ICA. This will list your products. If you have multiple regional applications for the same product, the region of each is shown between brackets.

The More Tools category presents the following options

• My Illumina Dashboard to monitor instruments, streamline purchases and keep track of upcoming activities.

• Link to the Support Center for additional information and help.

• Link to the order management from where you can keep track of your current and past orders.

Release Notes

In the section of the documentation, posts are made for new versions of deployments of the core platform components.

Illumina® Connected Analytics is a cloud-based software platform intended to be used to manage, analyze, and interpret large volumes of multi-omics data in a secure, scalable, and flexible environment. The versatility of the system allows the platform to be used for a broad range of applications.

Get Started gives an overview of how to access and configure ICA with Network settings providing access prerequisites.

New

To see what is new in the latest version, use the software release notes and the document revision history.

Home

The home section provides an overview of the main ICA sections such as (the main work location), (asset packages), , (to capture additional information), and images (containerised applications) and how to configure (your own) .

Project

Projects are your primary work locations which contain your and . Here you will create and use them for . You configure who can access your project by means of the settings. The results can be processed with the help of , or . Projects can be considered as a binder for your work and information.

Command-Line Interface

This section contains information on how to , , , and the command line interface, as alternative for the ICA GUI.

Sequencer Integration

Information and tutorials on .

Tutorials

There is a set of step-by-step Tutorials

Reference

In the Reference section, you can find more information on the , , ,

For an overview of the available subscription tiers and functionality, please refer to on the Illumina website.

Software Registration

If you are a new user, please consult the Illumina Connected Software Registration Guide for detailed guidance on setting up an account and registering a subscription.

Tenant Setup

The platform requires a provisioned tenant in the () system with access to the Illumina Connected Analytics (ICA) application. Once a tenant has been provisioned, a tenant administrator will be assigned. The tenant administrator has permission to manage account access including add users, create workgroups, and add additional tenant administrators.

Each tenant is assigned a domain name used to login to the platform. The domain name is used in the login URL to navigate to the appropriate login page in a web browser. The login URL is https://<domain_name>.login.illumina.com with <domain_name> replaced by the actual domain name.

New user accounts can be created

For more details on identity and access management, please see the help.

• by the tenant administrator by logging in to their domain and navigating to Illumina Account Management under their profile at the top right

• or by the user by accessing https://platform.login.illumina.com and selecting the option Don't have an account.

Once the account has been added to the domain, the tenant administrator may assign registered users to which bundle users with permission to use the ICA application. Registered users can be made workgroup administrators by tenant administrators or existing workgroup administrators.

API Keys

If you want to use the (CLI) or the (API), you can use an as credentials when logging in. API Keys operate similar to a user name and password combination and must be kept secure and rotated on a regular basis (preferably yearly). `

When keys are compromised or no longer in use, they must be revoked. This is done through the by navigating to the User menu item on the left and selecting "API Keys", followed by selecting the key and using the trash icon next to it.

Generate an API Key

API Keys are limited to 10 per user and are managed through the product dashboard after logging in through the . See for more information.

{% hint style="warning" %} For security reasons, do not use accounts with administrator level access to generate API keys. Create a specific CLI user with basic permissions instead. This will minimize the possible impact of compromised keys. {% endhint %}

{% hint style="warning" %} Once the API key generation window is closed, the key contents will not be accessible through the domain login page, so be sure to store it securely for future reference. {% endhint %}

Access via Web UI

The web application provides a visual user interface (UI) for navigating resources in the platform, managing projects, and extended features beyond the API. To access the web application, navigate to the .

• On the left, you have the navigation bar (1) which will auto-collapse on smaller screens. To collapse it, use the double arrow symbol (2). When collapsed, use the >> symbol to expand it.

• The central part (3) of the display is the item on which you are performing your actions and the breadcrumb menu (4) to return to the projects overview or a previous level. You can also use your browser's back button to return to the level from which you came.

Access via the CLI

The command-line interface offers a developer-oriented experience for interacting with the APIs to manage resources and launch analysis workflows. Find instructions for using the command-line interface including download links for your operating system in the .

Access via the API

The HTTP-based application programming interfaces (APIs) are listed in the section of the documentation. The reference documentation provides the ability to call APIs from the browser page and shows detailed information about the API schemas. HTTP client tooling such as Postman or cURL can be used to make direct calls to the API outside of the browser.

{% hint style="info" %} When accessing the API using the API Reference page or through REST client tools, the Authorization header must be provided with the value set to Bearer <token> where <token> is replaced with a valid JSON Web Token (JWT). For generating a JWT, see . {% endhint %}

Object Identifiers

The object data models for resources that are created in the platform include a unique id field for identifying the resource. These fixed machine-readable IDs are used for accessing and modifying the resource through the API or CLI, even if the resource name changes.

JSON Web Token (JWT)

Accessing the platform APIs requires authorizing calls using JSON Web Tokens (JWT). A JWT is a standardized trusted claim containing authentication context. This is a primary security mechanism to protect against unauthorized cross-account data access.

A JWT is generated by providing user credentials (API Key or username/password) to the token creation endpoint. Token creation can be performed using the API directly or the CLI.

Flow provides tooling for building and running secondary analysis pipelines. The platform supports analysis workflows constructed using Common Workflow Language (CWL) and Nextflow. Each step of an analysis pipeline executes a containerized application using inputs passed into the pipeline or output from previous steps.

You can configure the following components in Illumina Connected Analytics Flow:

• Reference Data — Reference Data for Graphical CWL flows. See .

• Pipelines — One or more tools configured to process input data and generate output files. See

Please see the for all content related to Cloud Analysis Auto-Launch:

The event log shows an overview of system events with options to search and filter. For every entry, it lists the following:

• Event date and time

• Category (error, warn or info)

• Code

• Description

• Tenant

Up to 200,000 results will be be returned. If your desired records are outside the range of the returned records, please refine the filters or use the search function at the top right.

Export is restricted to the amount of entries shown per page. You can use the selector at the bottom to set this to up to 1000 entries per page.

The platform provides Connectors to facilitate automation for operations on data (ie, upload, download, linking).

• Service connectors sync data between your local computer or server and the project's cloud-based data storage.

• Project connectors link data between individual projects.

The CLI supports outputs in table, JSON, and YAML formats. The format is set using the output-format configuration setting through a command line option, environment variable, or configuration file.

Dates are output as UTC times when using JSON/YAML output format and local times when using table format.

To set the output format, use the following setting:

--output-format <string>

• json - Outputs in JSON format

• yaml - Outputs in YAML format

• table - Outputs in tabular format

Running Jobs in a Bench SGE Cluster

Once a cluster is started, the cluster manager can be accessed from the workspace node.

Job resources

User

Every Base user has 1 snowflake username: ICA_U_<id>

User/Project-Bundle

The GWAS and PheWAS tabs in ICA Cohorts allow you to visualize precomputed analysis results for phenotypes/diseases and genes, respectively. Note that these do not reflect the subjects that are part of the cohort that you created.

ICA Cohorts currently hosts GWAS and PheWAS analysis results for approximately 150 quantitative phenotypes (such as "LDL direct" and "sitting height") and about 700 diseases.

Visualize Results from Precomputed Genome-Wide Association Studies (GWAS)

Introduction to Cohorts

ICA Cohorts is a cohort analysis tool integrated with Illumina Connected Analytics (ICA). ICA Cohorts combines subject- and sample-level metadata, such as phenotypes, diseases, demographics, and biometrics, with molecular data stored in ICA to perform tertiary analyses on selected subsets of individuals.

Bench workspaces require setting a Docker image to use as the image for the workspace. Illumina Connected Analytics (ICA) provides a default Docker image with JupyterLab installed.

JupyterLab supports Jupyter Notebook documents (.ipynb). Notebook documents consist of a sequence of cells which may contain executable code, markdown, headers, and raw text.

The JupyterLab Docker image contains the following environment variables:

ICA Python Library

Included in the default JupyterLab Docker image is a python library with APIs to perform actions in ICA, such as add data, launch pipelines, and operate on Base tables. The python library is generated from the using .

The ICA Python library API documentation can be found in folder /etc/ica/data/ica_v2_api_docs within the JupyterLab Docker image.

See the for examples on using the ICA Python library.

Since there is a storage cost associated with the data in your projects, it is good practice to regularly check how much cost is being generated by your projects and evaluate which data can be removed from cloud storage. The instructions provided here will help you quickly determine which data is generating the highest storage costs.

Monitoring Storage Cost

To see how much storage costs are currently being generated for your tenant, you can look at the usage explorer at https://platform.illumina.com/usage/ or from within ICA, navigate to the 9-dot symbol () in the top right next to your name and choose the usage explorer from the menu.

From the usage explorer overview screen, you can see below the graphical representation which projects are incurring the highest storage costs.

Project Files in ICA

When you have determined which projects are incurring the largest storage costs, you can find out which files within that project are taking up the most space. To find the largest files in your project,

1. Go to Projects > your_project > Data and switch to list view with the () icon left above your files.

2. Use the column icon () top right to add the size column to your view. You can drag the size column to the desired position in your list view or use the move left and use right options which appear when you select the three vertical dots.

3. Select Sort descending to show the largest files first.

1. Once you have the list sorted like this, you can evaluate if those large files are still needed, if the can be sent to (manage > archive) or if they can be deleted (manage > delete).

ICA provides a tool called Bench for interactive data analysis. This is a sandboxed workspace which runs a docker image with access to the data and pipelines within a project. This workspace runs on the Amazon S3 system and comes with associated processing and provisioning costs. It is therefore best practice to not keep your Bench instances running indefinitely, but stopping them when not in use.

Access

Having access to Bench depends on the following conditions:

• Bench needs to be included in your ICA subscription.

• The project owner needs to enable Bench for their project.

• Individual users of that project need to be given access to Bench.

Enabling Bench for your project

After creating a project, go to Projects > your_project > Bench > Workspaces page and click the Enable button. The entitlements you have determine the available resources for your Bench workspaces. If you have multiple entitlements, all the resources of your individual entitlements are taken into account. Once bench is enabled, users with matching have access to the Bench module in that project.

Setting user level access.

Once Bench has been enabled for your project, the combination of roles and teams settings determines if a user can access Bench.

• Tenant administrators and project owners are always able to access Bench and perform all actions.

• The teams settings page at Projects > your_project > Project Settings > Team determines the role for the user/workgroup.

  • No Access means you have no access to the Bench workspace for that project.

ICA supports running pipelines defined using Common Workflow Language (CWL).

Compute Type

To specify a compute type for a CWL CommandLineTool, either define the ram and number of cores or use the resource type and size. The ICA Compute Type will automatically be determined based on CWL ResourceRequirement coresMin/coresMax (CPU) and ramMin/ramMax (Memory) values using a "best fit" strategy to meet the minimum specified requirements (See the Compute Types table to see to what the resources are mapped).

For example, take the following ResourceRequirements:

This will result in a best fit of standard-large ICA Compute Type request for the task.

See the example below to use the ResourceRequirement in the cwl workflow with the Predefined

Standard vs Economy

For each compute type, you can choose between the

• Standard - (Default) or

• Economy - tiers.

You can set economy mode with the "tier" parameter

Considerations

If no Docker image is specified, Ubuntu will be used as default. Both : and / can be used as separator.

CWL Overrides

ICA supports overriding workflow requirements at load time using Command Line Interface (CLI) with JSON input. Please refer to for more details on the CWL overrides feature.

In ICA you can provide the "override" recipes as a part of the input JSON. The following example uses CWL overrides to change the environment variable requirement at load time.

You can use samples to group information related to a sample, including input files, output files, and analyses. You can consider samples as creating a binder to collect related information. When you then link that sample to another project, you bring over the empty binder, but not the files contained in it. In that project, you can then add your own data to it.

You can search for samples (excluding their metadata) with the Search button at the top right.

Add New Sample

To add a new sample, do as follows.

1. Select Projects > your_project > Samples.

2. To add a new sample, select + Create, and then enter a unique name and description for the sample.

3. To add files to the sample, see

Your sample is added to the Samples page. To view information on the sample, select the sample name in the overview.

Add Files to Samples

You can add files to a sample after creating the sample. Any files that are not currently linked to a sample are listed on the Unlinked Files tab.

To add an unlinked file to a sample, do as follows.

1. Go to Projects > your_project > Samples > Unlinked files tab.

2. Select a file or files, and then select one of the following options:

   • Create sample — Create a new sample that includes the selected files.

Alternatively, you can add unlinked files from the sample details.

1. Going to Projects > your_project > Samples > your_sample.

2. Select your sample to open the details.

3. Go to the Data tab and select link.

Data can only be linked to a single sample, so once you have linked data to a sample, it will no longer appear in the list of data to choose form.

Unlink Files from Samples

To remove files from samples,

1. Go to Projects > your_project > Samples > your_sample > Data

2. Select the files you want to remove.

3. Select Unlink.

Link Samples to Project

A Sample can be linked to a project from a separate project to make it available in read-only capacity.

1. Navigate to the Samples view in the Project

2. Click the Link button

3. Select the Sample(s) to link to the project

Delete Samples

If you want to remove a sample, select it and use the delete option from the top navigation row. You will be presented a choice of how to handle the data in the sample.

• Unlink all data without deleting it.

• Delete input data and unlink other data.

• Delete all data.

You can verify the integrity of the data by comparing the hash which is usually (with some exceptions) an MD5 (Message Digest Algorithm 5) checksum. This is a common cryptographic hash function that generates a fixed-size, 128-bit hash value from any input data. This hash value is unique to the content of the data, meaning even a slight change in the data will result in a significantly different MD5 checksum. AWS S3 calculates this checksum when data is uploaded and stores it in the ETag (Entity tag).

For files smaller than 16 MB, you can directly retrieve the MD5 checksum using our API endpoints. Make an API GET call to the https://ica.illumina.com/ica/rest/api/projects/{projectId}/data/{dataId} endpoint specifying the data Id you want to check and the corresponding project ID. The response you receive will be in JSON format, containing various file metadata. Within the JSON response, look for the objectETag field. This value is the MD5 checksum for the file you have queried. You can compare this checksum with the one you compute locally to ensure file integrity.

This ETag does not change and can be used as a file integrity check even when that file is archived, unarchived and/or copied to another location. Changes to the metadata have no impact on the ETag

For larger files, the process is different due to computation limitations. In these cases, we recommend using a dedicated pipeline on our platform to explicitly calculate the MD5 checksum. Below you can find both a main.nf file and the corresponding XML for a possible Nextflow pipeline to calculate the MD5 checksum for FASTQ files.

Download links for the CLI can be found at the Release History.

After the file is downloaded, place the CLI in a folder that is included in your $PATH environment variable list of paths, typically /usr/local/bin. Open the Terminal application, navigate to the folder where the downloaded CLI file is located (usually your Downloads folder), and run the following command to copy the CLI file to the appropriate folder. If you do not have write access to your /usr/local/bin folder, then you may use sudo (which requires a password) prior to the cp command. For example:

Mount projectdata using CLI

icav2 allows project data to be mounted on a local system. This feature is currently available on Linux and Mac systems only. Although not supported, users have successfully used Windows Subsystem for Linux (WSL) on Windows to use icav2 projectdata mount command. Please refer to the WSL documentation for installing WSL.

Prerequisites

• icav2 (>=2.3.0) and in the local system.

• • For MAC refer to .

Mount projectdata

Identify the project id by running the following command:

Provide the project id under "ID" column above to the mount command to mount the project data for the project.

Check the content of the mount.

icav2 utilizes the FUSE driver to mount project data, providing both read and write capabilities. However, there are some limitations on the write capabilities that are enforced by the underlying AWS S3 storage. For more information, please refer to .

WARNING Do NOT use the CP -f command to copy or move data to a mounted location. This will result in data loss as data on the destination location will be deleted.

Unmount project data

You can unmount the project data using the 'unmount' command.

The ICA CLI uses an Illumina API Key to authenticate. An Illumina API Key can be acquired through the product dashboard after logging into a domain. See API Keys for instructions to create an Illumina API Key.

Authenticate using icav2 config set command. The CLI will prompt for an x-api-key value. Input the API Key generated from the product dashboard here. See the example below (replace EXAMPLE_API_KEY with the actual API Key).

icav2 config set
Creating /Users/johngenome/.icav2/config.yaml
Initialize configuration settings [default]
server-url [ica.illumina.com]: 
x-api-key : EXAMPLE_API_KEY
output-format (allowed values table,yaml,json defaults to table) : 
colormode (allowed values none,dark,light defaults to none) :

The CLI will save the API Key to the config file as an encrypted value.

If you want to overwrite existing environment values, use the command icav2 config set. To remove an existing configuration/session file, use the command icav2 config reset.\

Check the server and confirm you are authenticated using icav2 config get

If during these steps or in the future you need to reset the authentication, you can do so using the command: icav2 config reset

A storage configuration provides ICA with information to connect to an external cloud storage provider, such as AWS S3. The storage configuration validates that the information provided is correct, and then continuously monitors the integration.

Refer to the following pages for instructions to setup supported external cloud storage providers:

This walk-through is intended to represent a typical workflow when building and studying a cohort of oncology cases.

Create a Cancer Cohort and View Subject Details

1. Click Create Cohort button.

The ICA CLI can be used for uploading, downloading and viewing information about data stored within ICA projects. If not already authenticated, please see the section. Once the CLI has been authenticated with your account, use the command below to list all projects:

icav2`` projects list

The first column of the output (in default table format) will show the ID. This is the project-id and will be used in the examples below.

The project details page contains the properties of the project, such as the location, owner, storage and linked bundles. This is also the place where you add assets in the form of linked .

The project details are configured during project creation and may be updated by the project owner, entities with the project Adminstrator role, and tenant administrators.

Adding Linked Bundle Assets

Running a pyspark application

The JupyterLab environment is by default configured with 3 additional kernels

• PySpark –

Non-indexed folders () are designed for optimal performance in situations where no file actions are needed. They serve as fast storage in situations like temporary analysis file storage where you don't need access or searches via the GUI to individual files or subfolders within the folder. Think of a non-indexed folder as a data container. You can access the container which contains all the data, but you can not access the individual data files within the container from the GUI. As non-indexed folders contain data, they count towards your total project storage.

You can see the size of a non-indexed folder as part of the data details screen (projects > your_project > data > your_non-indexed_data > data details tab) and in the data view (projects > your_project > data).

Bench Workspaces use a FUSE driver to mount project data directly into a workspace file system. There are both read and write capabilities with some limitations on write capabilities that are enforced by the underlying AWS S3 storage.

As a user, you are allowed to do the following actions from Bench (when having the correct user permissions compared to the workspace permissions) or through the CLI:

• Copy project data

• Delete project data

Reference Data are reference genome sets which you use to help look for deviations and to compare your data against.

Creating Reference Data

Reference data properties are located at the main navigation level and consist of the following free text fields.

The ICA CLI accepts configuration settings from multiple places, such as environment variables, configuration file, or passed in as command line arguments. When configuration settings are retrieved, the following precedence is used to determine which setting to apply:

1. Command line options - Passed in with the command such as --access-token

2. Environment variables - Stored in system environment variables such as ICAV2_ACCESS_TOKEN

The Activity view (Projects > your_project > Activity) shows the status and history of long-running activities including Data Transfers, Base Jobs, Base Activity, Bench Activity and Batch Jobs.

Data Transfers

The Data Transfers tab shows the status of data uploads and downloads. You can sort, search and filter on various criteria and export the information. Show ongoing transfers (top right) allows you to filter out the completed and failed transfers to focus on current activity.

Transfers with a yellow background indicate that rules have been modified in ways that prevent planned files from being uploaded. Please verify your service connectors to resolve this.

In ICA Cohorts, metadata describe any subjects and samples imported into the system in terms of attributes, including:

• subject:

  • demographics such as age, sex, ancestry;

Let's create the Nextflow Scatter Gather pipeline with a JSON input form.

Select Projects > your_project > Flow > Pipelines. From the Pipelines view, click the +Create > Nextflow > JSON based button to start creating a Nextflow pipeline.

In the Details tab, add values for the required Code (unique pipeline name) and Description fields. Nextflow Version and Storage size defaults to preassigned values.

Nextflow files

split.nf

First, we present the individual processes. Select Nextflow files > + Create and label the file split.nf. Copy and paste the following definition.

sort.nf

Next, select +Create and name the file sort.nf. Copy and paste the following definition.

merge.nf

Select +Create again and label the file merge.nf. Copy and paste the following definition.

main.nf

Edit the main.nf file by navigating to the Nextflow files > main.nf tab and copying and pasting the following definition.

Here, the operators flatten and collect are used to transform the emitting channels. The Flatten operator transforms a channel in such a way that every item of type Collection or Array is flattened so that each single entry is emitted separately by the resulting channel. The collect operator collects all the items emitted by a channel to a List and return the resulting object as a sole emission.

Inputform files

On the Inputform files tab, edit the inputForm.json to allow selection of a file.

inputForm.json

Click the Simulate button (at the bottom of the text editor) to preview the launch form fields.

The onSubmit.js and onRender.js can remain with their default scripts and are just shown here for reference.

onSubmit.js

onRender.js

Click the Save button to save the changes.

You can compare up to four previously created individual cohorts, to view differences in variants and mutations, RNA expression, copy number variation, and distribution of clinical attributes. Once comparisons are created, they are saved in the Comparisons left-navigation tab of the Cohorts module.

Create a comparison view

1. Select Cohorts from the left-navigation panel.

2. Select 2 to 4 cohorts already created. If you have not created any cohorts, See Create a Cohort documentation.

3. Click Compare Cohorts in the right-navigation panel.

4. Note you are now in the Comparisons left-navigation tab of the Cohorts module.

5. In the Charts Section, if the COHORTS item is not displayed, click the gear icon in the top right and add Cohorts as the first attribute and click Save.

6. The COHORTS item in the charts panel will provide a count of subjects in each cohort and act as a legend for color representation throughout comparison screens.

7. For each clinical attribute category, a bar chart is displayed. Use the gear icon to select attributes to display in the charts panel.

You can share a comparison with other team members in the same ICA Project. Please refer to the section on "Sharing a Cohort" on "Create a Cohort" for details on sharing, unsharing, deleting, and archiving, which are analogous for sharing comparisons.

Attribute Comparison

1. Select the Attributes tab

2. Attribute categories are listed and can be expanded using the down-arrows next to the category names. The categories available are based on cohorts selected. Categories and attributes are part of the ICA Cohorts metadata template that map to each Subject.

3. For example, use the drop-down arrow next to Vital status to view sub-categories and frequencies across selected cohorts.

Variants Comparison

1. Select the Genes tab

2. Search for a gene of interest using its HUGO/HGNC gene symbol

3. Variants and mutations will be displayed as one needle plot for each cohort that is part of the comparison (see in this online help for more details)

Survival Summary

1. Select the Survival Summary tab.

2. Attribute categories are listed and can be expanded using the down-arrows next to the category names.

3. Select the drop-down arrow for Therapeutic interventions.

Survival Comparison

1. Click Survival Comparison tab.

2. A Kaplan-Meier Curve is rendered based on each cohort.

3. P-Value Displayed at the top of Survival Comparison indicates whether there is statistically significant variance between survival probabilities over time of any pair of cohorts (CI=0.95).

When comparing two cohorts, the P-Value is shown above the two survival curves. For three or four cohorts, P-Values are shown as a pair-wise heatmap, comparing each cohort to every other cohort.

Marker Frequency Comparison

1. Select the Marker Frequency tab.

2. Select either Gene expression (default), Somatic mutation, or Copy number variation

Correlation Comparison

1. Select the Correlation tab.

2. Similar to the single-cohort view (Cohort Analysis | Correlation), choose two clinical attributes and/or genes to compare.

3. Depending on the available data types for the two selections (categorical and/or continuous), Cohorts will display a bubble plot, violin plot, or scatter plot.

In order to create a Tool or Bench image, a Docker image is required to run the application in a containerized environment. Illumina Connected Analytics supports both public Docker images and private Docker images uploaded to ICA.

Importing a Public External Image (Tools)

1. Navigate to System Settings > Docker Repository.

2. Click Create > External image to add a new external image.

3. Add your full image URL in the Url field, e.g. docker.io/alpine:latest or registry.hub.docker.com/library/alpine:latest. Docker Name and Version will auto-populate. (Tip: do not add http:// or https:// in your URL)

1. (Optional) Complete the Description field.

2. Click Save.

3. The newly added image will appear in your Docker Repository list. You can differentiate between internal and external images by looking at the Source column. If this column is not visible, you can add it with the columns icon ().

Importing a Private Image (Tools + Bench Images)

In order to use private images in your tool, you must first upload them as a TAR file.

1. Navigate to Projects > your_project .

2. Upload your private image as a TAR file, either by dragging and dropping the file in the Data tab, using the CLI or a Connector. For more information please refer to project .

3. Select your uploaded TAR file and click in the top menu on Manage > Change Format .

Copying Docker Images to other Regions

1. Navigate to System Settings > Docker Repository.

2. Either

   • Select the required image(s) and go to Manage > Add Region.

To remove regions, go to Manage > Remove Region or unselect the regions from the Docker image detail view.

Downloading Docker Images

You can download your created Docker images at System Settings > Docker Images > your_Docker_image > Manage > Download.

In order to be able to download Docker images, the following requirements must be met:

• The Docker image can not be from an entitled bundle.

• Only self-created Docker images can be downloaded.

• The Docker image must be an internal image and in status Available.

File Size Considerations

Docker image size should be kept as small as practically possible. To this end, it is best practice to compress the image. After compressing and uploading the image, select your uploaded file and click Manage > Change Format in the top menu to change it to Docker format so ICA can recognize the file.

Introduction

DRAGEN can run in Bench workspaces

• In either FPGA mode (hardware-accelerated) or software mode when using FPGA instances. This can be useful when comparing performance gains by hardware acceleration or to distribute concurrent processes between the FPGA and cpu.

• In software mode when using non-FPGA instances.

The DRAGEN command line parameters to specify the location of the licence file are different.

• FPGA mode uses LICENSE_PARAMS=``"--lic-instance-id-location /opt/dragen-licence/instance-identity.protected --lic-credentials /opt/dragen-licence/instance-identity.protected/dragen-creds.lic"

• Software mode uses LICENSE_PARAMS="--sw-mode --lic-credentials /opt/dragen-licence/instance-identity.protected/dragen-creds-sw-mode.lic"

DRAGEN Bench Images

DRAGEN software is provided in specific Bench images with names starting with Dragen. For example (available versions may vary):

• Dragen 4.4.1 - Minimal provides DRAGEN 4.4.1 and SSH access

• Dragen 4.4.6 provides DRAGEN 4.4.6, SSH and JupyterLab.

Prerequisites

Memory

The instance type is selected during workspace creation (Projects > your_project > Bench > Workspaces). The amount of RAM available on the instance is critical. 256GiB RAM is a safe choice to run DRAGEN in production. All FPGA2 instances offer 256GiB or more of RAM.

When running in Software mode, use (348GiB RAM) or (144 GiB RAM) to ensure enough RAM is available for your runs.

FPGA-mode

Using an fpga2-medium .

Example

Software-mode

Using a standard-xlarge .

Software mode is activated with the DRAGEN --sw-mode parameter.

Example

Cohorts Walk-through: Rare Genetic Disorders

This walk-through is meant to represent a typical workflow when building and studying a cohort of rare genetic disorder cases.

Login and Create a new ICA Project

Create a new Project to track your study:

1. Login to the ICA

2. Navigate to Projects

3. Create a new project using the New Project button.

Create and Review a Rare Disease Cohort

1. Navigate to the ICA Cohorts module by clicking Cohorts in the left navigation panel.

2. Click Create Cohort button.

3. Enter a name for your cohort, like Rare Disease + 1kGP at top, left of pencil icon.

Analyze Your Rare Disease Cohort Data

1. A recent GWAS publication identified 10 risk genes for intellectual disability (ID) and autism. Our task is to evaluate them in ICA Cohorts: TTN, PKHD1, ANKRD11, ARID1B, ASXL3, SCN2A, FHL1, KMT2A, DDX3X, SYNGAP1.

2. First let’s Hide charts for more visual space.

3. Click the Genes tab where you need to query a gene to see and interact with results.

The platform GUI provides the Project Connector utility which allows data to be linked automatically between projects. This creates a one-way dynamic link for files and samples from source to destination, meaning that additions and deletions of data in the source project also affect the destination project. This differs from copying or moving which create editable copies of the data. In the destination project, you can delete data which has been moved or copied and unlink data which has been linked.

Prepare Source Project

1. Select the source project (project that will own the data to be linked) from the Projects page (Projects > your_source_project).

2. Select Project Settings > Details.

3. Select Edit

Creating a New Project Connector

1. Select the destination project (the project to which data from the source project will be linked) from the Projects page (Projects > your_destination_project).

2. From the projects menu, select Project Settings > Connectivity > Project Connector

3. Select + Create and complete the necessary fields.

Filter Expression Examples

The examples below will restrict linking Files based on the Format field.

• Only Files with Format of FASTQ will be linked:

  [?($.details.format.code == 'FASTQ')]

• Only Files with Format of VCF will be linked:

  [?($.details.format.code == 'VCF')]

The examples below will restrict linked Files based on a filenames.

• Exact match to 'Sample-1_S1_L001_R1_001.fastq.gz':

  [?($.details.name == 'Sample-1_S1_L001_R1_001.fastq.gz')]

• Ends with '.fastq.gz':

  [?($.details.name =~ /.*\.fastq.gz/)]

The examples below will restrict linking Samples based on User Tags and Sample name, respectively.

• Only Samples with the User Tag 'WGS-Project-1'

  [?('WGS-Project-1' in $.tags.userTags)]

• Link a Sample with the name 'BSSH_Sample_1':

  [?($.name == 'BSSH_Sample_1')]

You can access the databases and tables within the Base module using Python from your local machine. Once retrieved as e.g. pandas object, the data can be processed further. In this tutorial, we will describe how you could create a Python script which will retrieve the data and visualize it using Dash framework. The script will contain the following parts:

• Importing dependencies and variables.

• Function to fetch the data from Base table.

• Creating and running the Dash app.

Importing dependencies and variables

This part of the code imports the dependencies which have to be installed on your machine (possibly with pip). Furthermore, it imports the variables API_KEY and PROJECT_ID from the file named config.

Function to fetch the data from Base table

We will be creating a function called fetch_data to obtain the data from Base table. It can be broken into several logically separated parts:

• Retrieving the token to access the Base table together with other variables using API.

• Establishing the connection using the token.

• SQL query itself. In this particular example, we are extracting values from two tables Demo_Ingesting_Metrics and BB_PROJECT_PIPELINE_EXECUTIONS_DETAIL. The table Demo_Ingesting_Metrics contains various metrics from DRAGEN analyses (e.g. the number of bases with quality at least 30 Q30_BASES) and metadata in the column ica which needs to be flattened to access the value

Here is the corresponding snippet:

Creating and running the Dash app

Once the data is fetched, it is visualized in an app. In this particular example, a scatter plot is presented with END_DATE as x axis and the choice of the customer from the dropdown as y axis.

Now we can create a single Python script called dashboard.py by concatenating the snippets and running it. The dashboard will be accessible in the browser on your machine.

You can access the databases and tables within the Base module using snowSQL command-line interface. This is useful for external collaborators who do not have access to ICA core functionalities. In this tutorial we will describe how to obtain the token and use it for accessing the Base module. This tutorial does not cover how to install and configure snowSQL.

Obtaining OAuth token and URL

Once the Base module has been enabled within a project, the following details are shown in Projects > your_project > Project Settings > Details.

After clicking the button Create OAuth access token, the pop-up authenticator is displayed.

After clicking the button Generate snowSQL command the pop-up authenticator presents the snowSQL command.

Copy the snowSQL command and run it in the console to log in.

You can also get the OAuth access token via API by providing <PROJECT ID> and <YOUR KEY>.

Example:

API Call:

Response

Template snowSQL:

Now you can perform a variety of tasks such as:

1. Querying Data: execute SQL queries against tables, views, and other database objects to retrieve data from the Snowflake data warehouse.

2. Creating and Managing Database Objects: create tables, views, stored procedures, functions, and other database objects in Snowflake. you can also modify and delete these objects as needed.

3. Loading Data: load data into Snowflake from various sources such as local files, AWS S3, Azure Blob Storage, or Google Cloud Storage.

Overall, snowSQL CLI provides a powerful and flexible interface to work with Snowflake, allowing external users to manage data warehouse and perform a variety of tasks efficiently and effectively without access to the ICA core.

Example Queries:

Show all tables in the database:

Create a new table:

List records in a table:

Load data from a file: To load data from a file, you can start by create a staging area in the internal storage using the following commend:

You can then upload the local file to the internal storage using the following command:

You can check if the file was uploaded properly using LIST command:

Finally, Load data by using COPY TO command. The command assumes the data.tsv is a tab delimited file. You can easily modify the following command to import JSON file setting TYPE=JSON.

Load data from a string: If you have data as JSON string, you can import the data into the tables using following commands.

Load data into specific columns: If you want to load sample_name into the table, you can remove the "count" from the column and the value list as below:

List the views of the database to which you are connected. As shared database and catalogue views are created within the project database, they will be listed. However, it does not show views which are granted via another database, role or from bundles.

Show grants, both directly on the tables and views and grants to roles which in turn have grants on tables and views.

Nextflow supports scatter-gather patterns natively through Channels. The initial example uses this pattern by splitting the FASTA file into chunks to channel records in the task splitSequences, then by processing these chunks in the task reverse.

In this tutorial, we will create a pipeline which will split a TSV file into chunks, sort them, and merge them together.

Creating the pipeline

Select Projects > your_project > Flow > Pipelines. From the Pipelines view, click the +Create > Nextflow > XML based button to start creating a Nextflow pipeline.

In the Details tab, add values for the required Code (unique pipeline name) and Description fields. Nextflow Version and Storage size defaults to preassigned values.

First, we present the individual processes. Select +Nextflow files > + Create and label the file split.nf. Copy and paste the following definition.

Next, select +Create and name the file sort.nf. Copy and paste the following definition.

Select +Create again and label the file merge.nf. Copy and paste the following definition.

Add the corresponding main.nf file by navigating to the Nextflow files > main.nf tab and copying and pasting the following definition.

Here, the operators flatten and collect are used to transform the emitting channels. The Flatten operator transforms a channel in such a way that every item of type Collection or Array is flattened so that each single entry is emitted separately by the resulting channel. The collect operator collects all the items emitted by a channel to a List and return the resulting object as a sole emission.

Finally, copy and paste the following XML configuration into the XML Configuration tab.

Click the Generate button (at the bottom of the text editor) to preview the launch form fields.

Click the Save button to save the changes.

Running the pipeline

Go to the Pipelines page from the left navigation pane. Select the pipeline you just created and click Start New Analysis.

Fill in the required fields indicated by red "*" sign and click on Start button. You can monitor the run from the Analyses page. Once the Status changes to Succeeded, you can click on the run to access the results page.

In Projects > your_project > Flow > Analyses > your_analysis > Steps you can see that the input file is split into multiple chunks, then these chunks are sorted and merged.

Introduction to Base

Base is a genomics data aggregation and knowledge management solution suite. It is a secure and scalable integrated genomics data analysis solution which provides Information management and knowledge mining. You can analyze, aggregate and query data for new insights that can inform and improve diagnostic assay development, clinical trials, patient testing and patient care. Clinically relevant data needs to be generated and extracted from routine clinical testing and clinical questions need to be asked across all data and information sources. As a large data store, Base provides a secure and compliant environment to accumulate data, allowing for efficient exploration of the aggregated data. This data consists of test results, patient data, metadata, reference data, consent and QC data.

Use Cases

Base can be used by for different use cases:

• Clinical and Academic Researchers:

  • Big data storage solution housing all aggregated sample test outcomes

  • Analyze information by way of a convenient query formalism

Base Action Possibilities

• Data Warehouse Creation: Build a relational database for your Project in which desired data sets can be selected and aggregated. Typical data sets include pipeline output metrics and other suitable data files generated by the ICA platform which can be complemented by additional public (or privately built) databases.

• Report and Export: Once created, a data warehouse can be mined using standard database query instructions. All Base data is stored in a structured and easily accessible way. An interface allows for the selection of specific datasets and conditional reporting. All queries can be stored, shared, and re-used in the future. This type of standard functionality supports most expected basic mining operations, such as variant frequency aggregation. All result sets can be downloaded or exported in various standard data formats for integration in other reporting or analytical applications.

Access

The Base module can be found at Projects > your_project > Base. In order to use Base, you need to meet the following requirements:

Subscription

On the domain level, Base needs to be included in the subscription (full and premium subscriptions give access to Base).

Enabling Base

Once a project is created, the project owner must navigate to Projects > your_project > Base and click the Enable button. From that moment on, every user who has the proper permissions has access to the Base module in that project.

Enabling User Access

Access to the projects and Base is configured on the Projects > your_project > Project settings > Team page. Here you can add or edit a user or workgroup and give them .

Activity

The status and history of Base activities and jobs are shown on the page.

Developing on the cloud incurs inherent runtime costs due to compute and storage used to execute workflows. Here are a few tips that can facilitate development.

• Leverage the cross-platform nature of these workflow languages. Both CWL and Nextflow can be run locally in addition to on ICA. When possible, testing should be performed locally before attempting to run in the cloud. For Nextflow, can be utilized to specify settings to be used either locally or on ICA. An example of advanced usage of a config would be applying the to a set of process names (or labels) so that they use the higher performance local scratch storage attached to an instance instead of the shared network disk,

• When trying to test on the cloud, it's oftentimes beneficial to create scripts to automate the deployment and launching / monitoring process. This can be performed either using the

On the Schedule page at Projects > your_project > Base > Schedule, it’s possible to create a job for importing different types of data you have access to into an existing table.

When creating or editing a schedule, Automatic import is performed when the Active box is checked. The job will run at 10 minute intervals. In addition, for both active and inactive schedules, a manual import is performed when selecting the schedule and clicking the »run button.

Configure a schedule

ICA Cohorts comes front-loaded with a variety of publicly accessible data sets, covering multiple disease areas and also including healthy individuals.

This tutorial aims to guide you through the process of creating CWL tools and pipelines from the very beginning. By following the steps and techniques presented here, you will gain the necessary knowledge and skills to develop your own pipelines or transition existing ones to ICA.

Build and push to ICA your own Docker image

The foundation for every tool in ICA is a Docker image (externally published or created by the user). Here we present how to create your own Docker image for the popular tool (FASTQC).

Copy the contents displayed below to a text editor and save it as a Dockerfile. Make sure you use an editor which does not add formatting to the file.

How to lift a simple NextFlow pipeline?

In this tutorial, we will be using the example RNASeq pipeline to demonstrate the process of lifting a simple Nextflow pipeline over to ICA.

This approach is applicable in situations where your main.nf file contains all your pipeline logic and illustrates what the liftover process would look like.

Managing a Bench cluster

Introduction

Workspaces can have their own dedicated cluster which consists of a number of nodes. First the workspace node, which is used for interacting with the cluster, is started. Once the workspace node is started, the workspace cluster can be started.