When using the applications provided on the platform for diagnostic purposes, it is the responsibility of the user to determine regulatory requirements and to validate for intended use, as appropriate.

The platform is hosted in regions listed below.

The platform hosts a suite of RESTful HTTP-based application programming interfaces (APIs) to perform operations on data and analysis resources. A web application user-interface is hosted alongside the API to deliver an interactive visualization of the resources and enables additional functionality beyond automated analysis and data transfer. Storage and compute costs are presented via usage information in the account console, and a variety of compute resource options are specifiable for applications to fine tune efficiency.

Getting Started

The user documentation provides material for learning the basics of interacting with the platform including examples and tutorials. Start with the Get Started documentation to learn more.

Getting Help

Use the search bar on the top right to navigate through the help docs and find specific topics of interest.

If you have any questions, contact Illumina Technical Support by phone or email:

Illumina Technical Support | [email protected] | 1-800-809-4566

For customers outside the United States, Illumina regional Technical Support contact information can be found at www.illumina.com/company/contact-us.html.

To see the current ICA version you are logged in to, click your username found on the top right of the screen and then select About.

Other Illumina Products

To view a list of the products to which you have access, select the 9 dots symbol at the top right of ICA. This will list your products. If you have multiple regional applications for the same product, the region of each is shown between brackets.

The More Tools category presents the following options

• My Illumina Dashboard to monitor instruments, streamline purchases and keep track of upcoming activities.

• Link to the Support Center for additional information and help.

• Link to the order management from where you can keep track of your current and past orders.

Release Notes

In the Release Notes section of the documentation, posts are made for new versions of deployments of the core platform components.

Help shape the future of ICA

Get Started gives an overview of how to access and configure ICA with Network settings providing access prerequisites.

New

To see what is new in the latest version, use the software release notes and the document revision history.

Home

The home section provides an overview of the main ICA sections such as projects (the main work location), bundles (asset packages), logging, metadata (to capture additional information), Docker and tool images (containerised applications) and how to configure (your own) storage.

Project

Projects are your primary work locations which contain your data and samples. Here you will create pipelines and use them for analyses. You configure who can access your project by means of the teams settings. The results can be processed with the help of Base, Bench or Cohorts. Projects can be considered as a binder for your work and information.

Command-Line Interface

This section contains information on how to download, install, authenticate, configure and use the command line interface, as alternative for the ICA GUI.

Sequencer Integration

Information and tutorials on cloud analysis auto launch.

Tutorials

There is a set of step-by-step Tutorials

• Nextflow

• CWL CLI

• Base

• Bench

• API

• DRAGEN by CLI

• Data Transfer

• Pipeline Chaining

• DRAGEN Analysis

Reference

In the Reference section, you can find more information on the API, Pricing, Security and Compliance,

For an overview of the available subscription tiers and functionality, please refer to this page on the Illumina website.

Flow provides tooling for building and running secondary analysis pipelines. The platform supports analysis workflows constructed using Common Workflow Language (CWL) and Nextflow. Each step of an analysis pipeline executes a containerized application using inputs passed into the pipeline or output from previous steps.

You can configure the following components in Illumina Connected Analytics Flow:

• Reference Data — Reference Data for Graphical CWL flows. See .

• Pipelines — One or more tools configured to process input data and generate output files. See .

• Analyses — Launched instance of a pipeline with selected input data. See .

The event log shows an overview of system events with options to search and filter. For every entry, it lists the following:

• Event date and time

• Category (error, warn or info)

• Code

• Description

• Tenant

Up to 200,000 results will be be returned. If your desired records are outside the range of the returned records, please refine the filters or use the search function at the top right.

Export is restricted to the amount of entries shown per page. You can use the selector at the bottom to set this to up to 1000 entries per page.

User

Every Base user has 1 snowflake username: ICA_U_<id>

User/Project-Bundle

For each user/project-bundle combination a role is created: ICA_UR_<id>_<name project/bundle>__<id>

This role receives the viewer or contributor role of the project/bundle, depending on their permissions in ICA.

Roles

Every project or bundle has a dedicated Snowflake database.

For each database, 2 roles are created:

• <project/bundle name>_<id>_VIEWER

• <project/bundle name>_<id>_CONTRIBUTOR

Project viewer role

This role receives

• REFERENCE and SELECT rights on the tables/views within the project's PUBLIC schema.

• Grants on the viewer roles of the bundles linked to the project.

Project contributor role

This role receives the following rights on current an future objects in the project's/bundle database in the PUBLIC schema:

• ownership

• select, insert, update, delete, truncate and references on tables/views/materialized views

• usage on sequences/functions/procedures/file formats

• write, read and usage on stages

• select on streams

• monitor and operate on tasks

It also receives grant on the viewer role of the project.

Warehouses

For each project (not bundle!) 2 warehouses are created, whose size can be changed ICA at projects > your_project > project settings > details.

• <projectname>_<id>_QUERY

• <projectname>_<id>_LOAD

The CLI supports outputs in table, JSON, and YAML formats. The format is set using the output-format configuration setting through a command line option, environment variable, or configuration file.

Dates are output as UTC times when using JSON/YAML output format and local times when using table format.

To set the output format, use the following setting:

--output-format <string>

• json - Outputs in JSON format

• yaml - Outputs in YAML format

• table - Outputs in tabular format

Since there is a storage cost associated with the data in your projects, it is good practice to regularly check how much cost is being generated by your projects and evaluate which data can be removed from cloud storage. The instructions provided here will help you quickly determine which data is generating the highest storage costs.

Monitoring Storage Cost

To see how much storage costs are currently being generated for your tenant, you can look at the usage explorer at https://platform.illumina.com/usage/ or from within ICA, navigate to the 9-dot symbol () in the top right next to your name and choose the usage explorer from the menu.

From the usage explorer overview screen, you can see below the graphical representation which projects are incurring the highest storage costs.\

Project Files in ICA

When you have determined which projects are incurring the largest storage costs, you can find out which files within that project are taking up the most space. To find the largest files in your project,

1. Go to Projects > your_project > Data and switch to list view with the () icon left above your files.

2. Use the column icon () top right to add the size column to your view. You can drag the size column to the desired position in your list view or use the move left and use right options which appear when you select the three vertical dots.

3. Select Sort descending to show the largest files first.

1. Once you have the list sorted like this, you can evaluate if those large files are still needed, if the can be sent to archive (manage > archive) or if they can be deleted (manage > delete).

Introduction to Cohorts

ICA Cohorts is a cohort analysis tool integrated with Illumina Connected Analytics (ICA). ICA Cohorts combines subject- and sample-level metadata, such as phenotypes, diseases, demographics, and biometrics, with molecular data stored in ICA to perform tertiary analyses on selected subsets of individuals.

Overview Video

This video is an overview of Illumina Connnected Analytics. It walks through a Multi-Omics Cancer workflow that can be found here:

Features At-a-glance

• Intuitive UI for selecting subjects and samples to analyze and compare: deep phenotypical and clinical metadata, molecular features including germline, somatic, gene expression.

• Comprehensive, harmonized data model exposed to ICA Base and ICA Bench users for custom analyses.

• Run analyses in ICA Base and ICA Bench and upload final results back into Cohorts for visualization.

• Out-of-the-box statistical analyses including genetic burden tests, GWAS/PheWAS.

• Rich public data sets covering key disease areas to enrich private data analysis.

• Easy-to-use visualizations for gene prioritization and genetic variation inspection.

Functionality

Walk-throughs

Public Data Sets

ICA Cohorts contains a variety of freely available data sets covering different disease areas and sequencing technologies. For a list of currently available data, .

Please see the for all content related to Cloud Analysis Auto-Launch:

The platform provides Connectors to facilitate automation for operations on data (ie, upload, download, linking).

• sync data between your local computer or server and the project's cloud-based data storage.

• link data between individual projects.

Download links for the CLI can be found at the Release History.

sudo cp icav2 /usr/local/bin

sudo chmod a+x /usr/local/bin/icav2

 mkdir "C:\Program Files\Illumina"
 copy icav2.exe "C:\Program Files\Illumina"

Mount projectdata using CLI

icav2 allows project data to be mounted on a local system. This feature is currently available on Linux and Mac systems only. Although not supported, users have successfully used Windows Subsystem for Linux (WSL) on Windows to use icav2 projectdata mount command. Please refer to the WSL documentation for installing WSL.

Prerequisites

• icav2 (>=2.3.0) installed and authenticated in the local system.

• FUSE Driver

  • For MAC refer to macFuse.

  • For other operating systems, refer to OS specific documentation for FUSE driver installation.

• A project created on ICA v2 with data in it. If you don't already have a project, please follow the instructions here to create a project.

Mount projectdata

Identify the project id by running the following command:

% icav2 projects list
ID                                   	NAME                                            	OWNER  
422d5119-708b-4062-b91b-b398a3371eab	demo                                           	b23f3ea6-9a84-3609-bf1d-19f1ea931fa3

Provide the project id under "ID" column above to the mount command to mount the project data for the project.

% icav2 projectdata mount mnt --project-id 422d5119-708b-4062-b91b-b398a3371eab

Check the content of the mount.

% ls mnt
sampleX.final.count.tsv

icav2 utilizes the FUSE driver to mount project data, providing both read and write capabilities. However, there are some limitations on the write capabilities that are enforced by the underlying AWS S3 storage. For more information, please refer to this page.

WARNING Do NOT use the CP -f command to copy or move data to a mounted location. This will result in data loss as data on the destination location will be deleted.

Unmount project data

You can unmount the project data using the 'unmount' command.

% icav2 projectdata unmount
Project with identifier 422d5119-708b-4062-b91b-b398a3371eab was unmounted from mnt.

In ICA Cohorts, metadata describe any subjects and samples imported into the system in terms of attributes, including:

• subject:

  • demographics such as age, sex, ancestry;

  • phenotypes and diseases;

  • biometrics such as body height, body mass index, etc.;

  • pathological classification, tumor stages, etc.;

  • family and patient medical history;

• sample:

  • sample type such as FFPE,

  • tissue type,

  • sequencing technology: whole genome DNA-sequencing, RNAseq, single-cell RNAseq, among others.

You can use these attributes while creating a cohort to define the cases and/or controls that you want to include.

During import, you will be asked to upload a metadata sheet as a tab-delimited (TSV) file. An example sheet is available for download on the Import files page in the ICA Cohorts UI.

A metadata sheet will need to contain at least these four columns per row:

• Subject ID - identifier referring to individuals; use the column header "SubjectID".

• Sample ID - identifier for a sample. Sample IDs need to match the corresponding column header in VCF/GVCFs; each subject can have multiple samples, these need to be specified in individual rows for the same SubjectID; use the column header "SampleID".

• Biological sex - can be "Female (XX)", "Female"; "Male (XY)", "Male"; "X (Turner's)"; "XXY (Klinefelter)"; "XYY"; "XXXY" or "Not provided". Use the column header "DM_Sex" (demographics).

• Sequencing technology - can be "Whole genome sequencing", "Whole exome sequencing", "Targeted sequencing panels", or "RNA-seq"; use the column header "TC" (technology).

A description of all attributes and data types currently supported by ICA Cohorts can be found here: ICA_Cohorts_Supported_Attributes.xlsx

You can download an example of a metadata sheet, which contains some samples from The Cancer Genome Atlas (TCGA) and their publicly available clincal attributes, here: ICA_Cohorts_Example_Metadata.tsv

A list of concepts and diagnoses that cover all public data subjects to easily navigate the new concept code browser for diagnosis can be found here: PublicData_AllConditionsSummarized.xlsx

Running Jobs in a Bench SGE Cluster

Once a cluster is started, the cluster manager can be accessed from the workspace node.

Job resources

Every cluster member has a certain capacity which is determined by the selected Resource model for the cluster member.

The following complex values have been added to the SGE cluster environment and are requestable.

• static_cores (default: 1)

• static_mem (default: 2G)

These values are used to avoid oversubscription of a node which can result in Out-Of-Memory or unresponsiveness. You need to ensure these limits are not exceeded.

To ensure stability of the system, some headroom is deducted from the total node capacity.

Scaling

These two values are used by the SGE auto scaler when running in dynamic mode. The SGE auto scaler will summarise all pending jobs and their requested resources to determine the scale up/down operation within the defined range.

Cluster members will remain in the cluster for at least 300 seconds. The Auto scaler only executes one scale up/down operation at a time and is stabilised before taking on a new operation.

The operation of the auto scaler can be monitored in the log file /data/logs/sge-scaler.log

Submitting jobs

Submitting a single job

qsub -l static_mem=1G -l static_cores=1 /data/myscript.sh

Submitting a job array

qsub -l static_mem=1G -l static_cores=1 -t 1-100 /data/myscript.sh

Monitoring members

Listing all members of the cluster

qhost

Managing running/pending jobs

listing all jobs in the cluster

qstat -f

Showing the details of a job.

qstat -f -j <jobId>

Deleting a job.

qdel <jobId>

Managing executed jobs

Showing the details of an executed job.

qacct -j <jobId>

SGE Reference documentation

SGE command line options and configuration details can be found here.

The GWAS and PheWAS tabs in ICA Cohorts allow you to visualize precomputed analysis results for phenotypes/diseases and genes, respectively. Note that these do not reflect the subjects that are part of the cohort that you created.

ICA Cohorts currently hosts GWAS and PheWAS analysis results for approximately 150 quantitative phenotypes (such as "LDL direct" and "sitting height") and about 700 diseases.

Visualize Results from Precomputed Genome-Wide Association Studies (GWAS)

Navigate to the GWAS tab and start looking for phenotypes and diseases in the search box. Cohorts will suggest the best matches against any partial input ("cancer") you provide. After selecting a phenotype/disease, Cohorts will render a Manhattan plot, by default collapsed to gene level and organized by their respective position in each chromosome.

Circles in the Manhattan plot indicate binary traits, potential associations between genes and diseases. Triangles indicate quantitative phenotypes with regression Beta different from zero, and point up or down to depict positive or negative correlation, respectively.

Hovering over a circle/triangle will display the following information:

• gene symbol

• variant group (see below)

• P-value, both raw and FDR-corrected

• number of carriers of variants of the given type

• number of carriers of variants of any type

• regression Beta

For gene-level results, Cohorts distinguishes five different classes of variants: protein truncating; deleterious; missense; missense with a high ILMN PrimateAI score (indicating likely damaging variants); and synonymous variants. You can limit results to either of these five classes, or select All to display all results together.

• Deleterious variants (del): the union of all protein-truncating variants (PTVs, defined below), pathogenic missense variants with a PrimateAI score greater than a gene-specific threshold, and variants with a SpliceAI score greater than 0.2.

• Protein-truncating variants (ptv): variant consequences matching any of stop_gained, stop_lost, frameshift_variant, splice_donor_variant, splice_acceptor_variant, start_lost, transcript_ablation, transcript_truncation, exon_loss_variant, gene_fusion, or bidirectional_gene_fusion.

• missense_all: all missense variants regardless of their pathogenicity.

• missense, PrimateAI optimized (missense_pAI_optimized): only pathogenic missense variants with primateAI score greater than a gene-specific threshold.

• missenses and PTVs (missenses_and_ptvs_all): the union of all PTVs, SpliceAI > 0.2 variants and all missense variants regardless of their pathogenicity scores.

• all synonymous variants (syn).

To zoom in to a particular chromosome, click the chromosome name underneath the plot, or select the chromosome from the drop down box, which defaults to Whole genome.

Visualize Results from Precomputed Phenome-Wide Association Studies (PheWAS)

To browse PheWAS analysis results by gene, navigate to the PheWAS tab and enter a gene of interest into the search box. The resulting Manhattan plot will show phenotypes and diseases organized into a number of categories, such as "Diseases of the nervous system" and "Neoplasms". Click on the name of a category, shown underneath the plot, to display only those phenotypes/diseases, or select a category from the drop down, which defaults to All.

Bench Workspaces use a FUSE driver to mount project data directly into a workspace file system. There are both read and write capabilities with some limitations on write capabilities that are enforced by the underlying AWS S3 storage.

As a user, you are allowed to do the following actions from Bench (when having the correct user permissions compared to the workspace permissions) or through the CLI:

• Copy project data

• Delete project data

• Mount project data (CLI only)

• Unmount project data (CLI only)

When you have a running workspace, you will find a file system in Bench under the project folder along with the basic and advanced tutorials. When opening that folder, you will see all the data that resides in your project.

Copy project data

The FUSE driver allows the user to easily copy data from /data/project to the local workspace and vice versa. There is a file size limit of 500 GB per file for the FUSE driver.

Delete project data

The FUSE driver also allows you to delete data from your project. This is different from the use of Bench before where you took a local copy and still kept the original file in your project.

CLI

Using the FUSE driver through the CLI is not supported for Windows users. Linux users will be able to use the CLI without any further actions, Mac users will need to install the kernel extension from macFuse.

Mount and unmount of data needs to be done through the CLI. In Bench this happens automatically and is not needed anymore.

Restrictions

Trying to update files or saving you notebook in the project folder will typically result in File Save Error for fusedrivererror.ipynb Invalid response: 500 Internal Server Error.

Some examples of other actions or commands that will not work because of the above mentioned limitations:

• Save a jupyter notebook or R script on the /project location

• Add/remove a file from an existing zip file

• Redirect with append to an existing file e.g. echo "This will not work" >> myTextFile.txt

• Rename a file due to the existing association between ICA and AWS

• Move files or folders.

• Using vi or another editor

A file can be written only sequentially. This is a restriction that comes from the library the FUSE driver uses to store data in AWS. That library supports only sequential writing, random writes are currently not supported. The FUSE driver will detect random writes and the write will fail with an IO error return code. Zip will not work since zip writes a table of contents at the end of the file. Please use gzip.

Listing data (ls -l) reads data from the platform. The actual data comes from AWS and there can be a short delay between the writing of the data and the listing being up to date. As a result, a file that is written may appear temporarily as a zero length file, a file that is deleted may appear in the file list. This is a tradeoff, the FUSE driver caches some information for a limited time and during that time the information may seem wrong. Note that besides the FUSE driver, the library used by the FUSE driver to implement the raw FUSE protocol and the OS kernel itself may also do caching.

Jupyter notebooks

To use a specific file in a jupyter notebook, you will need to use '/data/project/filename'.

Old Bench workspaces

This functionality won't work for old workspaces unless you enable the permissions for that old workspace.

ICA provides a tool called Bench for interactive data analysis. This is a sandboxed workspace which runs a docker image with access to the data and pipelines within a project. This workspace runs on the Amazon S3 system and comes with associated processing and provisioning costs. It is therefore best practice to not keep your Bench instances running indefinitely, but stopping them when not in use.

Access

Having access to Bench depends on the following conditions:

• Bench needs to be included in your ICA subscription.

• The project owner needs to enable Bench for their project.

• Individual users of that project need to be given access to Bench.

Enabling Bench for your project

After creating a project, go to Projects > your_project > Bench > Workspaces page and click the Enable button. The entitlements you have determine the available resources for your Bench workspaces. If you have multiple entitlements, all the resources of your individual entitlements are taken into account. Once bench is enabled, users with matching permissions have access to the Bench module in that project.

Setting user level access.

Once Bench has been enabled for your project, the combination of roles and teams settings determines if a user can access Bench.

• Tenant administrators and project owners are always able to access Bench and perform all actions.

• The teams settings page at Projects > your_project > Project Settings > Team determines the role for the user/workgroup.

  • No Access means you have no access to the Bench workspace for that project.

  • Contributor gives you the right to start and stop the Bench workspace and to access the workspace contents, but not to create or edit the workspace.

  • Administrator gives you the right to create, edit, delete, start and stop the Bench workspace, and to access the actual workspace contents. In addition, the administrator can also build new derived Bench images and tools.

• Finally, a verification is done of your user rights against the required workspace permissions. You will only have access when your user rights meet or exceed the required workspace permissions. The possible required Workspace permissions include:

  • Upload / Download rights (Download rights are mandatory for technical reasons)

  • Project Level (No Access / Data Provider / Viewer / Contributor)

  • Flow (No Access / Viewer / Contributor)

  • Base (No Access / Viewer / Contributor)

ICA supports running pipelines defined using .

Compute Type

To specify a compute type for a CWL CommandLineTool, use the ResourceRequirement with a custom namespace.

Reference for available compute types and sizes.

The ICA Compute Type will be determined automatically based on coresMin/coresMax (CPU) and ramMin/ramMax (Memory) values using a "best fit" strategy to meet the minimum specified requirements (refer to the table).

For example, take the following ResourceRequirements:

This would result in a best fit of standard-large ICA Compute Type request for the tool.

• If the specified requirements can not be met by any of the presets, the task will be rejected and failed.

• FPGA requirements can not be set by means of CWL ResourceRequirements.

• The Machine Profile Resource in the graphical editor will override whatever is set for requirements in the ResourceRequirement.

Considerations

If no Docker image is specified, Ubuntu will be used as default. Both : and / can be used as separator.

CWL Overrides

ICA supports overriding workflow requirements at load time using Command Line Interface (CLI) with JSON input. Please refer to for more details on the CWL overrides feature.

In ICA you can provide the "override" recipes as a part of the input JSON. The following example uses CWL overrides to change the environment variable requirement at load time.

You can verify the integrity of the data by comparing the hash which is usually () an MD5 (Message Digest Algorithm 5) checksum. This is a common cryptographic hash function that generates a fixed-size, 128-bit hash value from any input data. This hash value is unique to the content of the data, meaning even a slight change in the data will result in a significantly different MD5 checksum. AWS S3 calculates this checksum when data is uploaded and stores it in the ETag (Entity tag).

For files smaller than 16 MB, you can directly retrieve the MD5 checksum using our endpoints. Make an API GET call to the https://ica.illumina.com/ica/rest/api/projects/{projectId}/data/{dataId} endpoint specifying the data Id you want to check and the corresponding project ID. The response you receive will be in JSON format, containing various file metadata. Within the JSON response, look for the objectETag field. This value is the MD5 checksum for the file you have queried. You can compare this checksum with the one you compute locally to ensure file integrity.

This ETag does not change and can be used as a file integrity check even when that file is archived, unarchived and/or copied to another location. Changes to the metadata have no impact on the ETag

For larger files, the process is different due to computation limitations. In these cases, we recommend using a dedicated pipeline on our platform to explicitly calculate the MD5 checksum. Below you can find both a main.nf file and the corresponding XML for a possible Nextflow pipeline to calculate the MD5 checksum for FASTQ files.

You can use samples to group information related to a sample, including input files, output files, and analyses. You can consider samples as creating a binder to collect related information. When you then link that sample to another project, you bring over the empty binder, but not the files contained in it. In that project, you can then add your own data to it.

You can search for samples (excluding their metadata) with the Search button at the top right.

Add New Sample

To add a new sample, do as follows.

1. Select Projects > your_project > Samples.

2. To add a new sample, select + Create, and then enter a unique name and description for the sample.

3. To add files to the sample, see

Your sample is added to the Samples page. To view information on the sample, select the sample name in the overview.

Add Files to Samples

You can add files to a sample after creating the sample. Any files that are not currently linked to a sample are listed on the Unlinked Files tab.

To add an unlinked file to a sample, do as follows.

1. Go to Projects > your_project > Samples > Unlinked files tab.

2. Select a file or files, and then select one of the following options:

   • Create sample — Create a new sample that includes the selected files.

   • Link to sample — Select an existing sample in the project to which you link the file.

Alternatively, you can add unlinked files from the sample details.

1. Going to Projects > your_project > Samples > your_sample.

2. Select your sample to open the details.

3. Go to the Data tab and select link.

4. If the data is not in your project, you will need to add it in Projects > your_project > Data

Data can only be linked to a single sample, so once you have linked data to a sample, it will no longer appear in the list of data to choose form.

Unlink Files from Samples

To remove files from samples,

1. Go to Projects > your_project > Samples > your_sample > Data

2. Select the files you want to remove.

3. Select Unlink.

Link Samples to Project

A Sample can be linked to a project from a separate project to make it available in read-only capacity.

1. Navigate to the Samples view in the Project

2. Click the Link button

3. Select the Sample(s) to link to the project

4. Click the Link button

Delete Samples

If you want to remove a sample, select it and use the delete option from the top navigation row. You will be presented a choice of how to handle the data in the sample.

• Unlink all data without deleting it.

• Delete input data and unlink other data.

• Delete all data.

Running a pyspark application

The JupyterLab environment is by default configured with 3 additional kernels

• PySpark –

• PySpark –

• PySpark –

When one of the above kernels is selected, the spark context is automatically initialised and can be accessed using the sc object.

PySpark - Local

The PySpark - Local runtime environment launches the spark driver locally on the workspace node and all spark executors are created locally on the same node. It does not require a spark cluster to run and can be used for running smaller spark applications which don’t exceed the capacity of a single node.

The spark configuration can be found at /data/.spark/local/conf/spark-defaults.conf.

PySpark - Remote

The PySpark – Remote runtime environment launches the spark driver locally on the workspace node and interacts with the Manager for scheduling tasks onto executors created across the Bench Cluster.

This configuration will not dynamically spin up executors, hence it will not trigger the cluster to auto scale when using a Dynamic Bench cluster.

The spark configuration can be found at /data/.spark/remote/conf/spark-defaults.conf.

PySpark – Remote - Dynamic

The PySpark – Remote - Dynamic runtime environment launches the spark driver locally on the workspace node and interacts with the Manager for scheduling tasks onto executors created across the Bench Cluster.

This configuration will increase/decrease the required executors which will result into a cluster that auto scales using a Dynamic Bench cluster

The spark configuration can be found at /data/.spark/remote/conf-dynamic/spark-defaults.conf.

Job resources

Every cluster member has a certain capacity depending on the selection of the model for the member.

A spark application consists of 1 or more jobs. Each Job consists out of one or more stages. Each stage consists out of one or more tasks. Task are handled by executors and executors are run on a worker (cluster member).

The following setting define the amount of cpus needed per task

The following settings define the size of a single executor which handles the execution of a task

The above example allows an executor to handle 4 tasks concurrently and share a total capacity of 4Gb of memory. Depending on the resource model chosen (e.g. standard-2xlarge) a single cluster member (worker node) is able to run multiple executors concurrently (e.g. 32 cores, 128 Gb for 8 concurrent executors on a single cluster member)

Spark User Interface

The Spark UI can be accessed via the Cluster. The Web Access URL is displayed in the Workspace details page

This Spark UI will register all applications submitted when using one of the Remote Jupyter kernels. It will provide an overview of the registered workers (Cluster members) and the applications running in the Spark cluster.

Spark Reference documentation

See the website

Bench workspaces require setting a Docker image to use as the image for the workspace. Illumina Connected Analytics (ICA) provides a default Docker image with installed.

JupyterLab supports (.ipynb). Notebook documents consist of a sequence of cells which may contain executable code, markdown, headers, and raw text.

The JupyterLab Docker image contains the following environment variables:

ICA Python Library

Included in the default JupyterLab Docker image is a python library with APIs to perform actions in ICA, such as add data, launch pipelines, and operate on Base tables. The python library is generated from the using .

The ICA Python library API documentation can be found in folder /etc/ica/data/ica_v2_api_docs within the JupyterLab Docker image.

See the for examples on using the ICA Python library.

The ICA CLI uses an Illumina API Key to authenticate. An Illumina API Key can be acquired through the product dashboard after logging into a domain. See for instructions to create an Illumina API Key.

Authenticate using icav2 config set command. The CLI will prompt for an x-api-key value. Input the API Key generated from the product dashboard here. See the example below (replace EXAMPLE_API_KEY with the actual API Key).

The CLI will save the API Key to the config file as an encrypted value.

If you want to overwrite existing environment values, use the command icav2 config set. To remove an existing configuration/session file, use the command icav2 config reset.\

Check the server and confirm you are authenticated using icav2 config get

If during these steps or in the future you need to reset the authentication, you can do so using the command: icav2 config reset

Reference Data are reference genome sets which you use to help look for deviations and to compare your data against.

Creating Reference Data

Reference data properties are located at the main navigation level and consist of the following free text fields.

• Types

• Species

• Reference Sets

Once these are configured,

1. Go to your data in Projects > your_project > Data.

2. Select the data you want to use as reference data and Manage > Use as reference data.

3. Fill out the configuration screen

You can see the result at the main navigation level > Reference Data (outside of projects) or in Projects > your_project > Flow > Reference Data.

Linking Reference Data to your Project

To use a reference set from within a project, you have first to add it. Select Projects > your_project > Flow > Reference Data > Link. Then select a reference set to add to your project.

Copying Reference Data to other Regions

1. Navigate to Reference Data (Not from within a project, but outside of project context, so at the main navigation level).

2. Select the data set(s) you wish to add to another region and select Copy to another project.

3. Select a project located in the region where you want to add your reference data.

4. You can check in which region(s) Reference data is present by opening the Reference set and viewing Copy Details.

5. Allow a few minutes for new copies to become available before use.

Creating a Pipeline with Reference Data

To create a pipeline with a reference data, use the mode. Projects > your_project > Flow > Pipelines > +Create > CWL Graphical. Use the reference data icon instead of regular input icon. On the right hand side use the Reference files submenu to specify the name, the format, and the filters. You can specify the options for an end-user to choose from and a default selection. You can select more than 1 file, but you can only select 1 at a time (so, repeat process to select multiple reference files). If you only select 1 reference file, that file will be the only one users can use with your pipeline. In the screenshot a reference data with two options is presented.

If your pipeline was built to give users the option of choosing among multiple input reference files, they will see the option to select among the reference files you configured, under Settings. After clicking the magnifying glass icon the user can select from provided options.

The Activity view shows the status and history of long-running activities including Data Transfers, Base Jobs, Base Activity, Bench Activity and Batch Jobs.

Data Transfers

The Data Transfers tab shows the status of data uploads and downloads. You can sort, search and filter on various criteria and export the information. Show ongoing transfers (top right) allows you to filter out the completed and failed transfers to focus on current activity.

Transfers with a yellow background indicate that service connector rules have been modified in ways that prevent planned files from being uploaded. Please verify your service connectors to resolve this.

Base Jobs

The Base Jobs tab gives an overview of all the actions related to a table or a query that have run or are running (e.g., Copy table, export table, Select * from table, etc.)

The jobs are shown with their:

• Creation time: When did the job start

• Description: The query or the performed action with some extra information

• Type: Which action was taken

• Status: Failed or succeeded

• Duration: How long the job took

• Billed bytes: The used bytes that need to be paid for

Failed jobs provide information on why the job failed. Details are accessed by double-clicking the failed job. Jobs in progress can be aborted here.

Base Activity

The Base Activity tab gives an overview of previous results (e.g., Executed query, Succeeded Exporting table, Created table, etc.) Collecting this information can take considerable time. For performance reasons, only the activity of the last month (rolling window) with a limit of 1000 records is shown and available for download as Excel or JSON. To get the data for the last year without limit on the number of records, use the export as file function. No activity data is retained for more than one year.

The activities are shown with:

• Start Time: The moment the action was started

• Query: The SQL expression.

• Status: Failed or succeeded

• Duration: How long the job took

• User: The user that requested the action

• Size: For SELECT queries, the size of the query results is shown. Queries resulting in less than 100Kb of data will be shown with a size of <100K

Bench Activity

The Bench Activity tab shows the actions taken on Bench Workspaces in the project.

The activities are shown with:

• Workspace: Workspace where the activity took place

• Date: Date and time of the activity

• User: User who performed the activity

• Action: Which activity was performed

Batch Jobs

The Batch Jobs tab allows users to monitor progress of Batch Jobs in the project. It lists Data Downloads, Sample Creation (double-click entries for details) and Data Linking (double-click entries for details). The (ongoing) Batch Job details are updated each time they are (re)opened, or when the refresh button is selected at the bottom of the details screen. Batch jobs which have a final state such as Failed or Succeeded are removed from the activity list after 7 days.

Which batch jobs are visible depends on the user role.

Cohorts Walk-through: Rare Genetic Disorders

This walk-through is meant to represent a typical workflow when building and studying a cohort of rare genetic disorder cases.

Login and Create a new ICA Project

Create a new Project to track your study:

1. Login to the ICA

2. Navigate to Projects

3. Create a new project using the New Project button.

4. Give your project a name and click Save.

5. Navigate to the ICA Cohorts module by clicking COHORTS in the left navigation panel then choose Cohorts.

Create and Review a Rare Disease Cohort

1. Navigate to the ICA Cohorts module by clicking Cohorts in the left navigation panel.

2. Click Create Cohort button.

3. Enter a name for your cohort, like Rare Disease + 1kGP at top, left of pencil icon.

4. From the Public Data Sets list select:

   1. DRAGEN-1kGP

   2. All Rare genetic disease cohorts

5. Notice that a cohort can also be created based on Technology, Disease Type and Tissue.

6. Under Selected Conditions in right panel, click on Apply

7. A new page opens with your cohort in a top-level tab.

8. Expand Query Details to see the study makeup of your cohort.

9. A set of 4 Charts will be open by default. If they are not, click Show Charts.

   1. Use the gear icon in the top-right of the Charts pane to change chart settings.

10. The bottom section is demarcated by 8 tabs (Subjects, Marker Frequency, Genes, GWAS, PheWAS, Correlation, Molecular Breakdown, CNV).

11. The Subjects tab displays a list of exportable Subject IDs and attributes.

    1. Clicking on a Subject ID link pops up a Subject details page.

Analyze Your Rare Disease Cohort Data

1. A recent GWAS publication identified 10 risk genes for intellectual disability (ID) and autism. Our task is to evaluate them in ICA Cohorts: TTN, PKHD1, ANKRD11, ARID1B, ASXL3, SCN2A, FHL1, KMT2A, DDX3X, SYNGAP1.

2. First let’s Hide charts for more visual space.

3. Click the Genes tab where you need to query a gene to see and interact with results.

4. Type SCN2A into the Gene search field and select it from autocomplete dropdown options.

5. The Gene Summary tab now lists information and links to public resources about SCN2A.

6. Click on the Variants tab to see an interactive Legend and analysis tracks.

   1. The Needle Plot displays gnomAD Allele Frequency for variants in your cohort.

      1. Note that some are in SCN2A conserved protein domains.

   2. In Legend, switch the Plot by option to Sample Count in your cohort.

   3. In Legend, uncheck all Variant Types except Stop gained. Now you should see 7 variants.

   4. Hover over pin heads to see pop-up information about particular variants.

7. The Primate AI track displays Scores for potential missense variants, based on polymorphisms observed in primate species. Points above the dashed line for the 75th percentile may be considered "likely pathogenic" as cross-species sequence is highly conserved; you often see high conservancy at the functional domains. Points below the 25th percentile may be considered "likely benign".

8. The Pathogenic variants track displays markers from ClinVar color-coded by variant type. Hover over to see pop-ups with more information.

9. The Exons track shows mRNA exon boundaries with click and zoom functionality at the ends.

10. Below the Needle Plot and analysis tracks is a list of "Variants observed in the selected cohort"

    1. Export Gene Variants table icon is above the legend on right side.

11. Now let's click on the Gene Expression tab to see a Bar chart of 50 normal tissues from GTEx in transcripts per million (TPM). SCN2A is highly expressed in certain Brain tissues, indicating specificity to where good markers for intellectual disability and autism could be expected.

12. As a final exercise in discovering good markers, click on the tab for Genetic Burden Test. The table here associates Phenotypes with Mutations Observed in each Study selected for our cohort, alongside Mutations Expected to derive p-values. Given all considerations above, SCN2A is good marker for intellectual disability (p < 1.465 x 10 -22) and autism (p < 5.290 x 10 -9).

13. Continue to check the other genes of interest in step 1.

You can compare up to four previously created individual cohorts, to view differences in variants and mutations, RNA expression, copy number variation, and distribution of clinical attributes. Once comparisons are created, they are saved in the Comparisons left-navigation tab of the Cohorts module.

Create a comparison view

1. Select Cohorts from the left-navigation panel.

2. Select 2 to 4 cohorts already created. If you have not created any cohorts, See Create a Cohort documentation.

3. Click Compare Cohorts in the right-navigation panel.

4. Note you are now in the Comparisons left-navigation tab of the Cohorts module.

5. In the Charts Section, if the COHORTS item is not displayed, click the gear icon in the top right and add Cohorts as the first attribute and click Save.

6. The COHORTS item in the charts panel will provide a count of subjects in each cohort and act as a legend for color representation throughout comparison screens.

7. For each clinical attribute category, a bar chart is displayed. Use the gear icon to select attributes to display in the charts panel.

You can share a comparison with other team members in the same ICA Project. Please refer to the section on "Sharing a Cohort" on "Create a Cohort" for details on sharing, unsharing, deleting, and archiving, which are analogous for sharing comparisons.

Attribute Comparison

1. Select the Attributes tab

2. Attribute categories are listed and can be expanded using the down-arrows next to the category names. The categories available are based on cohorts selected. Categories and attributes are part of the ICA Cohorts metadata template that map to each Subject.

3. For example, use the drop-down arrow next to Vital status to view sub-categories and frequencies across selected cohorts.

Variants Comparison

1. Select the Genes tab

2. Search for a gene of interest using its HUGO/HGNC gene symbol

3. Variants and mutations will be displayed as one needle plot for each cohort that is part of the comparison (see Cohort analysis -> Genes in this online help for more details)

4. As additional filter options, you can view only those variants that are occur in every cohort; that are unique to one cohort; that have been observed in at least two cohorts; or any variant.

Survival Summary

1. Select the Survival Summary tab.

2. Attribute categories are listed and can be expanded using the down-arrows next to the category names.

3. Select the drop-down arrow for Therapeutic interventions.

4. In each subcategory there is a sum of the subject counts across select cohorts.

5. For each cohort, designated by a color, there is a Subject count and Median survival (years) column.

6. Type Malignancy in the Search Box and an auto-complete dropdown suggests three different attributes.

7. Select Synchronous malignancy and the results are automatically opened and highlighted in orange.

Survival Comparison

1. Click Survival Comparison tab.

2. A Kaplan-Meier Curve is rendered based on each cohort.

3. P-Value Displayed at the top of Survival Comparison indicates whether there is statistically significant variance between survival probabilities over time of any pair of cohorts (CI=0.95).

When comparing two cohorts, the P-Value is shown above the two survival curves. For three or four cohorts, P-Values are shown as a pair-wise heatmap, comparing each cohort to every other cohort.

Marker Frequency Comparison

1. Select the Marker Frequency tab.

2. Select either Gene expression (default), Somatic mutation, or Copy number variation

3. For gene expression (up- versus down-regulated) and for copy number variation (gain versus loss), Cohorts will display a list of all genes with bidirectional barcharts

4. For somatic mutations, the barcharts are unidirectional and indicate the percentage of samples with a mutation in each gene per cohort.

5. Bars are color-coded by cohort, see the accompanying legend.

6. Each row shows P-value(s) resulting from pairwise comparison of all cohorts. In the case of comparing two cohorts, the numerical P-value will be displayed in the table. In the case of comparing three or more cohorts, the pairwise P-values are shown as a triangular heatmap, with details available as a tooltip.

Correlation Comparison

1. Select the Correlation tab.

2. Similar to the single-cohort view (Cohort Analysis | Correlation), choose two clinical attributes and/or genes to compare.

3. Depending on the available data types for the two selections (categorical and/or continuous), Cohorts will display a bubble plot, violin plot, or scatter plot.

The ICA CLI is a useful tool for uploading, downloading and viewing information about data stored within ICA projects. If not already authenticated, please see the Authentication section of the CLI help pages. Once the CLI has been authenticated with your account, use the command below to list all projects:

icav2 projects list

The first column of the output (table format, which is default) will show the ID. This is the project ID and will be used in the examples below.

Upload Data

To upload a file called Sample-1_S1_L001_R1_001.fastq.gz to the project, copy the project id and use the command syntax below:

icav2 projectdata upload Sample-1_S1_L001_R1_001.fastq.gz --project-id <project-id>

To verify the file has uploaded, run the following to get a list of all files stored within the specified project:

icav2 projectdata list --project-id <project-id>

This will show a file ID starting with fil. which can then be used to get more information about the file and its attributes:

icav2 projectdata get <file-id> --project-id <project-id>

It is necessary to use --project-id in the above example if not entered into a specific project context. In order to enter a project context use the command below.

icav2 projects enter <project-name or project-id>

This will infer the project id, so that it does not need to be entered into each command.

Download Data

The ICA CLI can also be used to download files via command line. This can be especially helpful if the download destination is a remote server or HPC cluster that you are logged into from a local machine. To download into the current folder, run the following from the command line terminal:

icav2 projectdata download <file-id> ./

The above assumes you have entered into a project context. If this is not the case, either enter the project that contains the desired data, or be sure to supply the --project-id option in the command.

Temporary Credentials

To fetch temporary AWS credentials for given project data, use the command icav2 projectdata temporarycredentials [path or data Id] [flags]. If the path is provided, the project id from the flag --project-id is used. If the --project-id flag is not present, then the project id is taken from the context. The returned AWS credentials for file or folder upload expire after 36 hours.

Data Transfer Options

For information on options such as using the ICA API and AWS CLI to transfer data, visit the Data Transfer Options tutorial.

The ICA CLI accepts configuration settings from multiple places, such as environment variables, configuration file, or passed in as command line arguments. When configuration settings are retrieved, the following precedence is used to determine which setting to apply:

1. Command line options - Passed in with the command such as --access-token

2. Environment variables - Stored in system environment variables such as ICAV2_ACCESS_TOKEN

3. Default config file - Stored by default in the ~/.icav2/config.yaml on macOS/Linux and C:\Users\USERNAME\.icav2\.config on Windows

Command Line Options

The following global flags are available in the CLI interface:

-t, --access-token string    JWT used to call rest service
-h, --help                   help for icav2
-o, --output-format string   output format (default "table")
-s, --server-url string      server url to direct commands
-k, --x-api-key string       api key used to call rest service

Environment Variables

Environment variables provide another way to specify configuration settings. Variable names align with the command line options with the following modifications:

• Upper cased

• Prefix ICAV2_

• All dashes replaced by underscore

For example, the corresponding environment variable name for the --access-token flag is ICAV2_ACCESS_TOKEN.

Disable Retry Mechanism

The environment variable ICAV2_ICA_NO_RETRY_RATE_LIMITING allows to disable the retry mechanism. When it is set to 1, no retries are performed. For any other value, http code 429 will result in 4 retry attempts:

• after 500 milliseconds

• after 2 seconds

• after 10 seconds

• after 30 seconds

Config File

Upon launching icav2 for the first time, the configuration yaml file is created and the default config settings are set. Enter an alternative server URL or press enter to leave it as the default. Then enter your API Key and press enter.

After installing the CLI, open a terminal window and enter the icav2 command. This will initialize a default configuration file in the home folder at .icav2/config.yaml.

To reset the configuration, use ./icav2 config reset

Resetting the configuration removes the configuration from the host device and cannot be undone. The configuration needs to be recreated.

Configuration settings is stored in the default configuration file:

access-token: ""
colormode: none
output-format: table
server-url: ica.illumina.com
x-api-key: !!binary SMWV6dEXAMPLE

The file ~/.icav2/.session.ica.yamlon macOS/Linux and C:\Users\USERNAME\.icav2\.session.ica on Windows will contain the access-token and project-id. These are output files and should not be edited as they are automatically updated.

Examples

ICAV2_X_API_KEY

This variable is used to set the API Key.

1. Command line options - Passed as --x-api-key <your_api_key> or -k <your_api_key>

2. Environment variables - Stored in system as ICAV2_X_API_KEY

3. Default config file - Use icav2 config set to update ~/.icav2/config.yaml(macOS/Linux) or C:\Users\USERNAME\.icav2\.config (Windows)

Software Registration

If you are a new user, please consult the Illumina Connected Software Registration Guide for detailed guidance on setting up an account and registering a subscription.

Tenant Setup

The platform requires a provisioned tenant in the Illumina account management (IAM) system with access to the Illumina Connected Analytics (ICA) application. Once a tenant has been provisioned, a tenant administrator will be assigned. The tenant administrator has permission to manage account access including add users, create workgroups, and add additional tenant administrators.

Each tenant is assigned a domain name used to login to the platform. The domain name is used in the login URL to navigate to the appropriate login page in a web browser. The login URL is https://<domain_name>.login.illumina.com with <domain_name> replaced by the actual domain name.

New user accounts can be created

For more details on identity and access management, please see the Illumina Connected Software help.

• by the tenant administrator by logging in to their domain and navigating to Illumina Account Management under their profile at the top right

• or by the user by accessing https://platform.login.illumina.com and selecting the option Don't have an account.

Once the account has been added to the domain, the tenant administrator may assign registered users to workgroups which bundle users with permission to use the ICA application. Registered users can be made workgroup administrators by tenant administrators or existing workgroup administrators.

API Keys

If you want to use the command-line interface (CLI) or the Application Programming Interface (API), you can use an API Key as credentials when logging in. API Keys operate similar to a user name and password combination and must be kept secure and rotated on a regular basis (preferably yearly). `

When keys are compromised or no longer in use, they must be revoked. This is done through the domain login URL by navigating to the User menu item on the left and selecting "API Keys", followed by selecting the key and using the trash icon next to it.

Generate an API Key

API Keys are limited to 10 per user and are managed through the product dashboard after logging in through the domain login URL. See Managing API Keys for more information.

{% hint style="warning" %} For security reasons, do not use accounts with administrator level access to generate API keys. Create a specific CLI user with basic permissions instead. This will minimize the possible impact of compromised keys. {% endhint %}

{% hint style="warning" %} Once the API key generation window is closed, the key contents will not be accessible through the domain login page, so be sure to store it securely for future reference. {% endhint %}

Access via Web UI

The web application provides a visual user interface (UI) for navigating resources in the platform, managing projects, and extended features beyond the API. To access the web application, navigate to the Illumina Connected Analytics portal.

• On the left, you have the navigation bar (1) which will auto-collapse on smaller screens. To collapse it, use the double arrow symbol (2). When collapsed, use the >> symbol to expand it.

• The central part (3) of the display is the item on which you are performing your actions and the breadcrumb menu (4) to return to the projects overview or a previous level. You can also use your browser's back button to return to the level from which you came.

• At the top right, you have icons to refresh contents (5) Illumina product access (6), access to the online help (7) and user information (8), .

Access via the CLI

The command-line interface offers a developer-oriented experience for interacting with the APIs to manage resources and launch analysis workflows. Find instructions for using the command-line interface including download links for your operating system in the CLI documentation.

Access via the API

The HTTP-based application programming interfaces (APIs) are listed in the API Reference section of the documentation. The reference documentation provides the ability to call APIs from the browser page and shows detailed information about the API schemas. HTTP client tooling such as Postman or cURL can be used to make direct calls to the API outside of the browser.

{% hint style="info" %} When accessing the API using the API Reference page or through REST client tools, the Authorization header must be provided with the value set to Bearer <token> where <token> is replaced with a valid JSON Web Token (JWT). For generating a JWT, see JSON Web Token (JWT). {% endhint %}

Object Identifiers

The object data models for resources that are created in the platform include a unique id field for identifying the resource. These fixed machine-readable IDs are used for accessing and modifying the resource through the API or CLI, even if the resource name changes.

JSON Web Token (JWT)

Accessing the platform APIs requires authorizing calls using JSON Web Tokens (JWT). A JWT is a standardized trusted claim containing authentication context. This is a primary security mechanism to protect against unauthorized cross-account data access.

A JWT is generated by providing user credentials (API Key or username/password) to the token creation endpoint. Token creation can be performed using the API directly or the CLI.

A storage configuration provides ICA with information to connect to an external cloud storage provider, such as AWS S3. The storage configuration validates that the information provided is correct, and then continuously monitors the integration.

Refer to the following pages for instructions to setup supported external cloud storage providers:

Credentials

The storage configuration requires credentials to connect to your storage. AWS uses the security credentials to authenticate and authorize your requests. On the System Settings > Credentials > Create > Storage Credential, you can enter these credentials. Long-term access keys consist of a combination of the access key ID and secret access key as a set.

Fill out the following fields:

• Type—The type of access credentials. This will usually be AWS user.

• Name—Provide a name to easily identify your access key.

• Access key ID—The access key you created.

• Secret access key—Your related secret access key.

You can share the credentials you own with other users of your tenant. To do so select your credentials at System Settings > Credentials and choose Share.

For more information, refer to the documentation.

Create a Storage Configuration

1. In the ICA main navigation, select System Settings > Storage > Create.

2. Configure the following settings for the storage configuration.

   • Type—Use the default value, eg, AWS_S3. Do not change.

   • Region—Select the region where the bucket is located.

   • Configuration name—You will use this name when creating volumes that reside in the bucket. The name length must be in between 3 and 63 characters.

   • Description—Here you can provide a description for yourself or other users to identify this storage configuration.

   • Bucket name—Enter the name of your S3 bucket.

   • Key prefix —You can provide a key prefix to allow only files inside the prefix to be accessible. Although this setting is optional, it is highly recommended to use a key prefix and mandatory when using . The key prefix must end with "/".

     • If a key prefix is specified, your projects will only have access to that folder and subfolders. For example, using the key prefix folder-1/ ensures that only the data from the folder-1 folder in your S3 bucket is synced with your ICA project. Using prefixes and distinct folders for each ICA project is the recommended configuration as it allows you to use the same S3 bucket for different projects.

     • Using no key prefix (not recommended) results in syncing all data in your S3 bucket (starting from root level) with your ICA project. Your project will have access to your entire S3 bucket, which prevents that S3 bucket from being used for other ICA projects.

   • Secret—Select the credentials to associate with this storage configuration. These were created on the Credentials tab.

   • Server Side Encryption [Optional]—If needed, you can enter the algorithm and key name for server-side encryption processes.

3. Select Save.

ICA performs a series of steps in the background to verify the connection to your bucket. This can take several minutes. You may need to manually refresh the list to verify that the bucket was successfully configured. Once the storage configuration setup is complete, the configuration can be used while .

With the action Manage > Set as default for region, you select which storage will be used as default storage in a region for new projects of your tenant. Only one storage can be default at a time for a region, so selecting a new storage as default will unselect the previous default. If you do not want to have a default, you can select the default storage and the action will become Unset as default for region.

The System Settings > Credentials > select your credentials > Manage > Share action is used to make the storage available to everyone in your tenant. By default, storage is private per user so that you have complete control over the contents. Once you decide you want to share the storage, simply select it and use the Share action. Do take into account that once shared, you can not unshare the storage. Once your shared storage is used in a project, it can also no longer be deleted.

Deleting Storage Configurations

In the ICA main navigation, select System Settings > Storage > select your storage > Manage > Delete. You can then create a new storage configuration to reuse the bucket name and key prefix.

Storage Configuration Verification

Every 4 hours, ICA will verify the storage configuration and credentials to ensure availability. When an error is detected, ICA will attempt to reconnect once every 15 minutes. After 200 consecutively failed connection attempts (50 hours), ICA will stop trying to connect.

When you update your credentials, the storage configuration is automatically validated. In addition, you can manually trigger revalidation when ICA has stopped trying to connect by selecting the storage and then clicking Validate on the System Settings > Storage > select your storage > Manage > Validate.

Refer to this for the troubleshooting guide.

Supported Storage Classes

ICA supports the following storage classes. Please see the for more information on each:

This section describes how to connect an AWS S3 Bucket with enabled. General instructions for configuring your AWS account to allow ICA to connect to an S3 bucket are found on .

Create an S3 bucket with SSE-KMS

Follow the for how to create S3 bucket with SSE-KMS key.

In the "Default encryption" section, enable Server-side encryption and choose AWS Key Management Service key (SSE-KMS). Then select Choose your AWS KMS key.

Connect the S3-SSE-KMS to ICA

Follow the for connecting an S3 bucket to ICA.

In the step :

• Add permission to use KMS key by adding kms:Decrypt, kms:Encrypt, and kms:GenerateDataKey

• Add the ARN KMS key arn:aws:kms:xxx on the first "Resource"

• On Unversioned buckets, the permssions will match the following:

• On Versioned OR Suspended buckets, the permssions will match the following:

At the end of the policy setting, there should be 3 permissions listed in the "Summary".

Create the S3-SSE-KMS configuration in ICA

Follow the for how to create a storage configuration in ICA.

On step 3 in process above, continue with the [Optional] Server Side Encryption to enter the algorithm and key name for server-side encryption processes.

• On "Algorithm", input aws:kms

• On "Key Name", input the ARN KMS key: arn:aws:kms:xxx

Additional set up for Cross Account Copy for S3 buckets with SSE-KMS encryption

In addition to following the instructions to , the KMS policy must include the following statement for AWS S3 Bucket with SSE-KMS Encyption (refer to the Role ARN table from the linked page for the ASSUME_ROLE_ARN value):

Bench has the ability to handle containers inside a running workspace. This allows you to install and package software more easily as a container image and provides capabilities to pull and run containers inside a workspace.

Bench offers a container runtime as a service in your running workspace. This allows you to do standardized container operations such as pulling in images from public and private registries, build containers at runtime from a Dockerfile, run containers and eventually publish your container to a registry of choice to be used in different ICA products such as ICA Flow.

Setup

The Container Service is accessible from your Bench workspace environment by default.

The container service uses the workspace disk to store any container images you pulled in or created.

To interact with the Container Service, a container remote client CLI is exposed automatically in the /data/.local/bin folder. The Bench workspace environment is preconfigured to automatically detect where the Container Service is made available using environment variables. These environment variables are automatically injected into your environment and are not determined by the Bench Workspace Image.

Container Management

Use either docker or podman cli to interact with the Container Service. Both are interchangeable and support all the standardized operations commonly known.

Pulling a Container Image

To run a container, the first step is to either build a container from a source container or pull in a container from a registry

Public Registry

A public image registry does not require any form of authentication to pull the container layers.

The following command line example shows how to pull in a commonly known image.

Private Registry

To pull images from a private registry, the Container Service needs to authenticate to the Private Registry.

The following command line example shows how to instruct the Container Service to login into the Private registry.hub.docker.com registry

Pushing a Container Image

Depending on the Registry setup you can publish Container Images with or without authentication. If Authentication is required, follow the login procedure described in

The following command line example shows how to publish a locally available Container Image to a private registry in Dockerhub.

Saving a Container Image as an Archive

The following example shows how to save a locally available Container Image as a compressed tar archive.

This lets you upload the into the Private ICA Docker Registry.

Listing Locally Available Container Images

The following example shows how to list all locally available Container Images

Deleting a Container Image

Container Images require storage capacity on the Bench Workspace disk. The capacity is shown when listing the locally available container images. The container Images are persisted on disk and remain available whenever a workspace stops and restarts.

The following example shows how to clean up a locally available Container Image

Running a Container

A Container Image can be instantiated in a Container running inside a Bench Workspace.

By default the workspace disk (/data) will be made available inside the running Container. This lets you to access data from the workspace environment.

When running a Container, the default user defined in the Container Image manifest will be used and mapped to the uid and the gid of the user in the running Bench Workspace (uid:1000, gid: 100). This will ensure files created inside the running container on the workspace disk will have the same file ownership permissions.

Run a Container as a normal user

The following command line example shows how to run an instance a locally available Container Image as a normal user

Run a Container as root user

Running a Container as root user maps the uid and gid inside the running Container to the running non-root user in the Bench Workspace. This lets you act as user with uid 0 and gid 0 inside the context of the container.

By enabling this functionality, you can install system level packages inside the context of the Container. This can be leveraged to run tools that require additional system level packages at runtime.

The following command line example shows how to run an instance of a locally available Container as root user and install system level packages

When no specific mapping is defined using the --userns flag, the user in the running Container user will be mapped to an undefined uid and gid based on an offset of id 100000. Files created in your workspace disk from the running Container will also use this uid and gid to define the ownership of the file.

Building a Container

To build a Container Image, you need to describe the instructions in a Dockerfile.

This next example builds a local Container Image and tags it as myimage:1.0 The Dockerfile used in this example is

The following command line example will build the actual Container Image

The platform GUI provides the Project Connector utility which allows data to be linked automatically between projects. This creates a one-way dynamic link for files and samples from source to destination, meaning that additions and deletions of data in the source project also affect the destination project. This differs from or which create editable copies of the data. In the destination project, you can delete data which has been moved or copied and unlink data which has been linked.

Prepare Source Project

1. Select the source project (project that will own the data to be linked) from the Projects page (Projects > your_source_project).

2. Select Project Settings > Details.

3. Select Edit

4. Under Data Sharing ensure the value is set to Yes

5. Select Save

Creating a New Project Connector

1. Select the destination project (the project to which data from the source project will be linked) from the Projects page (Projects > your_destination_project).

2. From the projects menu, select Project Settings > Connectivity > Project Connector

3. Select + Create and complete the necessary fields.

   • Check the box next to Active to ensure the connector will be active.

   • Name (required) — Provide a unique name for the connector.

   • Type (required) — Select the data type that will be linked (either File or Sample)

   • Source Project - Select the source poject whose data will be linked to.

   • Filter Expression (optional) — Enter an expression to restrict which files will be linked via the connector (see below)

   • Tags (optional) — Add tags to restrict what data will be linked via the connector. Any data in the source project with matching tags will be linked to the destination project.

Filter Expression Examples

The examples below will restrict linking Files based on the Format field.

• Only Files with Format of FASTQ will be linked:

  [?($.details.format.code == 'FASTQ')]

• Only Files with Format of VCF will be linked:

  [?($.details.format.code == 'VCF')]

The examples below will restrict linked Files based on a filenames.

• Exact match to 'Sample-1_S1_L001_R1_001.fastq.gz':

  [?($.details.name == 'Sample-1_S1_L001_R1_001.fastq.gz')]

• Ends with '.fastq.gz':

  [?($.details.name =~ /.*\.fastq.gz/)]

• Starts with 'Sample-':

  [?($.details.name =~ /Sample-.*/)]

• Contains '_R1_':

  [?($.details.name =~ /.*_R1_.*/)]

The examples below will restrict linking Samples based on User Tags and Sample name, respectively.

• Only Samples with the User Tag 'WGS-Project-1'

  [?('WGS-Project-1' in $.tags.userTags)]

• Link a Sample with the name 'BSSH_Sample_1':

  [?($.name == 'BSSH_Sample_1')]

You can access the databases and tables within the Base module using Python from your local machine. Once retrieved as e.g. pandas object, the data can be processed further. In this tutorial, we will describe how you could create a Python script which will retrieve the data and visualize it using Dash framework. The script will contain the following parts:

• Importing dependencies and variables.

• Function to fetch the data from Base table.

• Creating and running the Dash app.

Importing dependencies and variables

This part of the code imports the dependencies which have to be installed on your machine (possibly with pip). Furthermore, it imports the variables API_KEY and PROJECT_ID from the file named config.

Function to fetch the data from Base table

We will be creating a function called fetch_data to obtain the data from Base table. It can be broken into several logically separated parts:

• Retrieving the token to access the Base table together with other variables using API.

• Establishing the connection using the token.

• SQL query itself. In this particular example, we are extracting values from two tables Demo_Ingesting_Metrics and BB_PROJECT_PIPELINE_EXECUTIONS_DETAIL. The table Demo_Ingesting_Metrics contains various metrics from DRAGEN analyses (e.g. the number of bases with quality at least 30 Q30_BASES) and metadata in the column ica which needs to be flattened to access the value Execution_reference. Both tables are joined on this Execution_reference value.

• Fetching the data using the connection and the SQL query.

Here is the corresponding snippet:

Creating and running the Dash app

Once the data is fetched, it is visualized in an app. In this particular example, a scatter plot is presented with END_DATE as x axis and the choice of the customer from the dropdown as y axis.

Now we can create a single Python script called dashboard.py by concatenating the snippets and running it. The dashboard will be accessible in the browser on your machine.

Non-indexed folders () are designed for optimal performance in situations where no file actions are needed. They serve as fast storage in situations like temporary analysis file storage where you don't need access or searches via the GUI to individual files or subfolders within the folder. Think of a non-indexed folder as a data container. You can access the container which contains all the data, but you can not access the individual data files within the container from the GUI. As non-indexed folders contain data, they count towards your total project storage.

The GUI considers non-indexed folders as a single object. You can access the contents from a non-indexed folder

• as Analysis input/output

• in Bench

• via the API

Introduction to Base

Base is a genomics data aggregation and knowledge management solution suite. It is a secure and scalable integrated genomics data analysis solution which provides Information management and knowledge mining. You can analyze, aggregate and query data for new insights that can inform and improve diagnostic assay development, clinical trials, patient testing and patient care. Clinically relevant data needs to be generated and extracted from routine clinical testing and clinical questions need to be asked across all data and information sources. As a large data store, Base provides a secure and compliant environment to accumulate data, allowing for efficient exploration of the aggregated data. This data consists of test results, patient data, metadata, reference data, consent and QC data.

Use Cases

Base can be used by for different use cases:

• Clinical and Academic Researchers:

  • Big data storage solution housing all aggregated sample test outcomes

  • Analyze information by way of a convenient query formalism

  • Look for signals in combined phenotypic and genotypic data

  • Analyze QC patterns over large cohorts of patients

  • Securely share (sub)sets of data with other scientists

  • Generate reports and analyze trends in a straightforward and simple manner.

• Bioinformaticians:

  • Access, consult, audit, and query all relevant data and QC information for tests run

  • All accumulated data and accessible pipelines can be used to investigate and improve bioinformatics for clinical analysis

  • Metadata is captured via automatic pipeline version tracking, including information on individual tools and/or reference files used during processing for each sample analyzed, information on the duration of the pipeline, the execution path of the different analytical steps, or in case of failure, exit codes can be warehoused.

• Product Developers and Service Providers:

  • Better understand the efficiency of kits and tests

  • Analyze usage, understand QC data trends, improve products

  • Store and aggregate business intelligence data such as lab identification, consumption patterns and frequency, as well as allow renderings of test result outcome trends and much more.

Base Action Possibilities

• Data Warehouse Creation: Build a relational database for your Project in which desired data sets can be selected and aggregated. Typical data sets include pipeline output metrics and other suitable data files generated by the ICA platform which can be complemented by additional public (or privately built) databases.

• Report and Export: Once created, a data warehouse can be mined using standard database query instructions. All Base data is stored in a structured and easily accessible way. An interface allows for the selection of specific datasets and conditional reporting. All queries can be stored, shared, and re-used in the future. This type of standard functionality supports most expected basic mining operations, such as variant frequency aggregation. All result sets can be downloaded or exported in various standard data formats for integration in other reporting or analytical applications.

• Detect Signals and Patterns: extensive and detailed selection of subsets of patients or samples adhering to any imaginable set of conditions is possible. Users can, for example, group and list subjects based on a combination of (several) specific genetic variants in combination with patient characteristics such as therapeutic (outcome) information. The built-in integration with public datasets allows users to retrieve all relevant publications, or clinically significant information for a single individual or a group of samples with a specific variant. Virtually any possible combination of stored sample and patient information allow for detecting signals and patterns by a simple single query on the big data set.

• Profile/Cluster patients: use and re-analyze patient cohort information based on specific sample or individual characteristics. For instance, they might want to run a next agile iteration of clinical trials with only patients that respond. Through integrated and structured consent information allowing for time-boxed use, combined with the capability to group subjects by the use of a simple query, patients can be stratified and combined to export all relevant individuals with their genotypic and phenotypic information to be used for further research.

• Share your data: Data sharing is subject to strict ethical and regulatory requirements. Base provides built-in functionality to securely share (sub)sets of your aggregated data with third parties. All data access can be monitored and audited, in this way Base data can be shared with people in and outside of an organization in a compliant and controlled fashion.

Access

Base is a module that can be found in a project. It is shown in the menu bar of the project.

Permission to Enable Base

The access to activate the Base module is controlled based upon the chosen subscription (full and premium subscriptions give access to Base) when registering the account. This will all happen automatically after the first user logs into the system for that account. So from the moment the account is up and running, the Base module will also be ready to be enabled.

Enable Base

When a user has created a project, they can go to the Base pages and click the Enable button. From that moment on, every user who has the proper permissions has access to the Base module in that project.

Only the project owner can enable Illumina Connected Analytics Base. Make sure that your subscription for the domain includes Base.

1. Navigate to Projects > your_project > Base > Tables / Query / Schedule.

2. Select Enable

Access Base pages

Access to the projects and all modules located within the project is provided via the Team page within the project.

Activity

The status and history of Base activities and jobs are shown on the page.

Nextflow offers support for Scatter-gather pattern natively. The initial uses this pattern by splitting the FASTA file into chunks to channel records in the task splitSequences, then by processing these chunks in the task reverse.

In this tutorial, we will create a pipeline which will split a TSV file into chunks, sort them, and merge them together.

Creating the pipeline

Select Projects > your_project > Flow > Pipelines. From the Pipelines view, click the +Create > Nextflow > XML based button to start creating a Nextflow pipeline.

In the Details tab, add values for the required Code (unique pipeline name) and Description fields. Nextflow Version and Storage size defaults to preassigned values.

First, we present the individual processes. Select +Nextflow files > + Create and label the file split.nf. Copy and paste the following definition.

Next, select +Create and name the file sort.nf. Copy and paste the following definition.

Select +Create again and label the file merge.nf. Copy and paste the following definition.

Add the corresponding main.nf file by navigating to the Nextflow files > main.nf tab and copying and pasting the following definition.

Here, the operators flatten and collect are used to transform the emitting channels. The Flatten operator transforms a channel in such a way that every item of type Collection or Array is flattened so that each single entry is emitted separately by the resulting channel. The collect operator collects all the items emitted by a channel to a List and return the resulting object as a sole emission.

Finally, copy and paste the following XML configuration into the XML Configuration tab.

Click the Generate button (at the bottom of the text editor) to preview the launch form fields.

Click the Save button to save the changes.

Running the pipeline

Go to the Pipelines page from the left navigation pane. Select the pipeline you just created and click Start New Analysis.

Fill in the required fields indicated by red "*" sign and click on Start button. You can monitor the run from the Analyses page. Once the Status changes to Succeeded, you can click on the run to access the results page.

In Projects > your_project > Flow > Analyses > your_analysis > Steps you can see that the input file is split into multiple chunks, then these chunks are sorted and merged.

Introduction

DRAGEN can run in workspaces

• In either FPGA mode (hardware-accelerated) or software mode when using FPGA instances. This can be useful when comparing performance gains by hardware acceleration or to distribute concurrent processes between the FPGA and cpu.

• In software mode when using non-FPGA instances.

The DRAGEN command line parameters to specify the location of the licence file are different.

• FPGA mode uses LICENSE_PARAMS=``"--lic-instance-id-location /opt/dragen-licence/instance-identity.protected --lic-credentials /opt/dragen-licence/instance-identity.protected/dragen-creds.lic"

• Software mode uses LICENSE_PARAMS="--sw-mode --lic-credentials /opt/dragen-licence/instance-identity.protected/dragen-creds-sw-mode.lic"

DRAGEN Bench Images

DRAGEN software is provided in specific Bench images with names starting with Dragen. For example (available versions may vary):

• Dragen 4.4.1 - Minimal provides DRAGEN 4.4.1 and SSH access

• Dragen 4.4.6 provides DRAGEN 4.4.6, SSH and JupyterLab.

Prerequisites

Memory

The instance type is selected during workspace creation (Projects > your_project > Bench > Workspaces). The amount of RAM available on the instance is critical. 256GiB RAM is a safe choice to run DRAGEN in production. All FPGA2 instances offer 256GiB or more of RAM.

When running in Software mode, use (348GiB RAM) or (144 GiB RAM) to ensure enough RAM is available for your runs.

FPGA-mode

Using an fpga2-medium .

Example

Software-mode

Using a standard-xlarge .

Software mode is activated with the DRAGEN --sw-mode parameter.

Example

The project details page contains the properties of the project, such as the location, owner, storage and linked bundles. This is also the place where you add assets in the form of linked .

The project details are configured during project creation and may be updated by the project owner, entities with the project Adminstrator role, and tenant administrators.

Adding Linked Bundle Assets

1. Click the Edit button at the top of the Details page.

2. Click the + button, under LINKED BUNDLES.

3. Click on the desired bundle, then click the Link button.

4. Click Save.

If your linked bundle contained a pipeline, then it will appear in Projects > your_project > Flow > Pipelines.

Details List

Billing Mode

A project's billing mode determines the strategy for how costs are charged to billable accounts.

For example, with billing mode set to Tenant, if tenant A has created a project resource and uses it in their project, then tenant A will pay for the resource data, compute costs and storage costs of any output they generate within the project. When they share the project with tenant B, then tenant B will pay the compute and storage for the data which they generate in that project. Put simply, in billing mode tenant, the person who generates data pays for the processing and storage of that data, regardless of who owns the actual project.

If the project billing mode is updated after the project has been created, the updated billing mode will only be applied to resources generated after the change.

If you are using your own S3 storage, then the billing mode impacts where collaborator data is stored.

• Project billing will result in using your S3 storage for the data.

• Tenant billing will result in collaborator data being stored in Illumina-managed storage instead of your own S3 storage.

• Tenant billing, when your collaborators also have their own S3 storage and have it set as default, will result in their data being stored in their S3 storage.

Authentication Token

Use the Create OAuth access token button to generate an OAuth access token which is valid for 12 hours after generation. This token can be used by Snowflake and Tableau to access the data in your Base databases and tables for this Project.

See for more information.

On the Schedule page at Projects > your_project > Base > Schedule, it’s possible to create a job for importing different types of data you have access to into an existing table.

When creating or editing a schedule, Automatic import is performed when the Active box is checked. The job will run at 10 minute intervals. In addition, for both active and inactive schedules, a manual import is performed when selecting the schedule and clicking the »run button.

Configure a schedule

There are different types of schedules that can be set up:

• Files

• Metadata

• Administrative data.

Files

This type will load the content of specific files from this project into a table. When adding or editing this schedule you can define the following parameters:

• Name (required): The name of the scheduled job

• Description: Extra information about the schedule

• File name pattern (required): Define in this field a part or the full name of the file name or of the tag that the files you want to upload contain. For example, if you want to import files named sample1_reads.txt, sample2_reads.txt, … you can fill in _reads.txt in this field to have all files that contain _reads.txt imported to the table.

• Generated by Pipelines: Only files generated by these selected pipelines are taken into account. When left clear, files from all pipelines are used.

• Target Base Table (required): The table to which the information needs to be added. A drop-down list with all created tables is shown. This means the table needs to be created before the schedule can be created.

• Write preference (required): Define data handling; whether it can overwrite the data

• Data format (required): Select the data format of the files (CSV, TSV, JSON)

• Delimiter (required): to indicate which delimiter is used in the delimiter separated file. If the delimiter is not present in list, it can be indicated as custom.

• Active: The job will run automatically if checked

• Custom delimiter: the custom delimiter that is used in the file. You can only enter a delimiter here if custom delimiter is selected.

• Header rows to skip: The number of consecutive header rows (at the top of the table) to skip.

• References: Choose which references must be added to the table

• Advanced Options

  • Encoding (required): Select the encoding of the file.

  • Null Marker: Specifies a string that represents a null value in a CSV/TSV file.

  • Quote: The value (single character) that is used to quote data sections in a CSV/TSV file. When this character is encountered at the beginning and end of a field, it will be removed. For example, entering " as quote will remove the quotes from "bunny" and only store the word bunny itself.

  • Ignore unknown values: This applies to CSV-formatted files. You can use this function to handle optional fields without separators, provided that the missing fields are located at the end of the row. Otherwise, the parser can not detect the missing separator and will shift fields to the left, resulting in errors.

    • If headers are used: The columns that have matching fields are loaded, those that have no matching fields are loaded with NULL and remaining fields are discarded.

    • If no headers are used: The fields are loaded in order of occurrence and trailing missing fields are loaded with NULL, trailing additional fields are discarded.

Metadata

This type will create two new tables: BB_PROJECT_PIPELINE_EXECUTIONS_DETAIL and ICA_PROJECT_SAMPLE_META_DATA. The job will load metadata (added to the samples) into ICA_PROJECT_SAMPLE_META_DATA. The process gathers the metadata from the samples via the data linked to the project and the metadata from the analyses in this project. Furthermore, the schedular will add provenance data to BB_PROJECT_PIPELINE_EXECUTIONS_DETAIL. This process gathers the execution details of all the analyses in the project: the pipeline name and status, the user reference, the input files (with identifiers), and the settings selected at runtime. This enables you to track the lineage of your data and to identify any potential sources of errors or biases. So, for example, the following query will count how many times each of the pipelines was executed and sort it accordingly:

SELECT PIPELINE_NAME, COUNT(*) AS Appearances
FROM BB_PROJECT_PIPELINE_EXECUTIONS_DETAIL
GROUP BY PIPELINE_NAME
ORDER BY Appearances DESC;

To obtained the similar table for the failed runs, you can execute the following SQL query:

SELECT PIPELINE_NAME, COUNT(*) AS Appearances
FROM BB_PROJECT_PIPELINE_EXECUTIONS_DETAIL
WHERE PIPELINE_STATUS = 'Failed'
GROUP BY PIPELINE_NAME
ORDER BY Appearances DESC;

When adding or editing this schedule you can define the following parameters:

• Name (required): the name of this scheduled job.

• Description: Extra information about the schedule.

• Include sensitive meta data fields: in the meta data fields configuration, fields can be set to sensitive. When checked, those fields will also be added.

• Active: the job will run automatically if ticked.

• Source (Tenant Administrators Only):

  • Project (default): All administrative data from this project will be added.

  • Account: All administrative data from every project in the account will be added. When a tenant admin creates the tenant-wide table with administrative data in a project and invites other users to this project, these users will see this table as well.

Administrative data

This type will automatically create a table and load administrative data into this table. A usage overview of all executions is considered administrative data.

When adding or editing this schedule the following parameters can be defined:

• Name (required): The name of this scheduled job.

• Description: Extra information about the schedule.

• Include sensitive metadata fields: In the metadata fields configuration, fields can be set to sensitive. When checked, those fields will also be added.

• Active: The job will run automatically if checked.

• Source (Tenant Administrators Only):

  • Project (default): All administrative data from this project will be added.

  • Account: All administrative data from every project in the account will be added. When a tenant admin creates the tenant-wide table with administrative data in a project and invites other users to this project, these users will see this table as well.

Delete schedule

Schedules can be deleted. Once deleted, they will no longer run, and they will not be shown in the list of schedules.

Run schedule

When clicking the Run button, or Save & Run when editing, the schedule will start the job of importing the configured data in the correct tables. This way the schedule can be run manually. The result of the job can be seen in the tables.

This tutorial aims to guide you through the process of creating CWL tools and pipelines from the very beginning. By following the steps and techniques presented here, you will gain the necessary knowledge and skills to develop your own pipelines or transition existing ones to ICA.

Build and push to ICA your own Docker image

The foundation for every tool in ICA is a Docker image (externally published or created by the user). Here we present how to create your own Docker image for the popular tool (FASTQC).

Copy the contents displayed below to a text editor and save it as a Dockerfile. Make sure you use an editor which does not add formatting to the file.

FROM centos:7
WORKDIR /usr/local

# DEPENDENCIES
RUN yum -y install java-1.8.0-openjdk wget unzip perl && \
    yum clean all && \
    rm -rf /var/cache/yum

# INSTALLATION fastqc
RUN wget http://www.bioinformatics.babraham.ac.uk/projects/fastqc/fastqc_v0.11.9.zip --no-check-certificate && \
    unzip fastqc_v0.11.9.zip && \
    chmod a+rx /usr/local/FastQC/fastqc && rm -rf fastqc_v0.11.9.zip

# Adding FastQC to the PATH
ENV PATH $PATH:/usr/local/FastQC

# DEFAULTS
ENV LANG=en_US.UTF-8
ENV LC_ALL=en_US.UTF-8
ENTRYPOINT []

## how to build the docker image
## docker build --file fastqc-0.11.9.Dockerfile --tag fastqc-0.11.9:0 .
## docker run --rm -i -t --entrypoint /bin/bash fastqc-0.11.9:0

Open a terminal window, place this file in a dedicated folder and navigate to this folder location. Then use the following command:

docker build --file fastqc-0.11.9.Dockerfile --tag fastqc-0.11.9:1 .

Check the image has been successfully built:

docker images

Check that the container is functional:

docker run --rm -i -t --entrypoint /bin/bash fastqc-0.11.9:1

Once inside the container check that the fastqc command is responsive and prints the expected help message. Remember to exit the container.

Save a tar of the previously built image locally:

docker save fastqc-0.11.9:1 -o fastqc-0.11.9:1.tar.gz

Upload your docker image .tar to an ICA project (browser upload, Connector, or CLI).

Now go outside of the Project and go to System Settings > Docker Repository, Select Create > Image. Select your docker file and fill out a name and version and set your type to tool and Press Select.

Create a CWL tool

While outside of any Project go to System Settings > Tool Repository and Select +Create. Fill the mandatory fields (Name and Version) and look for a Docker image to link to the tool.

Tool creation in ICA adheres to the cwl standard.

You can create a tool by either pasting the tool definition in the code syntax field on the right or you can use the different tabs to manually define inputs, outputs, arguments, settings, etc …

In this tutorial we will use the CWL tool syntax method. Paste the following content in the General tab.

#!/usr/bin/env cwl-runner

# (Re)generated by BlueBee Platform

$namespaces:
  ilmn-tes: http://platform.illumina.com/rdf/iap/
cwlVersion: cwl:v1.0
class: CommandLineTool
label: FastQC
doc: FastQC aims to provide a simple way to do some quality control checks on raw
  sequence data coming from high throughput sequencing pipelines.
inputs:
  Fastq1:
    type: File
    inputBinding:
      position: 1
  Fastq2:
    type:
    - File
    - 'null'
    inputBinding:
      position: 3
outputs:
  HTML:
    type:
      type: array
      items: File
    outputBinding:
      glob:
      - '*.html'
  Zip:
    type:
      type: array
      items: File
    outputBinding:
      glob:
      - '*.zip'
arguments:
- position: 4
  prefix: -o
  valueFrom: $(runtime.outdir)
- position: 1
  prefix: -t
  valueFrom: '2'
baseCommand:
- fastqc

Since the user needs to specify the output folder for FASTQC application (-o prefix), we are using the $(runtime.outdir) runtime parameter to point to the designated output folder.

Create the pipeline

Navigate to Projects > your_project > Flow > Pipelines > +Create > CWL Graphical.

Fill the mandatory fields and click on the Definition tab to open the Graphical Editor.

Expand the Tool Repository menu (lower right) and drag your FastQC tool into the Editor field (center).

Now drag one Input and one Output file icon (on top) into the Editor field as well. Both may be given a Name (editable fields on the right when icon is selected) and need a Format attribute. Set the Input Format to fastq and Output Format to html. Connect both Input and Output files to the matching nodes on the tool itself (mouse over the node, then hold-click and drag to connect).

Press Save, you just created your first FastQC pipeline on ICA!

Run a pipeline

First make sure you have at least one Fastq file uploaded and/or linked to your Project. You may use Fastq files available in the Bundle.

Navigate to Pipelines and select the pipeline you just created, then press Start analysis

Fill the mandatory fields and click on the + button to open the File Selection dialog box. Select one of the Fastq files available to you.

Press Start analysis on the top right, the platform is now orchestrating the pipeline execution.

View Results

Navigate to Projects > your_project > Flow > Analyses and observe that the pipeline execution is now listed and will first be in Status Requested. After a few minutes the Status should change to In Progress and then to Succeeded.

Once this Analysis succeeds click it to enter the Analysis details view. You will see the FastQC HTML output file listed on the Output files tab. Click on the file to open Data Details view. Since it is an HTML file Format there is a View tab that allows visualizing the HTML within the browser.

This walk-through is intended to represent a typical workflow when building and studying a cohort of oncology cases.

Create a Cancer Cohort and View Subject Details

1. Click Create Cohort button.

2. Select the following studies to add to your cohort:

   1. TCGA – BRCA – Breast Invasive Carcinoma

   2. TCGA – Ovarian Serous Cystadenocarcinoma

3. Add a Cohort Name = TCGA Breast and Ovarian_1472

4. Click on Apply.

5. Expand Show query details to see the study makeup of your cohort.

6. Charts will be open by default. If not, click Show charts

7. Use the gear icon in the top-right to change viewable chart settings.

   Tip: Disease Type, Histological Diagnosis, Technology, Overall Survival have interesting data about this cohorts

8. The Subject tab with all Subjects list is displayed below Charts with a link to each Subject by ID and other high-level information, like Data Types measured and reported. By clicking a subject ID, you will be brought to the data collected at the Subject level.

9. Search for subject TCGA-E2-A14Y and view the data about this Subject.

10. Click the TCGA-E2-A14Y Subject ID link to view clinical data for this Subject that was imported via the metadata.tsv file on ingest.

    Note: the Subject is a 35 year old Female with vital status and other phenotypes that feed up into the Subject attribute selection criteria when creating or editing cohorts.

11. Click X to close the Subject details.

12. Click Hide charts to increase interactive landscape.

Data Analysis, Multi-Omic Biomarker Discovery, and Interpretation

1. Click the Marker Frequency tab, then click the Somatic Mutation tab.

2. Review the gene list and mutation frequencies.

3. Note that PIK3CA has a high rate of mutation in the Cohort (ranked 2nd with 33% mutation frequency in 326 of the 987 Subjects that have Somatic Mutation data in this cohort).

   1. Do Subjects with PIK3CA mutations have changes in PIK3CA RNA Expression?

4. Click the Gene Expression tab, search for PIK3CA

   1. PIK3CA RNA is down-regulated in 27% of the subjects relative to normal samples.

      1. Switch from normal to disease Reference where the Subject’s denominator is the median of all disease samples in your cohort.

      2. The count of matching vs. total subjects that have PIK3CA up-regulated RNA which may indicate a distinctive sub-phenotype.

5. Click directly on PIK3CA gene link in the Gene Expression table.

6. You are brought to the Gene tab under the Gene Summary sub-tab that lists information and links to public resources about PIK3CA.

7. Click the Variants tab and Show legend and filters if it does not open by default.

8. Below the interactive legend you see a set of analysis tracks: Needle Plot, Primate AI, Pathogenic variants, and Exons.

9. The Needle Plot allows toggling the plot by gnomAD frequency and Sample Count. Select Sample Count in the Plot by legend above the plot.

   1. There are 87 mutations distributed across the 1068 amino acid sequence, listed below the analysis tracks. These can be exported via the icon into a table.

10. We know that missense variants can severely disrupt translated protein activity. Deselect all Variant Types except for Missense from the Show Variant Type legend above the needle plot.

    1. Many mutations are in the functional domains of the protein as seen by the colored boxes and labels on the x-axis of the Needle Plot.

11. Hover over the variant with the highest sample count in the yellow PI3Ka protein domain.

    1. The pop-up shows variant details for the 64 Subjects observed with it: 63 in the Breast Cancer study and 1 in the Ovarian Cancer Study.

12. Use the Exon zoom bar from each end of the Amino Acid sequence to zoom in to the PI3Ka domain to better separate observations.

13. There are three different missense mutations at this locus changing the wildtype Glutamine at different frequencies to Lysine (64), Glycine (6), or Alanine (2).

14. The Pathogenic Variant Track shows 7 ClinVar entries for mutations stacked at this locus affecting amino acid 545. Pop up details with pathogenicity calls, phenotypes, submitter and a link to the ClinVar entry is seen by hovering over the purple triangles.

15. Note the Primate AI track and high Primate AI score.

    1. Primate AI track displays Scores for potential missense variants, based on polymorphisms observed in primate species. Points above the dashed line for the 75th percentile may be considered likely pathogenic as cross-species sequence is highly conserved; you often see high conservancy at the functional domains. Points below the 25th percentile may be considered "likely benign".

16. Click the Expression tab and notice that normal Breast and normal Ovarian tissue have relatively high PIK3CA RNA Expression in GTex RNAseq tissue data but ubiquitously expressed.

You can access the databases and tables within the Base module using snowSQL command-line interface. This is useful for external collaborators who do not have access to ICA core functionalities. In this tutorial we will describe how to obtain the token and use it for accessing the Base module. This tutorial does not cover how to install and configure snowSQL.

Obtaining OAuth token and URL

Once the Base module has been enabled within a project, the following details are shown in Projects > your_project > Project Settings > Details.

After clicking the button Create OAuth access token, the pop-up authenticator is displayed.

After clicking the button Generate snowSQL command the pop-up authenticator presents the snowSQL command.

Copy the snowSQL command and run it in the console to log in.

You can also get the OAuth access token via API by providing <PROJECT ID> and <YOUR KEY>.

Example:

API Call:

curl -X 'POST' \
  'https://ica.illumina.com/ica/rest/api/projects/<PROJECT ID>/base:connectionDetails' \
  -H 'accept: application/vnd.illumina.v3+json' \
  -H 'X-API-Key: <YOUR KEY>' 

Response

{
  "authenticator": "oauth",
  "accessToken": "XXXXXXXXXX",
  "dnsName": "use1sf01.us-east-1.snowflakecomputing.com",
  "userPrincipalName": "xxxxx",
  "databaseName": "xxxxx",
  "schemaName": "xxx",
  "warehouseName": "xxxxxx",
  "roleName": "xxx"
}

Template snowSQL:

snowsql -a use1sf01.us-east-1 -u <userPrincipalName> --authenticator=oauth -r <roleName> -d <databaseName> -s PUBLIC -w <warehouseName> --token="<accessToken>"

Now you can perform a variety of tasks such as:

1. Querying Data: execute SQL queries against tables, views, and other database objects to retrieve data from the Snowflake data warehouse.

2. Creating and Managing Database Objects: create tables, views, stored procedures, functions, and other database objects in Snowflake. you can also modify and delete these objects as needed.

3. Loading Data: load data into Snowflake from various sources such as local files, AWS S3, Azure Blob Storage, or Google Cloud Storage.

Overall, snowSQL CLI provides a powerful and flexible interface to work with Snowflake, allowing external users to manage data warehouse and perform a variety of tasks efficiently and effectively without access to the ICA core.

Example Queries:

Show all tables in the database:

>SHOW TABLES;

Create a new table:

create TABLE demo1(sample_name VARCHAR, count INT);

List records in a table:

SELECT * FROM demo1;

Load data from a file: To load data from a file, you can start by create a staging area in the internal storage using the following commend:

>CREATE STAGE myStage;

You can then upload the local file to the internal storage using the following command:

> PUT file:///path/to/data.tsv @myStage;

You can check if the file was uploaded properly using LIST command:

> LIST @myStage;

Finally, Load data by using COPY TO command. The command assumes the data.tsv is a tab delimited file. You can easily modify the following command to import JSON file setting TYPE=JSON.

> COPY INTO demo1(sample_name, count) FROM @mystage/data.tsv FILE_FORMAT = (TYPE = 'CSV' FIELD_DELIMITER = '\t');

Load data from a string: If you have data as JSON string, you can import the data into the tables using following commands.

> SET myJSON_str = '{"sample_name": "from-json-str", "count": 1}';
> INSERT INTO demo1(sample_name, count)
> SELECT
    PARSE_JSON($myJSON_str):sample_name::STRING,
    PARSE_JSON($myJSON_str):count::INT

Load data into specific columns: If you want to load sample_name into the table, you can remove the "count" from the column and the value list as below:

> SET myJSON_str = '{"sample_name": "from-json-str", "count": 1}';
> INSERT INTO demo1(sample_name)
  SELECT
    PARSE_JSON($myJSON_str):sample_name::STRING;

List the views of the database to which you are connected. As shared database and catalogue views are created within the project database, they will be listed. However, it does not show views which are granted via another database, role or from bundles.

>SHOW VIEW;

Show grants, both directly on the tables and views and grants to roles which in turn have grants on tables and views.

>SHOW GRANTS;

Managing a Bench cluster

Introduction

Workspaces can have their own dedicated cluster which consists of a number of nodes. First the workspace node, which is used for interacting with the cluster, is started. Once the workspace node is started, the workspace cluster can be started.

The cluster consists of 2 components

• The manager node which orchestrates the workload across the members.

• Anywhere between 0 and up to maximum 50 member nodes.

Clusters can run in two modes.

• Static - A static cluster has a manager node and a static number of members. At start-up of the cluster, the system ensures the predefined number of members are added to the cluster. These nodes will keep running as long as the entire cluster runs. The system will not automatically remove or add nodes depending on the job load. This gives the fastest resource availability, but at additional cost as unused nodes stay active, waiting for work.

• Dynamic - A dynamic cluster has a manager node and a dynamic number of workers up to a predefined maximum (with a hard limit of 50). Based on the job load the system will scale the number of members up or down. This saves resources as only as much worker nodes as needed to perform the work are being used.

Configuration

You manage Bench Clusters via the Illumina Connected Analytics UI in Projects > your_project > Bench > Workspaces > your_workspace > Details.

The following settings can be defined for a bench cluster:

Operations

Once the workspace is started, the cluster can be started at Projects > your_project > Bench > Workspaces > your_workspace > Details and the cluster can be stopped without stopping the workspace. Stopping the workspace will also stop all clusters in that workspace.

Managing Data in a Bench cluster

Data in a bench workspace can be divided into three groups:

• Workspace data is accessible in read/write mode and can be accessed from all workspace components (workspace node, cluster manager node, cluster member nodes ) at /data. The size of the workspace data is defined at the creation of the workspace but can be increased when editing a workspace in the Illumina connected analytics UI. This is persistent storage and data remains when a workspace is shut down.

• Project data can be accessed from all workspace components at /data/project. Every component will have their own dedicated mount to the project. Depending on the project data permissions you will be able to access it in either Read-Only or Read-Write mode.

• Scratch data is available on the cluster members at /scratch and can be used to store intermediate results for a given job dedicated to that member. This is temporary storage, and all data is deleted when a cluster member is removed from the cluster.

Fast Read-Only Access

All mounts occur in /data/mounts/, see data access and workspace-ctl data.

Managing these mounts is done via the workspace cli /data/.local/bin/workspace-ctl in the workspace. Every node will have his dedicated mount.

For fast data access, bench offers a mount solution to expose project data on every component in the workspace. This mount provides read-only access to a given location in the project data and is optimized for high read throughput per single file with concurrent access to files. It will try to utilise the full bandwidth capacity of the node.

All mounts occur in path /data/mounts/

Show mounts

workspace-ctl data get-mounts

Creating a mount

For fast read-only access, link folders with the CLI command workspace-ctl data create-mount --mode read-only.

workspace-ctl data create-mount --mount-path /data/mounts/mydata --source /data/project/mydata

Removing a mount

workspace-ctl data delete-mount --mount-path /data/mounts/mydata

ICA Cohorts comes front-loaded with a variety of publicly accessible data sets, covering multiple disease areas and also including healthy individuals.