Data Upload from User Storage (Connected Insights - Cloud and Connected Insights - Local)

Connected Insights - Cloud uses the Data Uploader tool that supports uploading VCF files and analysis output for user-provided machines and storage devices.

Connected Insights - Local includes the Data Uploader tool that is installed on the DRAGEN server as part of the Connected Insights - Local installation and supports uploading VCF files and analysis output from the storage device used during installation. The Data Uploader tool is detected and identified asDefault-CLI-Installation in the Data Upload section of the Configuration page. If Default-CLI-Installation does not show as Online, refer to Software Errors and Corrective Actions.

The mounted storage used during installation is the monitoring location where the secondary analysis output data is stored and read by the Data Uploader.When setting up the Data Uploader for monitoring, the mounted storage path (for example, /mnt/{ici_mount}/) must be replaced with /data/.

The Data Uploader logs can be found at /staging/ici/logs/tss-cli/. With Connected Insights - Local, these logs can also be downloaded from theActivity tab in the Case Details pane on the Cases page. For more information, refer to Case Details.

Setup (Connected Insights - Cloud Only)

This section identifies the requirements for uploading data from user storage, enabling ingestion from the network drive, and adding an existing pipeline. For Connected Insights - Cloud, this section also covers how to download and install the Data Uploader tool and has instructions for generating an API key.

Requirements

To upload data from user storage, the following requirements must be met:

• Make sure you have Java 8 or a newer version installed on your computer. You can check by opening a terminal or command prompt and typing java -version. If you don't have it installed, you can download and install it from the official Java website.

• Data Uploader with Java Virtual Machine (JVM) 8+ that is compatible with Mac, Windows, or Linux CentOS.

• An API key and access to the workgroup to which the data is uploaded. The API key comes with the installer. To generate a newAPI key, refer to [Optional] Generate an API Key.

• Administrator access on the computer where Data Uploader is installed.

Before you begin, perform the following actions:

1. Create custom case data.

2. Create test components.

3. Create test definitions.

For more information, refer to Configuration.

Download and Install Data Uploader

Download and install the Data Uploader tool as follows.

1. In Connected Insights, select the gear at the top right of the page.

2. In the General tab, under Data Upload, select the From Local Storage tab.

3. Under Download and Launch the Data Uploader, select the storage device operating system from the drop-down list.

4. Select Download Data Uploader. A progress bar displays during the download.

5. Copy the installation directory of the storage device. Make sure that Data Uploader is in a location that has access to your secondary analysis output folder.

6. Use the following tar command to extract the files: Replace {ici uploader script} with the applicable file name. tar xvzf {ici uploader script}.tar.gz

   ❗ Extracting files can vary depending on the operating system. Most terminals support the tar command. For Windows, you can use a zip file extraction application (eg, 7-Zip) to extract the content of the tar folder.

7. Make sure that the files in the following table are in the unzipped Data Uploader file.

   Component	Description
   ici-uploader-daemon.jar	Used to run Data Uploader as a daemon.
   ici-uploader.jar	Used to run Data Uploader on demand.
   uploader-config.json	This file contains the configuration for Data Uploader to allow it to connect to Connected Insights.
   ici-uploader.exe	The installer for ici-uploader on Mac or Windows.
   Mac: com.illumina.isi.daemon.plist	The installer for ici-uploader on Mac or Windows.
   Linux: com.illumina.isi.daemon.service	The installer for ici-uploader on Mac or Windows.
   ici-uploader or ici-uploader.exe	The installer for ici-uploader on Mac or Windows.
   wrapper.exe	This file is used to set up Data Uploader as a system service on Windows.
   README.txt	Third-party licensing information for Data Uploader.

8. Install Data Uploader as follows:

   a. [Windows] Start the command prompt as an administrator and run ici-uploader.exe install.

   ❗ For windows environment, user must be in the installation directory to install/uninstall the application. ici-uploader.exe start-daemon

   b. [Mac and Linux] Run the following command on the terminal ./ici-uploader install

   ❗ If the installation fails, start the Connected Insights installation manually using ici-uploader start-daemon

   c. Follow any prompts in Data Uploader. d. Enter your API key and press Enter.

   ❗ If you did not download with an auto-generated API key, provide one to the installer when prompted. If you are installing Data Uploader for the first time, you must generate an API key. For more information, refer to [Optional] Generate an API Key.

   e. Under Define and Monitor Data Uploads, make sure that your machine displays.

Data Uploader is now set up for auto and manual ingestion. You can also change the name of the installed Data Uploader by selecting ... and Edit Server Name . If you must download any logs associated with the Data Uploader from the last 24 hours, select ... and Download Logs.

❗ If the user is not a system administrator, the daemon can be started with the ici-uploader start-daemon command for Mac/Linux or with the ici-uploader.exe start-daemon command for Windows. This command does not run the process as a system service and requires you to start and stop the service manually. To stop the daemon, run the command ici-uploader stop-daemon

After Data Uploader has been downloaded and installed on the user storage, you can set up configurations for the tool to upload data into Connected Insights automatically. Each configured pipeline monitors user storage for new molecular data.

f. User can stop the ici-uploader (running as system service) using the following command ici-uploader stop-service command for Mac/Linux or with the ici-uploader.exe stop-service command for Windows. User can start the uploader via ici-uploader start-service.

❗If data-uploader is already running as system-service, and user also runs the uploader manually via ici-uploader start-daemon, it may cause issues. User must stop already running data-uploader with ici-uploader stop-service before starting uploader manually.

Enable Upload from Network Drive

If you are using Linux, automatic case upload is already enabled and this section is not applicable. For Mac, contact Illumina Technical Support. For Windows, run the Connected Insights Auto-launch Service to make the network drive available for Data Uploader as follows:

1. When the Data Uploader is running, open the Services application from the Start menu and locate the Connected Insights Auto-launch Service.

2. Right-click Connected Insights Auto-launch Service and select Properties from the drop-down menu.

3. In Properties, select the Log On tab and enter your account ID and password. Confirm the password.

4. Select the General tab.

5. Select Stop, then select Start to start the service. The network drive is available for ingestion for Data Uploader.

After completing these steps, the Data Uploader details are visible in Connected Insights.

Select Compatible Pipeline

1. In Configuration Settings, select the radio button next to Choose compatible pipeline from catalog.

2. Select a pipeline from the drop-down list (for example, DRAGEN TruSight Oncology 500 Analysis Software v2.5.2).

   ❗ When running the DRAGEN Somatic Whole Genome pipeline in the Tumor-Only mode, you must set --output-file-prefix to match sample-id (the RGSM of the sample in the FASTQ list) of the run.

3. For Test Definition, select the applicable definition.

4. For Choose a folder to monitor for case metadata (optional), enter the path for the folder in the secondary analysis folder created by Data Uploader.

5. Select Save.

Generate an Api Key

If you are using Data Uploader for the first time, then you must generate a new API key. All Data Uploader operations require an API key for authentication. The Data Uploader bundle can be downloaded with an autogenerated API key. If the bundle includes an API key, skip this section.

❗ If you are installing Data Uploader on multiple machines, manually create and track your API key by running the following auto-updating and manual run command:

ici-uploader configure --api-key={apiKey} Make sure that the API key is within single quotation marks (for example,'{SYSTEM_API-KEY}')

1. In Connected Insights, select Manage API Keys from the Account drop-down menu.

2. Select Generate.

3. Enter a name for the API key.

4. To generate a global API key, select All workgroups and roles.

5. Select Generate.

6. In the API Key Generated window, select one of the following options:

   • Show — Reveals the API key.

   • Download API Key — Downloads the API key in .TXT file format.

7. Select Close after you have stored the API key.

   ❗ The API key cannot be viewed again after closing this window. Download the API key or save it in a secure location.

   The API key is added to the Manage API keys list.

8. Perform any of the following actions in the Manage API Keys list:

   • Select Regenerate to generate a new API with the existing API key name.

   • Select Edit to edit the API key name or change the workgroups and roles selection.

   • Select Delete to delete the API key.

Automation of Data Upload from User Storage ( Connected Insights - Cloud and Connected Insights - Local )

The following information is applicable to both Connected Insights - Local and Connected Insights - Cloud. Create each configuration as follows.

1. Input the monitoring location for secondary analysis output.

   • The monitoring location is the full path to the location where secondary analysis output data is deposited in user storage.

     • For example: /rest/of/path/{monitoring location}/run/{sample folders, sample sheet, inputs}

     • For DRAGEN server standalone results: /rest/of/path/{monitoring location}/{sample folders, sample sheet, inputs} For both examples, replace /rest/of/path/{monitoring location} with your file path.

   ❗ For Connected Insights - Local only, the monitoring location is the full path to the location where secondary analysis output data is stored in the mounted external storage drive (within /mnt/{ici_mount}/, where /mnt/{ici_mount}/ is replaced with /data/).

2. Associate the workflow schema for this pipeline. a. Under Choose compatible pipeline from the catalog, select the applicable pipeline from the drop-down list (for example, DRAGEN WGS Somatic v4.2). b. If you are running a custom workflow, then upload a workflow schema that corresponds to the data that the configuration uploads. For more information, refer to Custom Pipeline Configuration.

3. Select the test definition to be associated with cases created by this configuration. For more information, refer to Test Definition Setup.

4. [Optional] Input the location where custom case data is stored. This location is the full path to where custom case data is deposited in user storage. For more information, refer to Custom Case Data Upload.

5. Name the configuration for future reference on the user interface.

6. Select Save to complete the configuration. If Data Uploader is running, it monitors this configuration for data upload.

If the upload fails, recreate the case or resume the upload by performing the actions in Software Errors and Corrective Actions.

On-Demand Data Upload from User Storage (Connected Insights - Cloud Only)

1. Upload an analysis output by running the following command: ici-uploader analysis upload --folder={path-to-analysis} --pipeline-name={pipelineName}

   • --folder {path-to-analysis} — The absolute path to the analysis output to upload into Connected Insights. This folder contains the sample sheet.

   • --pipeline-name={pipelinename} — The name of the pipeline created in Connected Insights to apply to cases uploaded from this analysis. Pipeline names must include only letters, numbers, underscores, and dashes. The name cannot include spaces or special characters.

   • [Optional] --runId={path-to-config} — The id of the run to be created in place of the run ID determined by the run folder name.

   • [Optional] --pair-id={pair-id} — The pair-id of the analysis to upload from the Sample Sheet when limiting upload to a single analysis.

   • [Optional] --case-display-id={case-display-id} — The id of the case to be created in place of the pair-id when uploading a single analysis with --pair-id={pair-id}.

   • [Optional] ici-uploader logs show — Run to display the logs in ICI_Data_Upload_log.json.

   • [Optional] Downloads > ici-uploader logs download — Download Data Uploader logs in a zipped folder by running the command prompt.

2. Upload the custom case data file associated to one or more cases by running the following command: ici-uploader case-data --filePath={absolute-path-of-csv-file}

For more information on custom case data files, refer to Custom Case Data Upload.

Case Processing Performance (Connected Insights - Local Only)

The following table shows the approximate time it takes for example datasets to upload and receive the Ready for Interpretation status. The duration is evaluated by analyzing a batch of samples ingested through the Connected Insights tertiary analysis in conjunction with the DRAGEN secondary analysis performed on similar datasets on the DRAGEN server.

Sample Size	Duration
8 TSO Solid DNA cases with an average of 90,000 variants	45 minutes
8 DRAGEN WGS tumor normal cases with 50,000 variants	15 minutes
8 WGS TO cases with 4,000,000 variants	2 hours