Before you proceed to this article, make sure you understand data storage management basics.

Update Azure Blob Storage Credentials

In Settings > Management Tab, add or edit the required credentials: CLIENT_ID, CLIENT_SECRET, TENANT_ID, and ACCOUNT_URL.

See the table below to learn where to look for them in your Azure account.

Blob Integration Setup

Create an App registration

1. In Microsoft Entra ID, click on App registrations.

1. Select New registration.

2. Fill the name of the application & press "register."

3. You got to the registered app page: (CLIENT_ID / TENANT_ID) From this you can retrieve: Application ID and Tenant ID. Both are marked in the screenshot.

1. Press "Certificates & secrets"

2. Press on "New Client secret"

1. Fill the "Description" and change expires to 12 months. (or according to your organization policy), than press "Add"

8. Get the CLIENT_SECRET from this page.

1. Give this App registration roles and read access to the relevant Blob.

Azure Blob configuration

1. Go to Azure Storage accounts

1. Get into the relevant Storage account

1. Press on "containers"

1. Press on the relevant container

2. Press on "Properties"

3. Copy the ACCOUNT_URL

For Internal support:

Errors for bad connections can be found in CloudWatch on particular FRY log stream

Search for: BlobApi, BlobFs, azure.

To directly import files from your own storage, link it to an organization's storage in Emedgene.

How to link your storage to Emedgene:

How to edit storage information:

Click Manage on the right to the storage details.

How to remove a link to storage:

Click Delete on the right to the storage details.

Whenever an organization is created, we automatically allocate bucket folders in AWS S3 cloud storage to it:

• Path for upload

Folder intended to store input case files.

Authorized user has view and upload privileges.

• Path for download

This folder contains a partially annotated (excluding results of proprietary algorithms) VCF file per case.

Authorized user has view and download privileges.

• Path for DRAGEN output

This folder contains DRAGEN output files.

Authorized user has view and download privileges.

To get access to your upload, download and DRAGEN output folders, you need to get a key pair consisting of an access key ID and a secret access key. , , and credentials is available for users with Manager and Manage S3 Credentials .

You can create and use up to two dynamic access keys at the same time.

When you require technical support, you have the option to generate a new key pair specifically for the troubleshooting process. After the issue has been resolved, you can delete the credentials to ensure security of your system.

The newly generated credentials will only be saved in AWS Identity and Access Management (IAM) and not in our database.

How to create a key pair

1. In Settings > Management > S3 Credentials, click on Create Access Key.

2. You can retrieve the secret access key only when you initially create the key pair. If you lose it, you have to create a new key pair. To immediately copy the secret access key to a secure location, use the Copy to clipboard button.

How to deactivate a key pair

In Settings > Management > S3 Credentials, click on Deactivate in the corresponding key pair card.

How to activate an inactive key pair

In Settings > Management > S3 Credentials, click on Activate in the corresponding key pair card.

How to delete a key pair

In Settings > Management > S3 Credentials, click on Delete in the corresponding key pair card. Only inactive key pairs can be deleted.

How to get your ICA credentials:

To connect ICA:

Log in to Emedgene and navigate to Settings in the upper right-hand corner of the page.

Click on the Management tab and then on Add Storage.

Choose Illumina BaseSpace storage type.

Fill Client Key, Client Secret and App Token as provided from BaseSpace (a description on how to get this information is provided below) and click Add storage to complete the setup.

Via Command Line

Prerequisite

Install BaseSpace CLI (Command Line Interface)

Follow the instructions on the if needed. Be aware of the Basespace Regional Instance you are working on (us, euc1, aps2, euw2)

Authenticate

On BSSH, login to the workgroup you want to connect as the storage.

Once the BaseSpace CLI is installed, run the authentication command in the terminal.

The command will direct you to a link which requires to login.

After the authentication was completed successfully, find the access token in the config file.

The result should look like -

Populate the App_token with the accessToken value, and Server with the apiServer URL from the BSSH config file.

Client_key will be displayed in subsequent menus, so a descriptive name such as the workgroup name can be used.

Client_secret is unused when the App_token is available and can be set to "x".

Via BaseSpace Developer Portal

Go to the BaseSpace and login. Be aware of the Basespace Regional Instance you are working on (us, , , )

Go to My Apps and click Create a new Application.

Fill details for the application and click on create an application.

Fill details and press save.

You will need to fill all the fields that it requested, please add “NA” to them.

Go to My Apps and click on your new app. Then go to the credentials tab.

You will find the Client ID (Client Key), Client Secret and App Token to enter to Emedgene platform.

Adding BSSH account to your Emedgene account

1. Log in into the desired Emedgene organization.

2. Go to Settings

3. Go to Management tab

4. Click on Add Storage

1. Add the information from your “Credentials” of the App previously created in BSSH.

Scope

Bring Your Own Key (BYOK) is a security feature that allows organizations to use their own encryption keys to protect their data. This ensures that they maintain control over their encryption keys and, consequently, their data.

Supported Key Management Services

Illumina integrates with leading Key Management Services (KMS), including Azure Key Vault and AWS KMS, so organizations can maintain full control over their encryption keys. These integrations combine Illumina’s Bring Your Own Key (BYOK) feature with your preferred KMS provider to deliver robust key management and enhanced data security.

Azure Key Vault

is a cloud service that provides a secure way to store and manage sensitive information like API keys, passwords, and certificates. It offers robust features for key management, including key generation, storage, and lifecycle management.

AWS KMS

(KMS) allows you to create and control encryption keys used to encrypt your data across a wide range of AWS services and applications. It provides centralized management of encryption keys and integrates seamlessly with other AWS services.

Setup

Azure Key Vault Setup

The API server encrypts the organization's information before storing it in the database and decrypts it when needed (e.g., during pipeline execution). The key vault is managed by the organization.

To configure encryption in Emedgene, you need the following information from Azure Key Vault:

Application tokens:

• Client Id

• Tenant Id

• Client Secret

The key information:

• Key URL

Create a new application

Add a client secret

Create a new key

Find key details

AWS Key Management Service (KMS) Setup

Description is coming soon.

Architecture

The API server will encrypt the client's information before storing it in a database and decrypt that information when needed (e.g., running the pipeline). The key vault is managed by the client, and Emedgene will only be provided with access to encrypt/decrypt functions in that key vault. This guarantees that clients control access to the information.

Illustration of data flow when creating a case in Emedgene platform:

Illustration of data flow when reading a case data from emedgene platform:

A preliminary step to this solution is having a key vault owned by the client, and a key that Emedgene is given access to.

The client will create an access policy in the key vault of type “Application” and provide the matching key and secret to Emedgene. The access policy must contain permissions to perform encrypt and decrypt actions.

In order for Emedgene to integrate with the key, depending on the key vault provider, the client needs to provide the following information:

• Client Id

• Client Secret

• Tenant Id

• Key vault name

Searching Encrypted Fields

Since some of our platform search capabilities run directly on the DB, we can’t directly search any data that is encrypted. To overcome this, we will implement a hashing search functionality as follows.

• The case data will still be fully encrypted in the DB as it is today

• Specific fields we want to make “searchable” - as defined by the customer, we will save their hash value alongside the encrypted data.

• Hashing will be done using SHA-256, and will include a secure random generated salt of 32 characters, which will be added to the value.

Illustration of data flow when searching in Emedgene platform:

Illustration of data flow when creating a case with searchable field in Emedgene platform:

Appendix

If you have an Enterprise account and you would like Emedgene-managed DRAGEN solution to save the DRAGEN output files in your own bucket, reach out to [email protected].

Emedgene visualizes data in IGV directly from your AWS S3 bucket. In order to do it, you should enable CORS for the Emedgene application URLs.

This feature is only related to saving Dragen output files in your own bucket when using Dragen through Emedgene (without ICA).

If you are looking to:

• Import data from AWS S3 to Emedgene go to

• Integrating any data storage to Emedgene go to

• Download any data from Emedgene go to

Bring Your Own Bucket

Bring Your Own Bucket, also known as BYOK, enables you to control your DRAGEN file outputs.

Emedgene-managed DRAGEN solution saves the DRAGEN output files in a detected AWS S3 bucket that you have access to using your .

However, if you have an Enterprise account and you would like Emedgene-managed DRAGEN solution to save the DRAGEN output files in your own bucket, reach out to [email protected] and follow this steps:

1. Create an AWS bucket

Emedgene requires access to the root folder, which means a dedicated bucket might be appropriated.

2. Edit Bucket policy

Bucket policy should allow Emedgene user access to the bucket.

Example bucket policy:

3. Allow illumina.com and emedgene.com for CORS

Emedgene directly from your AWS S3 bucket. In order to do it, you should enable for the Emedgene application URLs.

Example CORS policy:

4. Test and validate the configuration with Illumina support

We will require to run a case and validate the managed DRAGEN pipeline finish successfully and all features are available in the platform.

Managing AWS S3 Lifecycle policy

If a customer enables an AWS S3 Lifecycle policy in order to archive or change the S3 tiers for different files, they might create an adverse effect on the platform.

Google Cloud Storage Credentials update procedure

How to get the client credentials?

1. Go to the google cloud Console.

2. Navigate to IAM & Admin - In the left sidebar, go to IAM & Admin > Service Accounts.

1. Create a New Service Account: Click on the "Create Service Account" button at the top.

1. Fill in the Service Account Details:

   • Service account name: Give your service account a name.

   • Service account ID: This will be automatically generated based on the name.

1. Assign Roles to the Service Account:

   • In the Grant this service account access to project step, you’ll assign the necessary roles.

   • Grant these role:

Add the storage provider to Emedgene platform:

• Add the above 3 values into the appropriate fields:

  • Client_credentials_base64: pasting the output of 8.

  • Bucket: the bucket name.

CORS - Visualisation

1. Download and install the Google Cloud SDK from the Google Cloud SDK Install page.

2. Select Your Platform (Windows, macOS, or Linux), download and run.

3. Initialize and Authenticate with Google Cloud: In the Cloud SDK Shell/terminal, run: gcloud init This will open a browser window to authenticate your Google account. Follow the instructions to log in and select your project.

notice:

• origin: if using Illumina cloud:

  https://host_name.emg.illumina.com

  else, Emedgene cloud:

  https://host_name.emedgene.com

1. Apply CORS Configuration to Your Bucket: run the next command. gcloud storage buckets update gs://your-bucket-name --cors-file=cors.json

2. Verify the CORS Configuration: gcloud storage buckets describe gs://your-bucket-name