Whenever an organization is created, we automatically allocate bucket folders in AWS S3 cloud storage to it:

• Path for upload

<Region_Cloud>-emg-auto-samples/<org_name>/upload/

Folder intended to store input case files.

Authorized user has view and upload privileges.

• Path for download (32.0+)

<Region_Cloud>-emg-downloads/<org_name>/ 

This folder contains a partially annotated (excluding results of proprietary algorithms) VCF file per case.

Authorized user has view and download privileges.

• Path for DRAGEN output (32.0+)

<Region_Cloud>-emg-auto-results/<org_name>/ 

This folder contains DRAGEN output files.

Authorized user has view and download privileges.

To get access to your upload, download and DRAGEN output folders, you need to get a key pair consisting of an access key ID and a secret access key. Creating, deactivating, activating and deleting credentials is available for users with Manager and Manage S3 Credentials roles.

You can create and use up to two dynamic access keys at the same time.

When you require technical support, you have the option to generate a new key pair specifically for the troubleshooting process. After the issue has been resolved, you can delete the credentials to ensure security of your system.

The newly generated credentials will only be saved in AWS Identity and Access Management (IAM) and not in our database.

How to create a key pair

1. In Settings > Management > S3 Credentials, click on Create Access Key.

2. You can retrieve the secret access key only when you initially create the key pair. If you lose it, you have to create a new key pair. To immediately copy the secret access key to a secure location, use the Copy to clipboard button.

How to deactivate a key pair

In Settings > Management > S3 Credentials, click on Deactivate in the corresponding key pair card.

How to activate an inactive key pair

In Settings > Management > S3 Credentials, click on Activate in the corresponding key pair card.

How to delete a key pair

In Settings > Management > S3 Credentials, click on Delete in the corresponding key pair card. Only inactive key pairs can be deleted.

Log in to Emedgene and navigate to Settings in the upper right-hand corner of the page.

Click on the Management tab and then on Add Storage.

Choose Illumina BaseSpace storage type.

Fill Client Key, Client Secret and App Token as provided from BaseSpace (a description on how to get this information is provided below) and click Add storage to complete the setup.

• Via Command Line

• Via BaseSpace Developer Portal

Via Command Line

Prerequisite

Install BaseSpace CLI (Command Line Interface)

# Linux
$ wget "https://launch.basespace.illumina.com/CLI/latest/amd64-linux/bs" -O $HOME/bin/bs
# Mac
$ wget "https://launch.basespace.illumina.com/CLI/latest/amd64-osx/bs" -O $HOME/bin/bs
# or
$ brew tap basespace/basespace && brew install bs-cli
# Windows
$ wget "https://launch.basespace.illumina.com/CLI/latest/amd64-windows/bs.exe" -O bs.exe

Follow the instructions on the BaseSpace CLI Installation Page if needed.

Authenticate

On BSSH, login to the workgroup you want to connect as the storage.

Once the BaseSpace CLI is installed, run the authentication command in the terminal.

$ bs auth

The command will direct you to a link which requires to login.

After the authentication was completed successfully, find the access token in the config file.

$ cat .basespace/default.cfg

The result should look like -

apiServer   = https://api.basespace.illumina.com
accessToken = 

Populate the App_token with the accessToken value, and Server with the apiServer URL from the BSSH config file.

Client_key will be displayed in subsequent menus, so a descriptive name such as the workgroup name can be used.

Client_secret is unused when the App_token is available and can be set to "x".

Via BaseSpace Developer Portal

Go to the BaseSpace developer portal and login.

Go to My Apps and click Create a new Application.

Fill details for the application and click on create an application.

Fill details and press save.

Go to My Apps and click on your new app. Then go to the credentials tab.

You will find the Client ID (Client Key), Client Secret and App Token to enter to Emedgene platform.

However, if you have an Enterprise account and you would like Emedgene managed DRAGEN solution to save the DRAGEN output files in your own bucket, reach out to techsupport@illumina.com and follow this steps:

Emedgene visualizes data in IGV directly from your AWS S3 bucket. In order to do it, you should enable CORS for the Emedgene application URLs.

This feature is only related to saving Dragen output files in your own bucket when using Dragen through Emedgene (without ICA).

If you are looking to:

• Import data from AWS S3 to Emedgene go to Manage data storages

• Integrating any data storage to Emedgene go to Manage data storages

• Download any data from Emedgene go to Manage S3 credentials

Bring Your Own Bucket

Bring Your Own Bucket, also known as BYOK, enables you to control your DRAGEN file outputs.

Emedgene managed DRAGEN solution saves the DRAGEN output files in a detected AWS S3 bucket that you have access to using your S3 credentials.

However, if you have an Enterprise account and you would like Emedgene managed dragen solution to save the DRAGEN output files in your own bucket, reach out to techsupport@illumina.com and follow this steps:

1. Create an AWS bucket

Emedgene requires access to the root folder, which means a dedicated bucket might be appropriated.

2. Edit Bucket policy

Bucket policy should allow Emedgene user access to the bucket.

Example bucket policy:

{
    Coming Soon
}

3. Allow illumina.com and emedgene.com for CORS

Emedgene visualizes data in IGV directly from your AWS S3 bucket. In order to do it, you should enable CORS for the emedgene application URLs.

Example CORS policy:

{
    Coming Soon
}

4. Test and validate the configuration with Illumina support

We will require to run a case and validate the managed DRAGEN pipeline finish successfully and all features are available in the platform.

Managing AWS S3 Lifecycle policy

If a customer enables an AWS S3 Lifecycle policy in order to archive or change the S3 tiers for different files, they might create an adverse effect on the platform.

Coming in V37.0: Google Cloud storage support

Google Cloud Storage Credentials update procedure

How to get the client credentials?

1. Go to the google cloud Console.

2. Navigate to IAM & Admin - In the left sidebar, go to IAM & Admin > Service Accounts.

1. Create a New Service Account: Click on the "Create Service Account" button at the top.\
2. Fill in the Service Account Details:

   • Service account name: Give your service account a name.

   • Service account ID: This will be automatically generated based on the name.

   • Description: Optionally, provide a description for the service account.

   Click "Create and Continue".

   example:\
3. Assign Roles to the Service Account:

   • In the Grant this service account access to project step, you’ll assign the necessary roles.

   • Grant these role:

     • "storage object viewer" (read-only access)

4. Create the Service Account:

   • After assigning the roles, click "Done".

5. Generate and Download a Key:

   • Find your newly created service account, click the three dots on the right, and select "Manage Keys".

   • Click Add Key > Create New Key and choose the JSON format.

   • Download the key and store it securely, as it is used for authentication in your code or applications.

6. Encode the key in base 64:

   • use python function: put this function and your json (here named json_file.json) in the same directory and run.\

     Copy

     import json
     import base64
     
     
     def encode_json_to_base64(json_file):
         # Read JSON data from file
         with open(json_file, 'r') as file:
             json_data = json.load(file)
     
         # Convert the JSON data to a string
         json_str = json.dumps(json_data)
     
         # Encode the string to bytes, then to Base64
         json_bytes = json_str.encode('utf-8')
         base64_bytes = base64.b64encode(json_bytes)
     
         # Convert Base64 bytes back to a string
         base64_str = base64_bytes.decode('utf-8')
         # Print the Base64-encoded string
         print(base64_str)
     
     
     encode_json_to_base64('json_file.json')

   • save the output printed.

Add the storage provider to Emedgene platform:

• Add the above 3 values into the appropriate fields:

  • Client_credentials_base64: pasting the output of 8.

  • Bucket: the bucket name.

  • Path: for default, fill with / else, put your path in the bucket. Seperate directories with /

CORS - Visualisation

1. Download and install the Google Cloud SDK from the Google Cloud SDK Install page. LINK

2. Select Your Platform (Windows, macOS, or Linux), download and run.

3. Initialize and Authenticate with Google Cloud: In the Cloud SDK Shell/terminal, run: gcloud init This will open a browser window to authenticate your Google account. Follow the instructions to log in and select your project.

4. Set CORS Configuration via gcloud: Create a JSON file (cors.json) on your machine with the CORS rules. Example\ it should look like:

   Copy

   [
       {
         "origin": ["https://<host_name>.emg.illumina.com"],
         "method": ["GET"],
         "responseHeader": ["emgauthorization"],
         "maxAgeSeconds": 3600
       }
   ]

notice:

• origin: if using Illumina cloud:

  https://host_name.emg.illumina.com

  else, Emedgene cloud:

  https://host_name.emedgene.com

1. Apply CORS Configuration to Your Bucket: run the next command. gcloud storage buckets update gs://your-bucket-name --cors-file=cors.json

2. Verify the CORS Configuration: gcloud storage buckets describe gs://your-bucket-name

To directly import files from your storage, link storage to your organization in Emedgene.

How to create a link between storage and organization:

1. Click on the user initials or profile picture at the rightmost corner of the Top navigation panel to open the dropdown menu. Select Settings.

1. Select the Management tab. Under Storage is a list of currently linked storages. To add a new one press on Add Storage button.

1. Choose a Storage Type from:

   1. Azure Data Lake;

   2. Azure Blob;

   3. AWS S3;

   4. File Transport Protocol (FTP);

   5. Secure File Transport Protocol (SFTP);

   6. Illumina Basespace (BSSH);

   7. Illumina Connected Analytics (ICA).

1. Fill in the required credentials.

2. Click on Add storage:

1. Check the connection to confirm that the storage is successfully linked.

To do this, find the storage in the Storage List and check the cloud icon next to its name: 1. If it's green, the connection is set correctly; 2. If it's red and strikethrough, something went wrong. Hover over the icon to see details.

How to edit storage information:

In the Storage List press Manage on the right to the storage details.

How to remove a link to storage:

In the Storage List press Delete on the right to the storage details.

Scope

Bring Your Own Key (BYOK) is a security feature that allows clients to use their own encryption keys to protect their data. This ensures that clients maintain control over their encryption keys and, consequently, their data. Only Enterprise level support accounts can access this feature, and it requires assistance from Illumina support for setup.

Supported Key Management Services

Illumina supports integration with popular Key Management Services (KMS) such as Azure Key Vault and AWS KMS for managing your encryption keys. This integration allows clients to use their existing key management solutions for generating, storing, and managing their keys securely.

Azure Key Vault

is a cloud service that provides a secure way to store and manage sensitive information like API keys, passwords, and certificates. It offers robust features for key management, including key generation, storage, and lifecycle management.

AWS KMS

(KMS) allows you to create and control encryption keys used to encrypt your data across a wide range of AWS services and applications. It provides centralized management of encryption keys and integrates seamlessly with other AWS services.

These integrations ensure robust key management capabilities and enhance the security of your data through a combination of Illumina's BYOK feature and your preferred KMS provider.

Risk of Losing a Key

Losing the encryption key means that all data encrypted with that key will be inaccessible. This can lead to permanent loss of access to crucial information. It is imperative that clients securely store and manage their keys to prevent such risks.

Setup

Azure Key Vault Setup

Emedgene’s API server will encrypt the client’s information before storing in Emedgene’s database and decrypt that information when needed (e.g. running the pipeline). The key vault is managed by the customer. The customer needs to provide the following information.

Please see below instructions on how to get or create it

Application Tokens:

• Client Id

• Client Secret

• Tenant Id

The key information:

• Key URL

Create a new Application

1. Navigate to App registration

2. Register a new application, click “Register”

3. When you created the app, please copy Application (client) ID and Directory (tenant) ID

4. Go to Certificates and secrets (in the left menu)

5. Press “New client secret” and provide the “Value”

Create a new Key

1. Press New Key (Create key vault)

2. Specify key vault name, region (ie. East US) and pricing tier

3. Click “Next” to Access Policies

4. Press “Add access policy” and set Key permissions:

   1. Key Management Operations: -

   2. Cryptographic Operations: Decrypt, Encrypt, Unwrap Key, Wrap Key

5. Then set Secret permissions:

   1. Secret Permission: Get

   2. Select principal: select the application you created before (in Create a new Application step)

6. Finish with “Review + create”

Find key details

1. Navigate to the newly created Key vault

2. Select keys on the left side, select the key

3. Select the current version and copy “Key Identifier” https://<key-vault-name>.vault.azure.net/keys/<key-name>/<key-version>\

AWS Key Management Service (KMS) Setup

Description is coming soon.

Architecture

Emedgene’s API server will encrypt the client’s information before storing in Emedgene’s database and decrypt that information when needed (e.g. running the pipeline). The key vault is managed by the client, and Emedgene will only be provided with access to encrypt/decrypt functions in that key vault. This guarantees that the clients controls access to the information.

Illustration of data flow when creating a case in Emedgene platform:

Illustration of data flow when reading a case data from Emedgene platform:

A preliminary step to this solution is having a key vault owned by the client, and a key that Emedgene is given access to.

The client will create an access policy in the key vault of type “Application” and provide the matching key and secret to Emedgene. The access policy must contain permissions to perform encrypt and decrypt actions.

In order for Emedgene to integrate with the key, depending on the key vault provider, the client needs to provide the following information:

• Client Id

• Client Secret

• Tenant Id

• Key vault name

• Key name

Searching Encrypted Fields

Since some of our platform search capabilities run directly on the DB, we can’t directly search any data that is encrypted. To overcome this, we will implement a hashing search functionality as follows.

• The case data will still be fully encrypted in the DB as it is today

• Specific fields we want to make “searchable” - as defined by the customer, we will save their hash value alongside the encrypted data.

• Hashing will be done using SHA-256, and will include a secure random generated salt of 32 characters, which will be added to the value.

• The salt is unique and will not be used anywhere else in the platform.

• When the user enters a string to search, we will hash that value using all the salt values, and search those hash values.

Illustration of data flow when searching in Emedgene platform:

Illustration of data flow when creating a case with searchable field in Emedgene platform:

Appendix

\

Before you proceed to this article, make sure you understand data storage management basics.

Update Azure Blob Storage Credentials

In Settings > Management Tab, add or edit the required credentials: CLIENT_ID, CLIENT_SECRET, TENANT_ID, and ACCOUNT_URL.

See the table below to learn where to look for them in your Azure account.

Blob Integration Setup

Create an App registration

1. In Microsoft Entra ID, click on App registrations.

1. Select New registration.

2. Fill the name of the application & press "register."

3. You got to the registered app page: (CLIENT_ID / TENANT_ID) From this you can retrieve: Application ID and Tenant ID. Both are marked in the screenshot.

1. Press "Certificates & secrets"

2. Press on "New Client secret"

1. Fill the "Description" and change expires to 12 months. (or according to your organization policy), than press "Add"

8. Get the CLIENT_SECRET from this page.

1. Give this App registration roles and read access to the relevant Blob.

Azure Blob configuration

1. Go to Azure Storage accounts

1. Get into the relevant Storage account

1. Press on "containers"

1. Press on the relevant container

2. Press on "Properties"

3. Copy the ACCOUNT_URL\

For Internal support:

Errors for bad connections can be found in CloudWatch on particular FRY log stream

Search for: BlobApi, BlobFs, azure.