A storage configuration provides ICA with information to connect to an external cloud storage provider, such as AWS S3. The storage configuration validates that the information provided is correct, and then continuously monitors the integration.

Refer to the following pages for instructions to setup supported external cloud storage providers:

• Connect AWS S3 Bucket

Credentials

The storage configuration requires credentials to connect to your storage. AWS uses the security credentials to authenticate and authorize your requests. On the System Settings > Storage > Credentials tab > Create storage credential, you can enter these credentials. Long-term access keys consist of a combination of the access key ID and secret access key as a set.

Fill out the following fields:

• Type—The type of access credentials. This will usually be AWS user.

• Name—Provide a name to easily identify your access key.

• Access key ID—The access key you created.

• Secret access key—Your related secret access key.

For more information, refer to the AWS security credentials documentation.

Create a Storage Configuration

1. In the ICA main navigation, select System Settings > Storage > Configuration tab > New configuration.

2. Configure the following settings for the storage configuration.

   • Type—Use the default value, eg, AWS_S3. Do not change.

   • Region—Select the region where the bucket is located.

   • Configuration name—You will use this name when creating volumes that reside in the bucket. The name length must be in between 3 and 63 characters.

   • Description—Here you can provide a description for yourself or other users to identify this storage configuration.

   • Bucket name—Enter the name of your S3 bucket.

   • Key prefix [Optional]—You can provide a key prefix to allow only files inside the prefix to be accessible. The key prefix must end with "/".

   • If a key prefix is specified, your projects will only have access to that folder and subfolders. For example, using the key prefix folder-1/ ensures that only the data from the folder-1 directory in your S3 bucket is synced with your ICA project. Using prefixes and distinct folders for each ICA project is the recommended configuration as it allows you to use the same S3 bucket for different projects.

   • Using no key prefix results in syncing all data in your S3 bucket (starting from root level) with your ICA project. Your project will have access to your entire S3 bucket, which prevents that S3 bucket from being used for other ICA projects. Although possible, this configuration is not recommended.

   • Secret—Select the credentials to associate with this storage configuration. These were created on the Credentials tab.

   • Server Side Encryption [Optional]—If needed, you can enter the algorithm and key name for server-side encryption processes.

3. Select Save.

ICA performs a series of steps in the background to verify the connection to your bucket. This can take several minutes. You may need to manually refresh the list to verify that the bucket was successfully configured. Once the storage configuration setup is complete, the configuration can be used while creating a new project.

With the action Set as default for region, you select which storage will be used as default storage in a region for new projects of your tenant. Only one storage can be default at a time for a region, so selecting a new storage as default will unselect the previous default. If you do not want to have a default, you can select the default storage and the action will become Unset as default for region.

The System Settings > Storage > Credentials > Share storage credential action is used to make the storage available to everyone in your tenant. By default, storage is private per user so that you have complete control over the contents. Once you decide you want to share the storage, simply select it and use the Share storage credential action. Do take into account that once shared, you can not unshare the storage. Once your storage is used in a project, it can also no longer be deleted.

Storage Configuration Verification

Every 4 hours, ICA will verify the storage configuration and credentials to ensure availability. When an error is detected, ICA will attempt to reconnect once every 15 minutes. After 200 consecutively failed connection attempts (50 hours), ICA will stop trying to connect.

When you update your credentials, the storage configuration is automatically validated. In addition, you can manually trigger revalidation when ICA has stopped trying to connect by selecting the storage and then clicking Validate on the System Settings > Storage > Configuration tab.

Refer to this page for the troubleshooting guide.

Supported Storage Classes

ICA supports the following storage classes. Please see the AWS documentation for more information on each:

This section describes how to connect an AWS S3 Bucket with enabled. General instructions for configuring your AWS account to allow ICA to connect to an S3 bucket are found on .

Create an S3 bucket with SSE-KMS

Follow the for how to create S3 bucket with SSE-KMS key.

• Note: S3-SSE-KMS must be in the same region as your ICA v2.0 project. See the for more information.

In the "Default encryption" section, enable Server-side encryption and choose AWS Key Management Service key (SSE-KMS). Then select Choose your AWS KMS key.

• If you do not have an existing customer managed key, click Create a KMS key and follow from AWS.

Once the bucket is set, the user is recommended also to create a folder that will be connected to ICA as a prefix. If the user makes a new folder in the bucket that will be linked in the ICA storage configuration, the encryption must be enabled in AWS console.

Connect the S3-SSE-KMS to ICA

• Add permission to use KMS key by adding kms:Decrypt, kms:Encrypt, and kms:GenerateDataKey

• Add the ARN KMS key arn:aws:kms:xxx on the first "Resource"

• On Unversioned buckets, the permssions will match the following:
• On Versioned OR Suspended buckets, the permssions will match the following:

At the end of the policy setting, there should be 3 permissions listed in the "Summary".

Create the S3-SSE-KMS configuration in ICA

On step 3 in process above, continue with the [Optional] Server Side Encryption to enter the algorithm and key name for server-side encryption processes.

• On "Algorithm", input aws:kms

• On "Key Name", input the ARN KMS key: arn:aws:kms:xxx

• "Key prefix" is optional, but recommended. "Key prefix" refers to the folder name in the bucket the user previously created above.

Additional set up for Cross Account Copy for S3 buckets with SSE-KMS encryption

You can use your own S3 bucket with Illumina Connected Analytics (ICA) for data storage. This section describes how to configure your AWS account to allow ICA to connect to an S3 bucket.

These instructions utilize the AWS CLI. Follow the AWS CLI documentation for instructions to download and install.

Key points for connected AWS S3 buckets to ICA:

• The AWS S3 bucket must exist in the same AWS region as the ICA project. Refer to the table below for a mapping of ICA project regions to AWS regions:

*Note: BSSH is not deployed currently on the South Korea instance, therefore there will be limited functionality in this region with regard to sequencer integration.

• You can enable SSE using an Amazon S3-managed key (SSE-S3). Instructions for using KMS-managed (SSE-KMS) keys are found here.

Because of how Amazon S3 handles folders and does not send events for S3 folders, the following restrictions must be taken into account for ICA project data stored in S3.

• When creating an empty folder in S3, it will not be visible in ICA.

• When moving folders in S3, the original, but empty, folder will remain visible in ICA and must be manually deleted there.

• When deleting a folder and its contents in S3, the empty folder will remain visible in ICA and must be manually deleted there.

• Projects cannot be created with ./ as prefix since S3 does not allow uploading files with this key prefix.

• When configuring a new project in ICA to use a preconfigured S3 bucket, you can use the root folder if needed. However, this is not recommended as that S3 bucket is then no longer available for other ICA projects. Instead, please consider using subfolders in S3 for your projects.

❗️ For Bring Your Own Storage buckets, all unversioned, versioned and suspended buckets are supported. If you connect buckets with object versioning, the data in ICA will be automatically synced with the data in objectstore. For Bring Your Own Storage buckets with versioning enabled, when an object is deleted without specifying a particular version, a "Delete marker" is created on the objectstore to indicate that the object has been deleted. ICA will reflect the object state by deleting the record from the database. No further action on your side is needed to sync.

Configure Bucket CORS Permission

ICA requires cross-origin resource sharing (CORS) permissions to write to the S3 bucket for uploads via the browser. Refer to the Configuring cross-origin resource sharing (CORS) (expand the "Using the S3 console" section) documentation for instructions on enabling CORS via the AWS Management Console. Use the following configuration during the process:

• In the cross-origin resource sharing (CORS) section, enter the following content.

  Copy

  [
      {
          "AllowedHeaders": [
              "*"
          ],
          "AllowedMethods": [
              "HEAD",
              "GET",
              "PUT",
              "POST",
              "DELETE"
          ],
          "AllowedOrigins": [
              "https://ica.illumina.com"
          ],
          "ExposeHeaders": [
              "ETag",
              "x-amz-meta-custom-header"
          ]
      }
  ]

Create AWS IAM Policy

ICA requires specific permissions to access data in an AWS S3 bucket. These permissions are contained in an AWS IAM Policy.

Refer to the Creating policies on the JSON tab documentation for instructions on creating an AWS IAM Policy via the AWS Management Console. Use the following configuration during the process:

• On Unversioned buckets, paste the JSON policy document below. Replace YOUR_BUCKET_NAME with the actual name of your bucket. Note the example below provides access to all objects prefixes in the bucket.

  Copy

  {
      "Version": "2012-10-17",
      "Statement": [
          {
              "Effect": "Allow",
              "Action": [
                  "s3:PutBucketNotification",
                  "s3:ListBucket",
                  "s3:GetBucketNotification",
                  "s3:GetBucketLocation"
              ],
              "Resource": [
                  "arn:aws:s3:::YOUR_BUCKET_NAME"
              ]
          },
          {
              "Effect": "Allow",
              "Action": [
                  "s3:PutObject",
                  "s3:GetObject",
                  "s3:RestoreObject",
                  "s3:DeleteObject"
              ],
              "Resource": "arn:aws:s3:::YOUR_BUCKET_NAME/*"
          },
          {
              "Effect": "Allow",
              "Action": [
                  "sts:GetFederationToken"
              ],
              "Resource": [
                  "*"
              ]
          }
      ]
  }

• On Versioned OR Suspended buckets, paste the JSON policy document below. Replace YOUR_BUCKET_NAME with the actual name of your bucket. Note the example below provides access to all objects prefixes in the bucket.

  Copy

  {
      "Version": "2012-10-17",
      "Statement": [
          {
              "Effect": "Allow",
              "Action": [
                  "s3:PutBucketNotification",
                  "s3:ListBucket",
                  "s3:GetBucketNotification",
                  "s3:GetBucketLocation",
                  "s3:ListBucketVersions",
                  "s3:GetBucketVersioning"
              ],
              "Resource": [
                  "arn:aws:s3:::YOUR_BUCKET_NAME"
              ]
          },
          {
              "Effect": "Allow",
              "Action": [
                  "s3:PutObject",
                  "s3:GetObject",
                  "s3:RestoreObject",
                  "s3:DeleteObject",
                  "s3:DeleteObjectVersion",
                  "s3:GetObjectVersion"
              ],
              "Resource": "arn:aws:s3:::YOUR_BUCKET_NAME/*"
          },
          {
              "Effect": "Allow",
              "Action": [
                  "sts:GetFederationToken"
              ],
              "Resource": [
                  "*"
              ]
          }
      ]
  }

• (Optional) Set policy name to "illumina-ica-admin-policy"

To create the IAM Policy via the AWS CLI, create a local file named illumina-ica-admin-policy.json containing the policy content above and run the following command. Be sure the path to the policy document (--policy-document) leads to the path where you saved the file:

aws iam create-policy --policy-name illumina-ica-admin-policy --policy-document file://illumina-ica-admin-policy.json

Create AWS IAM User

An AWS IAM User is needed to create an Access Key for ICA to connect to the AWS S3 Bucket. The policy will be attached to the IAM user to grant the user the necessary permissions.

Refer to the Creating IAM users (console) documentation for instructions on creating an AWS IAM User via the AWS Management Console. Use the following configuration during the process:

• (optional) Set user name to "illumina_ica_admin"

• Select the Programmatic access option for the type of access

• Select Attach existing policies directly when setting the permissions, and choose the policy created in Create AWS IAM Policy

• (Optional) Retrieve the Access Key ID and Secret Access Key by choosing to Download .csv

To create the IAM user and attach the policy via the AWS CLI, enter the following command (AWS IAM users are global resources and do not require a region to be specified). This command creates an IAM user illumina_ica_admin, retrieves your AWS account number, and then attaches the policy to the user.

aws iam create-user --user-name illumina_ica_admin
ACCOUNT_ID=$(aws sts get-caller-identity --query Account --output text)
aws iam attach-user-policy --policy-arn arn:aws:iam::${ACCOUNT_ID}:policy/illumina-ica-admin-policy --user-name illumina_ica_admin

Create AWS Access Key

If the Access Key information was retrieved during the IAM user creation, skip this step.

Refer to the Managing access keys (console) AWS documentation for instructions on creating an AWS Access Key via the AWS Console. See the "To create, modify, or delete another IAM user's access keys (console)" sub-section.

Use the below command to create the Access Key for the illumina_ica_admin IAM user. Note the SecretAccessKey is sensitive and should be stored securely. The access key is only displayed when this command is executed and cannot be recovered. A new access key must be created if it is lost.

aws iam create-access-key --user-name illumina_ica_admin

    "AccessKey": {
        "UserName": "illumina_ica_admin",
        "AccessKeyId": "<access key id>",
        "Status": "Active",
        "SecretAccessKey": "<secret access key>",
        "CreateDate": "2020-10-22 09:42:24+00:00"
    }

The AccessKeyId and SecretAccessKey values will be provided to ICA in the next step.

S3 Bucket Policy

Connecting your S3 bucket to ICA does not require any additional bucket policies.

However, if a bucket policy is required for use cases beyond ICA, you need to ensure that the bucket policy supports the essential permissions needed by ICA without inadvertently restricting its functionality.

Here is one such example:

{
     "Version": "2012-10-17",
     "Statement": [
         {
             "Effect": "Deny",
             "Principal": {
                 "AWS": "*"
             },
             "Action": [
                 "s3:PutObject",
                 "s3:GetObject",
                 "s3:RestoreObject",
                 "s3:DeleteObject",
                 "s3:DeleteObjectVersion",
                 "s3:GetObjectVersion"
             ],
             "Resource": "arn:aws:s3:::YOUR_BUCKET_NAME/*",
             "Condition": {
                 "ArnNotLike": {
                     "aws:PrincipalArn": [
                         "arn:aws:iam::YOUR_ACCOUNT_ID:user/illumina_ica_admin",
                         "arn:aws:sts::YOUR_ACCOUNT_ID:federated-user/illumina_ica_admin"
                     ]
                 }
             }
         }
     ]
 }

In this example, we have a restriction enabled on the bucket policy to disallow any kind of access to the bucket. However, there is an exception rule added for the IAM user that ICA is using to connect to the S3 bucket. The exception rule is allowing ICA to perform the above S3 action permissions necessary for ICA functionalities.

Additionally, the exception rule is applied to the STS federated user session principal associated with ICA. Since ICA leverages the AWS STS to provide temporary credentials that allow users to perform actions on the S3 bucket, it is crucial to include these STS federated user session principals in your policy's whitelist. Failing to do so could result in 403 Forbidden errors when users attempt to interact with the bucket's objects using the provided temporary credentials.

Create ICA Storage Credential

To connect your S3 account to ICA, you need to add a storage credential in ICA containing the Access Key ID and Access Key created in the previous step. From the ICA home screen, navigate to System Settings > Storage > Credentials and click the +New button to create a new storage credential.

Provide a name for the storage credentials, ensure the type is set to "AWS user" and provide the Access Key ID and Secret Access Key.

With the secret credentials created, a storage configuration can be created using the secret credential. Refer to the instructions to Create a Storage Configuration for details.

Enabling Cross Account Access for Copy and Move Operations

ICA uses AssumeRole to copy and move objects from a bucket in an AWS account to another bucket in another AWS account. To allow cross account access to a bucket, the following policy statements must be added in the bucket policy:

    {
        "Version": "2012-10-17",
        "Statement": [
            {
                "Sid": "Allow cross account access",
                "Effect": "Allow",
                "Principal": {
                    "AWS": "ASSUME_ROLE_ARN"
                },
                "Action": [
                    "s3:PutObject",
                    "s3:DeleteObject",
                    "s3:ListMultipartUploadParts",
                    "s3:AbortMultipartUpload",
                    "s3:GetObject"
                ],
                "Resource": [
                    "arn:aws:s3:::YOUR_BUCKET_NAME",
                    "arn:aws:s3:::YOUR_BUCKET_NAME/*"
                ]
            }
        ]
    }

In the policy, replace the YOUR_BUCKET_NAME with the actual name of your bucket. The ARN of the cross account role you want to give permission to is specified in the Principal. Refer to the table below to determine which region-specific Role ARN should be used.

Storage Configuration Troubleshooting Guide

The following are common issues encountered when connecting an AWS S3 bucket through a storage configuration

Conflicting bucket notifications

This error occurs when an existing bucket notification's event information overlap with the notifications ICA is trying to add. Amazon S3 event notification only allows overlapping events with non-overlapping prefix. Depending on the conflicts on the notifications, the error can be presented in any of the following:

Volume Configuration cannot be provisioned: storage container is already set up for customer's own notification

Invalid parameters for volume configuration: found conflicting storage container notifications with overlapping prefixes

Failed to update bucket policy: Configurations overlap. Configurations on the same bucket cannot share a common event type

To fix the issue:

1. In the Amazon S3 Console, review your current S3 bucket's notification configuration and look for prefixes that overlaps with your Storage Configuration's key prefix

2. Delete the existing notification that overlaps with your Storage Configuration's key prefix

3. ICA will perform a series of steps in the background to re-verify the connection to your bucket.

GetTemporaryUploadCredentialsAsync failure

This error can occur when recreating a recently deleted storage configuration. To fix the issue, you have to delete the bucket notifications:

1. In the Amazon S3 Console select the bucket for which you need to delete the notifications from the list.

2. Choose properties

3. Navigate to the Event Notifications section and choose the check box for the event notifications with name gds:objectcreated, gds:objectremoved and gds:objectrestore and click Delete.

4. Wait 15 minutes for the storage to become available in ICA

If you do not want to wait 15 minutes, you can delete the current storage configuration, delete the bucket notifications in the bucket and create a new storage configuration.