This section describes how to connect an AWS S3 Bucket with SSE-KMS Encryption enabled. General instructions for configuring your AWS account to allow ICA to connect to an S3 bucket are found on this page.
Create an S3 bucket with SSE-KMS
Follow the AWS instructions for how to create S3 bucket with SSE-KMS key.
S3-SSE-KMS must be in the same region as your ICA v2.0 project. See the for more information.
In the "Default encryption" section, enable Server-side encryption and choose AWS Key Management Service key (SSE-KMS). Then select Choose your AWS KMS key.
If you do not have an existing customer managed key, click Create a KMS key and follow from AWS.
Once the bucket is set, create a folder with encryption enabled in the bucket that will be linked in the ICA storage configuration. This folder will be connected to ICA as a . Although it is technically possible to use the root folder, this is not recommended as it will cause the S3 bucket to no longer be available for other projects.
Connect the S3-SSE-KMS to ICA
Follow the for connecting an S3 bucket to ICA.
In the step :
Add permission to use KMS key by adding kms:Decrypt, kms:Encrypt, and kms:GenerateDataKey
Add the ARN KMS key arn:aws:kms:xxx on the first "Resource"
At the end of the policy setting, there should be 3 permissions listed in the "Summary".
Create the S3-SSE-KMS configuration in ICA
Follow the for how to create a storage configuration in ICA.
On step 3 in process above, continue with the [Optional] Server Side Encryption to enter the algorithm and key name for server-side encryption processes.
On "Algorithm", input aws:kms
On "Key Name", input the ARN KMS key: arn:aws:kms:xxx
Although "Key prefix" is optional, it is highly recommended to use this and not use the root folder of your S3 bucket. "Key prefix" refers to the folder name in the bucket which you created.
Cross-Account Copy Setup for S3 buckets with SSE-KMS encryption
KMS Policy
In addition to following the instructions to , the KMS policy must include the following statement for AWS S3 Bucket with SSE-KMS Encryption (refer to the Role ARN table from the linked page for the ASSUME_ROLE_ARN value):
Depending on the bucket type (Unversioned, Versioned or Suspended) the permissions must match the following.
A storage configuration provides ICA with information to connect to an external cloud storage provider, such as AWS S3. The storage configuration validates that the information provided is correct, and then continuously monitors the integration.
Refer to the following pages for instructions to setup supported external cloud storage providers:
The storage configuration requires credentials to connect to your storage. AWS uses the security credentials to authenticate and authorize your requests. On the System Settings > Credentials > Create > Storage Credential, you can enter these credentials. Long-term access keys consist of a combination of the access key ID and secret access key as a set.
Fill out the following fields:
Type—The type of access credentials. This will usually be AWS user.
Name—Provide a name to easily identify your access key.
Access key ID—The access key you created.
You can share the credentials you own with other users of your tenant. To do so select your credentials at System Settings > Credentials and choose Share.
For more information, refer to the documentation.
Create a Storage Configuration
In the ICA main navigation, select System Settings > Storage > Create.
Configure the following settings for the storage configuration.
Type—Use the default value, eg, AWS_S3. Do not change.
ICA performs a series of steps in the background to verify the connection to your bucket. This can take several minutes. You may need to manually refresh the list to verify that the bucket was successfully configured. Once the storage configuration setup is complete, the configuration can be used while .
With the action Manage > Set as default for region, you select which storage will be used as default storage in a region for new projects of your tenant. Only one storage can be default at a time for a region, so selecting a new storage as default will unselect the previous default. If you do not want to have a default, you can select the default storage and the action will become Unset as default for region.
The SystemSettings > Credentials > select your credentials > Manage > Share action is used to make the storage available to everyone in your tenant. By default, storage is private per user so that you have complete control over the contents. Once you decide you want to share the storage, simply select it and use the Share action. Do take into account that once shared, you can not unshare the storage. Once your shared storage is used in a project, it can also no longer be deleted.
Filenames beginning with / are not allowed, so be careful when entering full path names. Otherwise the file will end up on S3 but not be visible in ICA. If this happens, access your S3 storage directly and copy the data to where it was intended. If you are using an Illumina-managed S3 storage, submit a support request to delete the erroneous data.
Deleting Storage Configurations
In the ICA main navigation, select System Settings > Storage > select your storage > Manage > Delete. You can then create a new storage configuration to reuse the bucket name and key prefix.
will also unlock your storage configuration so that it can be reused for another project. Data stored by the hidden project will remain in your S3 storage, so you may need to perform manual cleanup before reusing the storage.
Storage Configuration Verification
Every 4 hours, ICA will verify the storage configuration and credentials to ensure availability. When an error is detected, ICA will attempt to reconnect once every 15 minutes. After 200 consecutively failed connection attempts (50 hours), ICA will stop trying to connect.
When you update your credentials, the storage configuration is automatically validated. In addition, you can manually trigger revalidation when ICA has stopped trying to connect by selecting the storage and then clicking Validate on the System Settings > Storage > select your storage > Manage > Validate.
Refer to this for the troubleshooting guide.
Supported Storage Classes
ICA supports the following storage classes. Please see the for more information on each:
Object Class
ICA Status
If you are using , which allows S3 to automatically move files into different cost-effective storage tiers, please do NOT include the Archive and Deep Archive Access tiers, as these are not supported by ICA yet. Instead, you can use lifecycle rules to automatically move files to Archive after 90 days and Deep Archive after 180 days. Lifecycle rules are supported for user-managed buckets.
Secret access key—Your related secret access key.
Region—Select the region where the bucket is located.
Configuration name—You will use this name when creating volumes that reside in the bucket. The name length must be in between 3 and 63 characters.
Description—Here you can provide a description for yourself or other users to identify this storage configuration.
Bucket name—Enter the name of your S3 bucket.
Key prefix —You can provide a key prefix to allow only files inside the prefix to be accessible. Although this setting is optional, it is highly recommended to use a key prefix and mandatory when usingdedicated folders in your S3 storage. The key prefix must end with "/".
If a key prefix is specified, your projects will only have access to that folder and subfolders. For example, using the key prefix folder-1/ ensures that only the data from the folder-1 folder in your S3 bucket is synced with your ICA project. Using prefixes and distinct folders for each ICA project is the recommended configuration as it allows you to use the same S3 bucket for different projects.
Using no key prefix (not recommended) results in syncing all data in your S3 bucket (starting from root level) with your ICA project. Your project will have access to your entire S3 bucket, which prevents that S3 bucket from being used for other ICA projects.
Secret—Select the credentials to associate with this storage configuration. These were created on the Credentials tab.
Server Side Encryption [Optional]—If needed, you can enter the algorithm and key name for server-side encryption processes.
You can use your own S3 bucket with Illumina Connected Analytics (ICA) for data storage. This section describes how to configure your AWS account to allow ICA to connect to an S3 bucket.
These instructions utilize the AWS CLI. Follow the AWS CLI documentation for instructions to download and install.
When configuring a new project in ICA to use a preconfigured S3 bucket, create a folder on your S3 bucket in the AWS console. This folder will be connected to ICA as a prefix.
Failure to create a folder will result in the root folder of your S3 bucket being assigned which will block your S3 bucket from being used for other ICA projects with the error "Conflict while updating file/folder. Please try again later."
Because of how and does not send events for S3 folders, the following restrictions must be taken into account for ICA project data stored in S3.
When creating an empty folder in S3, it will not be visible in ICA.
Prerequisites
The AWS S3 bucket must exist in the same AWS region as the ICA project. Refer to the table below for a mapping of ICA project regions to AWS regions:
ICA Project Region
AWS Region
(*) BSSH is not currently deployed on the South Korea instance, resulting in limited functionality in this region with regard to sequencer integration.
You can use unversioned, versioned and suspendedbuckets as own S3 storage.
If you connect buckets with object versioning, the data in ICA will be automatically synced with the data in objectstore. When an object is deleted without specifying a particular version, a Delete marker is created on the objectstore to indicate that the object has been deleted. ICA will reflect the object state by deleting the record from the database. No further action on your side is needed to sync.
You can enable SSE using an Amazon S3-managed key (SSE-S3). Instructions for using KMS-managed (SSE-KMS) keys are found .
Configuration
1 - Configure Bucket CORS Permission
ICA requires cross-origin resource sharing (CORS) permissions to write to the S3 bucket for uploads via the browser. Refer to (expand the "Using the S3 console" section) documentation for instructions on enabling CORS via the AWS Management Console.
In the cross-origin resource sharing (CORS) section, enter the following content.
2 - Create Data Access Permission - AWS IAM Policy
ICA requires specific permissions to access data in an AWS S3 bucket. These permissions are contained in an AWS IAM Policy.
Permissions
Refer to the documentation for instructions on creating an AWS IAM Policy via the AWS Management Console. Use the following configuration during the process:
paste the JSON policy document below. Note the example below provides access to all objects prefixes in the bucket.
Replace YOUR_BUCKET_NAME with the name of the S3 bucket you created for ICA. Replace YOUR_FOLDER_NAME with the name of the folder in your S3 bucket.
On Versioned OR Suspended buckets, paste the JSON policy document below. Note the example below provides access to all objects prefixes in the bucket.
(Optional) Set policy name to "illumina-ica-admin-policy"
To create the IAM Policy via the AWS CLI, create a local file named illumina-ica-admin-policy.json containing the policy content above and run the following command. Be sure the path to the policy document (--policy-document) leads to the path where you saved the file:
3 - Create AWS IAM User
An AWS IAM User is needed to create an Access Key for ICA to connect to the AWS S3 Bucket. The policy will be attached to the IAM user to grant the user the necessary permissions.
Refer to the documentation for instructions on creating an AWS IAM User via the AWS Management Console. Use the following configuration during the process:
(optional) Set user name to "illumina_ica_admin"
Select the Programmatic access option for the type of access
Select Attach existing policies directly when setting the permissions, and choose the policy created in
To create the IAM user and attach the policy via the AWS CLI, enter the following command (AWS IAM users are global resources and do not require a region to be specified). This command creates an IAM user illumina_ica_admin, retrieves your AWS account number, and then attaches the policy to the user.
4. -Create AWS Access Key
If the Access Key information was retrieved during the , skip this step.
Refer to the AWS documentation for instructions on creating an AWS Access Key via the AWS Console. See the "To create, modify, or delete another IAM user's access keys (console)" sub-section.
Use the command below to create the Access Key for the illumina_ica_admin IAM user. Note the SecretAccessKey is sensitive and should be stored securely. The access key is only displayed when this command is executed and cannot be recovered. A new access key must be created if it is lost.
The AccessKeyId and SecretAccessKey values will be provided to ICA in the next step.
5 - S3 Bucket Policy
Connecting your S3 bucket to ICA does not require any additional bucket policies.
What if you need a bucket policy for use cases beyond ICA?
The bucket policy must then support the essential permissions needed by ICA without inadvertently restricting its functionality.
Be sure to replace the following fields:
6 - Block Public Access to S3 bucket (optional)
By default, public access to the S3 bucket is allowed. For increased security, it is advised to block public access with the following command:
To block public access to S3 buckets on account level, you can use the AWS Console on the website.
7 - Create ICA Storage Credential
To connect your S3 account to ICA, you need to add a storage credential in ICA containing the Access Key ID and Access Key created in the previous step. From the ICA home screen, navigate to System Settings > Credentials > Create > Storage Credential to create a new storage credential.
Provide a name for the storage credentials, ensure the type is set to "AWS user" and provide the Access Key ID and Secret Access Key.
With the secret credentials created, a storage configuration can be created using the secret credential. Refer to the instructions to for details.
The key prefix is mandatory in your storage credentials if you created a folder as recommended in step 2 .
8 - Enabling Cross Account Access for Copy and Move Operations
ICA uses AssumeRole to copy and move objects from a bucket in an AWS account to another bucket in another AWS account. To allow cross account access to a bucket, the following policy statements must be added in the S3 bucket policy:
Be sure to replace the following fields:
ASSUME_ROLE_ARN: Replace this field with the ARN of the cross account role you want to give permission to. Refer to the table below to determine which region-specific Role ARN should be used.
The ARN of the cross account role you want to give permission to is specified in the Principal. Refer to the table below to determine which region-specific Role ARN should be used.
Region
Role ARN
Troubleshooting
Common Issues
The following are common issues encountered when connecting an AWS S3 bucket through a storage configuration
Error Type
Error Message
Description/Fix
Conflicting bucket notifications
This error occurs when an existing bucket notification's event information overlaps with the notifications ICA is trying to add. only allows overlapping events with non-overlapping prefix. Depending on the conflicts on the notifications, the error can be presented in any of the following:
Volume Configuration cannot be provisioned: storage container is already set up for customer's own notification
Invalid parameters for volume configuration: found conflicting storage container notifications with overlapping prefixes
Failed to update bucket policy: Configurations overlap. Configurations on the same bucket cannot share a common event type
Solution:
In the Amazon S3 Console, review your current S3 bucket's notification configuration and look for prefixes that overlap with your Storage Configuration's key prefix
Delete the existing notification that overlaps with your Storage Configuration's key prefix
ICA will perform a series of steps in the background to re-verify the connection to your bucket.
GetTemporaryUploadCredentialsAsync failure
This error can occur when recreating a recently deleted storage configuration.
To fix the issue, you have to delete the bucket notifications:
In the select the bucket for which you need to delete the notifications from the list.
Choose properties
Navigate to the Event Notifications section and choose the check box for the event notifications with name gds:objectcreated, gds:objectremoved and gds:objectrestore and click Delete.
If you do not want to wait 15 minutes, you can revalidate the current storage configuration for an immediate update on the System Settings > Storage > Manage > Validate.
When moving folders in S3, the original, but empty, folder will remain visible in ICA and must be manually deleted there.
When deleting a folder and its contents in S3, the empty folder will remain visible in ICA and must be manually deleted there.
Projects cannot be created with ./ as prefix since S3 does not allow uploading files with this key prefix.
Japan
ap-northeast-1
Singapore
ap-southeast-1
South Korea*
ap-northeast-2
UK
eu-west-2
United Arab Emirates
me-central-1
United States
us-east-1
Replace YOUR_BUCKET_NAME with the name of the S3 bucket you created for ICA. Replace YOUR_FOLDER_NAME with the name of the folder in your S3 bucket.
(Optional) Retrieve the Access Key ID and Secret Access Key by choosing to Download .csv
YOUR_BUCKET_NAME: Replace this field with the name of the S3 bucket you created for ICA.
YOUR_ACCOUNT_ID: Replace this field with your account ID number.
YOUR_IAM_USER: Replace this field with the name of your IAM user created for ICA.
In this example, restriction is enabled on the bucket policy to prevent any kind of access to the bucket. However, there is an exceptionrule added for the IAM user that ICA is using to connect to the S3 bucket. The exception rule is allowing ICA to perform the above S3 action permissions necessary for ICA functionalities.
Additionally, the exception rule is applied to the STS federated user session principal associated with ICA. Since ICA leverages the AWS STS to provide temporary credentials that allow users to perform actions on the S3 bucket, it is crucial to include these STS federated user session principals in your policy's whitelist. Failing to do so could result in 403 Forbidden errors when users attempt to interact with the bucket's objects using the provided temporary credentials.
YOUR_BUCKET_NAME: Replace this field with the name of the S3 bucket you created for ICA.
arn:aws:iam::079623148045:role/ica_apn1_crossacct
Singapore (SG)
arn:aws:iam::079623148045:role/ica_aps1_crossacct
South Korea (KR)
arn:aws:iam::079623148045:role/ica_apn2_crossacct
UK (GB)
arn:aws:iam::079623148045:role/ica_euw2_crossacct
United Arab Emirates (AE)
arn:aws:iam::079623148045:role/ica_mec1_crossacct
United States (US)
arn:aws:iam::079623148045:role/ica_use1_crossacct
Conflict
Found conflicting storage container notifications for {prefix}{eventTypeMsg}
See
Conflict
Found conflicting storage container notifications with overlapping prefixes{prefixMsg}{eventTypeMsg}
See
Customer Container Notification Exists
Volume Configuration cannot be provisioned: storage container is already set up for customer's own notification
See
Invalid Access Key ID
Failed to update bucket policy: The AWS Access Key Id you provided does not exist in our records.
Check the status of the AWS Access Key ID in the console. If not active, activate it. If missing, create it.
Invalid Paramater
Missing credentials for storage container
Check the storage credential. AccessKeyId and/or SecretAccessKey is not set.
Invalid Parameter
Missing bucket name for storage container
Bucket name has not been set for the storage configuration.
Invalid Parameter
The storage container name has invalid characters
Storage container name can only contain lowercase letters, numbers, hyphens, and periods.
Invalid Parameter
Storage Container '{storageContainer}' does not exist
Update storage configuration container to a valid s3 bucket.
Invalid Parameter
Invalid parameters for volume configuration: {message}
Invalid Storage Container Location
Storage container must be located in the {region} region
Update storage configuration region to match storage container region.
Invalid Storage Container Location
Storage container must be located in one of the following regions: {regions}
Update storage configuration region to match storage container region.
Missing Configuration
Missing queue name for storage container notification
Missing Configuration
Missing system topic name for storage container notification
Missing Configuration
Missing lambda ARN for storage container notification
Missing Configuration
Missing subscription name for storage container notification
Missing Storage Account Settings
The storage account '{storageAccountName}' needs HNS (Hierarchical Namespace) enabled.
Missing Storage Container Settings
Missing settings for storage container
Wait 15 minutes for the storage to become available in ICA
Australia
ap-southeast-2
Canada
ca-central-1
Germany
eu-central-1
India
ap-south-1
Indonesia
ap-southeast-3
Israel
il-central-1
Australia (AU)
arn:aws:iam::079623148045:role/ica_aps2_crossacct
Canada (CA)
arn:aws:iam::079623148045:role/ica_cac1_crossacct
Germany (EU)
arn:aws:iam::079623148045:role/ica_euc1_crossacct
India (IN)
arn:aws:iam::079623148045:role/ica_aps3_crossacct
Indonesia (ID)
arn:aws:iam::079623148045:role/ica_aps4_crossacct
Israel (IL)
arn:aws:iam::079623148045:role/ica_ilc1_crossacct
Access Forbidden
Access forbidden: {message}
Mostly occurs because of lack of permission. Fix: Review IAM policy, Bucket policy, ACLs for required permissions
Unsupported principal
Unsupported principal: The policy type ${policy_type} does not support the Principal element. Remove the Principal element.
