Connect AWS S3 Bucket

You can use your own S3 bucket (unversioned, versioned, versioning-suspended) with Illumina Connected Analytics (ICA) for data storage. This section describes how to configure your AWS account to allow ICA to connect to an S3 bucket.

Connect AWS S3 Bucket to ICA Project

Prerequisite

AWS CLI

These instructions utilize the AWS CLI. Follow the AWS CLI documentation for instructions to download and install.

Best Practices

Do not use the root folder of your S3 storage

When configuring a new project in ICA to use a preconfigured S3 bucket, create a folder on your S3 bucket in the AWS console. This folder will be connected to ICA as a prefix.

Failure to create a folder will result in the root folder of your S3 bucket being assigned which will block your S3 bucket from being used for other ICA projects with the error "Conflict while updating file/folder. Please try again later."

Service Control Policies & Resource Control Policies

When configuring cross-account access for Bring Your Own Bucket (BYOB), organisational policy layers Service Control Policies (SCPs) and Resource Control Policies (RCPs) can prevent access even when the bucket policy is valid.

For cross-account S3 requests, AWS evaluates permissions in the following order:

1. Source Account Service Control Policies determine which actions the principal is allowed to perform, regardless of the destination resource.

2. Destination Account Resource Control Policies determine what external principals can do on resources within that account and act as a control layer above the bucket policy.

3. Bucket Policy and Identity-Based Policies are evaluated only after both SCP and RCP checks pass. This means that an explicit Deny, or the absence of a required Allow, in either the SCP or RCP results in an immediate final denial and the bucket policy is not evaluated.

Configuration

You can use either IAM User or IAM Role for setting the permissions with IAM Role offering better security for connecting to your own S3 storage.

IAM User

IAM user uses long-term credentials to connect external systems to your S3 storage. These credentials (access_key_id and secret_access_key) have to be kept secure and should preferably be regularly rotated, which requires updating the keys in all systems that use these keys.

IAM Role

IAM roles do not use long-term credentials. Instead temporary (12 hours) security permissions are provided when external systems assume the role. A permission policy determines which actions are allowed and a trust policy determines who (which software) can assume the role. When ICA requests to assume the role, the trust policy is checked to see if ICA is allowed to assume the role and if allowed, short-lived credentials are provided so ICA can borrow the permissions for that role.

You can enable SSE using an Amazon S3-managed key (SSE-S3). Instructions for using KMS-managed (SSE-KMS) keys are found here.

Considerations

Synchronization

Because of how Amazon S3 handles folders and does not send events for S3 folders, the following restrictions must be taken into account for ICA project data stored in S3.

• When you create an empty folder in S3, it will not be visible in ICA.

• When you move folders in S3, the original, but empty, folder will remain visible in ICA and must be manually deleted from there.

• When you delete a folder and its contents in S3, the empty folder will remain visible in ICA and must be manually deleted in from there.

• You can not create a project with ./ as prefix since S3 does not allow uploading files with this key prefix.

S3 region

The AWS S3 bucket must exist in the same AWS region as the ICA project. See the table below for a mapping of ICA project regions to AWS regions:

ICA Project Region	AWS Region
Australia	ap-southeast-2
Canada	ca-central-1
Germany	eu-central-1
India	ap-south-1
Indonesia	ap-southeast-3
Israel	il-central-1
Japan	ap-northeast-1
Singapore	ap-southeast-1
South Korea*	ap-northeast-2
UK	eu-west-2
United Arab Emirates	me-central-1
United States	us-east-1

(*) BSSH is not currently deployed on the South Korea instance, resulting in limited functionality in this region with regard to sequencer integration.

Versioned S3 Buckets

You can use unversioned (only one copy of an object exists), versioned (writing creates new versions) and suspended (versioning paused) buckets as own S3 storage.

If you connect buckets with object versioning, the data in ICA will be automatically synced with the data in object store. When an object is deleted without specifying a particular version, a Delete marker is created on the objectstore to indicate that the object has been deleted. ICA will reflect the object state by deleting the record from the database. No further action on your side is needed to sync.