> For the complete documentation index, see [llms.txt](https://help.connected.illumina.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://help.connected.illumina.com/emedgene/emedgene-analyze-manual/managing_data_storage/bring-your-own-bucket.md). # Bring Your Own S3 Bucket ## Connect an AWS S3 bucket as read-only input for Emedgene \ Use this guide to create an Amazon S3 bucket (or reuse a dedicated bucket) that Emedgene can list and read. Typical uses are importing FASTQ, BAM/CRAM, or VCF data, or pointing Emedgene at existing DRAGEN output for analysis. \\ \ This procedure is read only. Emedgene cannot create, overwrite, or delete objects in the bucket. \\ ### Before you start * An AWS account with permission to create buckets, IAM users, and policies. * An Emedgene organization that can add an S3 data storage (usually a Manager role). Confirm with if you are unsure. * A dedicated bucket is recommended. Emedgene should not share a bucket used by other applications. Not covered here | Goal | Use instead | | ------------------------------------------------------------------------------ | ------------------------------------------------------------------------------ | | Emedgene-managed DRAGEN writes output into your bucket (Bring Your Own Bucket) | BYOB setup with Illumina support (write permissions, often bucket-root access) | | DRAGEN through ICA | ICA / Illumina Connected Analytics documentation | | Download data from Emedgene-managed storage | Manage S3 credentials in Emedgene | ### What you will create Emedgene authenticates with an IAM user access key and secret. AWS allows the call only if both of the following allow it (when the user and bucket are in different accounts). In the same account, Allows add together, so a broad identity policy can bypass a narrow bucket policy. 1. IAM user (access key + secret for Emedgene). 2. Identity policy on that user (S3 list/get). 3. Bucket policy that names that user as `Principal`. 4. CORS so IGV in the browser can stream BAM/CRAM from S3. If the bucket uses SSE-KMS with a customer managed key, the same IAM user also needs KMS permissions on that key. ### 1. Create the bucket 1. In the AWS console, create a bucket in the Region where your data lives (or where you will upload it). 2. Leave Block Public Access enabled. 3. Do not use a public bucket policy or public ACLs. 4. Decide the object prefix Emedgene should use, for example: * entire bucket, or * a folder such as `DRAGEN_Output/` or `emedgene-input/`. If you restrict access to a prefix, you must register that prefix in Emedgene (for example `s3:///DRAGEN_Output/`). Listing the bucket root will fail. ### 2. Create an IAM user for Emedgene 1. Create an IAM user used only for Emedgene. Do not reuse a human admin user. 2. Enable Programmatic access and create an access key. 3. Store the access key ID and secret access key in a secrets manager. You will paste them into Emedgene storage settings. 4. Note the user ARN: `arn:aws:iam:::user/` Replace `` and `` in every policy below. ### 3. Attach an IAM identity policy to the user Identity policies have no `Principal`. Attach one of the following to ``. #### Entire bucket (read) Use this when Emedgene may list and read any object in the bucket, or when the bucket policy is prefix-scoped and the IAM user is in a different AWS account from the bucket (the bucket policy then limits the effective path). {% code overflow="wrap" %} ```json { "Version": "2012-10-17", "Statement": [ { "Sid": "EmedgeneListBucket", "Effect": "Allow", "Action": [ "s3:GetBucketLocation", "s3:ListBucket" ], "Resource": "arn:aws:s3:::" }, { "Sid": "EmedgeneGetObjects", "Effect": "Allow", "Action": [ "s3:GetObject", "s3:GetObjectVersion" ], "Resource": "arn:aws:s3:::/*" } ] } ``` {% endcode %} `s3:GetObjectVersion` is only required if the bucket has versioning enabled and Emedgene might request a specific version. You can omit it for unversioned buckets. #### Prefix only, same AWS account If the IAM user and the bucket are in the same account and you want to lock access to `DRAGEN_Output/`, the identity policy must be scoped as well. A bucket policy prefix condition does not reduce access when the identity policy already allows `/*`. {% code overflow="wrap" %} ```json { "Version": "2012-10-17", "Statement": [ { "Sid": "EmedgeneGetBucketLocation", "Effect": "Allow", "Action": "s3:GetBucketLocation", "Resource": "arn:aws:s3:::" }, { "Sid": "EmedgeneListPrefix", "Effect": "Allow", "Action": "s3:ListBucket", "Resource": "arn:aws:s3:::", "Condition": { "StringLike": { "s3:prefix": [ "DRAGEN_Output", "DRAGEN_Output/*" ] } } }, { "Sid": "EmedgeneGetPrefix", "Effect": "Allow", "Action": [ "s3:GetObject", "s3:GetObjectVersion" ], "Resource": "arn:aws:s3:::/DRAGEN_Output/*" } ] } ``` {% endcode %} Change `DRAGEN_Output` everywhere if you use a different folder. Keep both `DRAGEN_Output` and `DRAGEN_Output/*` so listing with and without a trailing slash can succeed. Do not attach `AmazonS3ReadOnlyAccess` (all buckets) to this user. ### 4. Add a bucket policy On the bucket, add a policy that names the same IAM user. #### Entire bucket (read) {% code overflow="wrap" %} ```json { "Version": "2012-10-17", "Statement": [ { "Sid": "EmedgeneBucketMetadata", "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam:::user/" }, "Action": [ "s3:GetBucketLocation", "s3:ListBucket" ], "Resource": "arn:aws:s3:::" }, { "Sid": "EmedgeneReadAllObjects", "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam:::user/" }, "Action": [ "s3:GetObject", "s3:GetObjectVersion" ], "Resource": "arn:aws:s3:::/*" } ] } ``` {% endcode %} } #### Prefix only (example: `DRAGEN_Output/`) `s3:GetObject` must use an object ARN (`...//DRAGEN_Output/*`). Using only `arn:aws:s3:::` does not grant object reads. Put `s3:GetBucketLocation` in its own statement. That API does not send `s3:prefix`; a prefix condition on the same statement would deny it. {% code overflow="wrap" %} ```json { "Version": "2012-10-17", "Statement": [ { "Sid": "EmedgeneGetBucketLocation", "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam:::user/" }, "Action": "s3:GetBucketLocation", "Resource": "arn:aws:s3:::" }, { "Sid": "EmedgeneListDragenOutput", "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam:::user/" }, "Action": "s3:ListBucket", "Resource": "arn:aws:s3:::", "Condition": { "StringLike": { "s3:prefix": [ "DRAGEN_Output", "DRAGEN_Output/*" ] } } }, { "Sid": "EmedgeneReadDragenOutput", "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam:::user/" }, "Action": [ "s3:GetObject", "s3:GetObjectVersion" ], "Resource": "arn:aws:s3:::/DRAGEN_Output/*" } ] } ``` {% endcode %} You do not need a second “input files” list path unless those files live under a different prefix. Add another `s3:prefix` value and matching `GetObject` resource for each extra folder. ### 5. Configure CORS (required for IGV) Emedgene visualizes alignments in IGV from the browser against your bucket. Enable CORS on the bucket. Use the https origins for your Emedgene environment. Confirm the exact hosts with Illumina support (they differ by region and tenant). Do not omit the scheme. Include HEAD as well as GET, and expose range-request headers so BAM/CRAM streaming works. Example (replace origins with the values support provides): {% code overflow="wrap" %} ``` [ { "AllowedHeaders": [ "*" ], "AllowedMethods": [ "GET", "HEAD" ], "AllowedOrigins": [ "https://.emg.illumina.com", "https://*.emg.illumina.com", "https://*.emedgene.com" ], "ExposeHeaders": [ "ETag", "Accept-Ranges", "Content-Range", "Content-Length", "Content-Type" ], "MaxAgeSeconds": 3000 } ] ``` {% endcode %} If S3 rejects a wildcard origin, list each full origin instead of `https://*.emg.illumina.com`. ### 6. SSE-KMS (only if the bucket uses a CMK) If objects are encrypted with a customer managed KMS key, attach a key policy (or an identity policy) that allows the same IAM user: {% code overflow="wrap" %} ``` { "Version": "2012-10-17", "Statement": [ { "Sid": "EmedgeneKmsRead", "Effect": "Allow", "Action": [ "kms:Decrypt", "kms:DescribeKey" ], "Resource": "arn:aws:kms:::key/" } ] } ``` {% endcode %} AWS managed `aws/s3` keys do not need this extra statement for the same-account IAM user. ### 7. Storage class and lifecycle Keep objects Emedgene must read in a Standard (or other immediately available) class. Lifecycle rules that transition to Glacier, Deep Archive, or some Instant Retrieval / Intelligent-Tiering archive tiers will cause: | Use | Typical failure | | ------------------------------------------ | -------------------------- | | FASTQ / BAM / CRAM / VCF as analysis input | Import or reanalysis fails | | BAM / CRAM for IGV | Visualization fails | Restore archived objects before using them in Emedgene, or exclude this prefix from archive rules. You own backup and retention. If objects are deleted or moved, registered Emedgene files will break. ### 8. Register the storage in Emedgene 1. In Emedgene, open data storage / S3 settings (name varies by version). 2. Enter the bucket name, Region, access key, and secret. 3. If you used a prefix policy, set the path to that prefix (for example `DRAGEN_Output/` or `s3:///DRAGEN_Output/`). 4. Save and test listing. Confirm a known file can be selected for a case. ### 9. Validate With the IAM user keys (or IAM Policy Simulator): * `GetBucketLocation` on `` succeeds. * `ListObjects` with prefix `DRAGEN_Output/` succeeds if you scoped that folder; list with an empty prefix fails when prefix-scoped. * `GetObject` on a file under the allowed prefix succeeds. * `GetObject` on a file outside the allowed prefix fails. * `PutObject` fails (read-only). * In Emedgene, open IGV on a BAM/CRAM in the bucket and confirm the track loads (CORS). ### Troubleshooting | Symptom | Likely cause | | ---------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------- | | Access Denied on list | Prefix condition does not match the path configured in Emedgene; or identity policy missing `s3:ListBucket` | | Can list but cannot download or visualize | `GetObject` resource is the bucket ARN without `/*`; or KMS decrypt missing | | Prefix bucket policy but user can still read other folders | Same-account identity policy grants `/*` — scope the identity policy | | IGV fails, CLI `get-object` works | CORS origin, method (HEAD), or `ExposeHeaders` | | Works in AWS console, fails in Emedgene | Console uses your admin user, not the Emedgene IAM user keys | For setup questions, contact . --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation by asking a question. Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters: ``` GET https://help.connected.illumina.com/emedgene/emedgene-analyze-manual/managing_data_storage/bring-your-own-bucket.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.