The platform provides Connectors to facilitate automation for operations on data (ie, upload, download, linking). The connectors are helpful when you want to sync data between ICA and your local computer or link data between projects in ICA.
The ICA CLI upload/download proves beneficial when handling large files/folders, especially in situations where you're operating on a remote server by connecting from your local computer. You can use icav2 projects enter <project-name/id> to set the project context for the CLI to use for the commands when relevant. If the project context is not set, you can supply the additional parameter --project-id <project-id> to specify the project for the command.
Upload Data
> icav2 projects list #note the project-name/id.
> icav2 projects enter <project-name/id> # set the project context
> icav2 projectdata upload <localFileFolder> <remote-path> # upload localFileFolder to remote-path
#Example:
> icav2 projects enter demo
> icav2 projectdata upload localFolder /uploads/
Download Data
> icav2 projectdata list # note the data-id
> icav2 projectdata download <data-id> # download the data.
Note: Because of how S3 manages storage, it doesn't have a concept of folders in the traditional sense. So, if you provide the "folder" ID of an empty "folder", you will not see anything downloaded.
ICA API
Another option to upload data to ICA is via ICA API. This option is helpful where data needs to be transferred via automated scripts. You can use the following two endpoints to upload a file to ICA.
Post - /api/projects/{projectId}/data with the following body which will create a partial file at the desired location and return a dataId for the file to be uploaded. {projectId} is the the project id for the destination project. You can find the projectId in yout projects details page (Project > Details > URN > urn:ilmn:ica:project:projectId#MyProject).
Create data in the project by making the API call below. If you don't already have the API-Key, refer to the instructions on the support page for guidance on generating one.
In the example above, we're generating a partial file named 'tempFile.txt' within a project identified by the project ID '41d3643a-5fd2-4ae3-b7cf-b89b892228be', situated inside a folder with the folder ID 'fol.579eda846f1b4f6e2d1e08db91408069'. You can access project, file, or folder IDs either by logging into the ICA web interface or through the use of the ICA CLI.
Retrieve the data/file ID from the response (for instance: fil.b13c782a67e24d364e0f08db9f537987) and employ the following format for the Post request - /api/projects/{projectId}/data/{dataId}:createUploadUrl:
Use the URL from the response to upload a file (tempFile.txt) as follows:
curl --upload-file tempFile.txt "url"
AWS CLI
ICA allows you to directly upload/download data from ICA using AWS CLI. It is especially helpful when dealing with an unstable internet connection to upload or download a large amount of data. If the transfer gets interrupted midway, you can employ the sync command to resume the transfer from the point it was stopped.
To connect to ICA storage, you must first download and install AWS CLI on your local system. You will need temporary credentials to configure AWS CLI to access ICA storage. You can generate temporary credentials through the ICA CLI, which can be used to authenticate AWS CLI against ICA. The temporary credentials can be obtained using this ICA API endpoint
Generate temporary credentials
> icav2 projectdata temporarycredentials --help
This command fetches temporal AWS and Rclone credentials for a given project-data. If path is given, project id from the flag --project-id is used. If flag not present project is taken from the context
Usage:
icav2 projectdata temporarycredentials [path or data Id] [flags]
Flags:
-h, --help help for temporarycredentials
--project-id string project ID to set current project context
Global Flags:
-t, --access-token string JWT used to call rest service
-o, --output-format string output format (default "table")
-s, --server-url string server url to direct commands
-k, --x-api-key string api key used to call rest service
Example cli to generate temporary credentials:
If you are trying to upload data to /cli-upload/ folder, you can get the temporary credentials to access the folder using icav2 projectdata temporarycredentials /cli-upload/. It will produce following output with accessKey, secretKey and sessionToken that you will need to configure AWS CLI to access this folder.
Copy the awsTempCredentials.accessKey, awsTempCredentials.secretKey and awsTempCredentials.sessionToken to build the credentials file: ~/.aws/credentials. It should look something like
The temporary credentials expire in 36 hours. If the temporary credentials expire before the copy is complete, you can use AWS sync command to resume from where it left off.
Following are a few AWS commands to demonstrate the use. The remote path in the commands below are constructed off of the output of temporarycredentials command in this format: s3://<awsTempCredentials.bucket>/<awsTempCredentials.objectPrefix>
Example AWS commands
#Copy single file to ICA
> aws s3 cp cp1 s3://stratus-gds-use1/53395234-6b20-4fb1-3587-08db9144d245/cli-upload/
#Sync local folder to ICA
> aws s3 sync cli-upload s3://stratus-gds-use1/53395234-6b20-4fb1-3587-08db9144d245/cli-upload
You can also write scripts to monitor the progress of your copy operation and regenerate and refresh the temporary credentials before they expire.
rclone
You may also use Rclone for data transfer if you prefer. The steps to generate temporary credentials is the same as above. You can run rclone config to set keys and tokens to configure rclone with the temporary credentials. You will need to select the advanced edit option when asked to enter the session key. After completing the config, your configure file (~/.config/rclone/rclone.conf) should look like this:
#Copy single file to ICA
> rclone copy file.txt s3-config:stratus-gds-use1/53395234-6b20-4fb1-3587-08db9144d245/cli-upload/
#Sync local folder to ICA
> rclone sync cli-upload s3-config:stratus-gds-use1/53395234-6b20-4fb1-3587-08db9144d245/cli-upload/