The platform provides Connectors to facilitate automation for operations on data (ie, upload, download, linking). See the child pages for details.
The platform GUI provides the Project Connector utility which allows data to be linked automatically between projects. This creates a one-way dynamic link for files and samples from source to destination, meaning that additions and deletions of data in the source project also affect the destination project. This differs from copying or moving which create editable copies of the data. In the destination project, you can delete data which has been moved or copied and unlink data which has been linked.
| one-way | files | folders | erases source data | propagate source edits | editable on destination | |
|---|---|---|---|---|---|---|
Select the source project (project that will own the data to be linked) from the Projects page (Projects > your_source_project).
Select Project Settings > Details.
Select Edit
Under Data Sharing ensure the value is set to Yes
Select Save
Select the destination project (the project to which data from the source project will be linked) from the Projects page (Projects > your_destination_project).
From the projects menu, select Project Settings > Connectivity > Project Connector
Select + Create and complete the necessary fields.
Check the box next to Active to ensure the connector will be active.
Name (required) — Provide a unique name for the connector.
Type (required) — Select the data type that will be linked (either File or Sample)
Source Project - Select the source poject whose data will be linked to.
Filter Expression (optional) — Enter an expression to restrict which files will be linked via the connector (see Filter Expression Examples below)
Tags (optional) — Add tags to restrict what data will be linked via the connector. Any data in the source project with matching tags will be linked to the destination project.
The examples below will link Files based on the Format field.
Only Files with Format of FASTQ will be linked:
[?($.details.format.code == 'FASTQ')]
Only Files with Format of VCF will be linked:
[?($.details.format.code == 'VCF')]
The examples below will restrict linked Files based on a filenames.
Exact match to 'Sample-1_S1_L001_R1_001.fastq.gz':
[?($.details.name == 'Sample-1_S1_L001_R1_001.fastq.gz')]
Ends with '.fastq.gz':
[?($.details.name =~ /.*\.fastq.gz/)]
Starts with 'Sample-':
[?($.details.name =~ /Sample-.*/)]
Contains '_R1_':
[?($.details.name =~ /.*_R1_.*/)]
The examples below will link Samples based on User Tags and Sample name, respectively.
Only Samples with the User Tag 'WGS-Project-1'
[?('WGS-Project-1' in $.tags.userTags)]
Only Samples with the name 'BSSH_Sample_1':
Link a Sample with the name 'BSSH_Sample_1':
[?($.name == 'BSSH_Sample_1')]
move
x
x
x
x
x
copy
x
x
x
x
manual link
x
x
x
project connector
x
x
x
ICA provides a Service Connector, which is a small program that runs on your local machine to sync data between the platform's cloud-hosted data store and your local computer or server. The Service Connector securely uploads data or downloads results using TLS 1.2. In order to do this, the Connector makes 2 connections:
A control connection, which the Connector uses to get configuration information from the platform, and to update the platform about its activities
A connection towards the storage node, used to transfer the actual data between your local or network storage and your cloud-based ICA storage.
This Connector runs in the background, and configuration is done in the Illumina Connected Analytics (ICA) platform, where you can add upload and download rules to meet the requirements of the current project and any new projects you may create.
The Service Connector looks at any new files and checks their size. As long as the file size is changing, it knows data is still being added to the file and it is not ready for transfer. Only when the file size is stable and does not change anymore will it consider the file to be complete and initiate transfer. Despite this, it is still best practice to not connect the Service Connector to active folders which are used as streaming output for other processes as this can result in incomplete files being transferred when the active processes have extended compute periods in which the file size remains unchanged.
The service connector will handle integrity checking during file transfer, which requires the calculation of hashes on the data. In addition, Transmission speed depends on the available data transfer bandwidth and connection stability. For these reasons, uploading large amounts of data can take considerable time. This can in turn result in temporarily seeing empty folders at the destination location since these are created at the beginning of the transfer process.
Select Projects > your_project.
From the projects menu, select Project Settings > Connectivity.
Select + Request New.
Fill out the fields in the New Connector configuration page.
Unique name (required) — Enter the name of the connector.
Description (optional) — Enter the connector description.
Mode (required) — Specify if the connector can upload data, download data, both or neither.
Operating system (required) — Select your server or computer operating system.
Debug Information Accessible by Illumina (optional) - Illumina support can request connector debugging information to help diagnose issues. For security reasons, support can only collect this data if the option Debug Information Accessible by Illumina is active. You can choose to either proactively enable this when encountering issues to speed up diagnosis or to only activate it when support requests access. You can at any time revoke access again by deselecting the option.
Add any upload or download rules. See Connector Rules below.
Select Install and wait for the installer to download. An initialization key will be displayed in the platform now. Copy this value as it will be needed during installation.
Launch the installer after the download completes and follow the on-screen prompts to complete the installation, including entering the initialization key copied in the previous step. Do not install the connector in the upload folder as this will result in the connector attempting to upload itself and the associated log files.
Run the downloaded .exe file. During the installation, the installer will ask for the initialization key. Fill out the initialization key you see in the platform.
The installer will create an Illumina Service Connector, register it as a Windows service, and start the service. That means, if you wait for about 60 seconds, and then refresh the screen in the Platform by using the refresh button in the right top corner of the page, the connector should display as connected.
You can only install 1 connector on Windows. If for some reason, you need to install a new one, first uninstall the old one. You only need to do this when there is a problem with your existing connector. Upgrading a connector is also possible. To do this, you don’t need to uninstall the old one first.
Double click the downloaded .dmg file. Double click Illumina Service Connector in the window that opens to start the installer. Run through the installer, and fill out the initialization key when asked for it.
To start the connector once installed or after a reboot, open the app. You can find the app on the location where you installed it. The connector icon will appear in your dock when the app is running.
In the platform on the Connectivity page, you can check whether your local connector has been connected with the platform. This can take 60 seconds after you started your connector locally, and you may need to refresh the Connectivity page using the refresh button in the top right corner of the page to see the latest status of your connector.
The connector app needs to be closed to shut down your computer. You can do this from within your dock.
Installations require Java SE 8 or 11. You can check this with ‘java –version’ from a command line terminal. With Java installed, you can run the installer from the command line using the command bash illumina_unix_develop.sh.
Depending on whether you have an X server running or not, it will display a UI, or follow a command line installation procedure. You can force a command line installation by adding a –c flag: bash illumina_unix_develop.sh -c.
The connector can be started by running ./illuminaserviceconnector start from the directory in which the connector was installed.
In the upload and download rules, you define different properties when setting up a connector. A connector can be used by multiple projects and a connector can have multiple upload and download rules. Configuration can be changed anytime. Changes to the configuration will be applied approximately 60 seconds after changes are made in ICA if the connector is already connected. If the connector is not already started when configuration changes are made in ICA, it will take about 60 seconds after the connector is started for the configuration changes to be propagated to the connector. The following are the different properties you can configure when setting up a connector. After adding a rule and installing the connector, you can use the Active checkbox to disable rules.
Below is an example of a new connector setup with an Upload Rule to find all files ending with .tar or .tar.gz located within the local folder C:\Users\username\data\docker-images.
An upload rule tells the connector which folder on your local disk it needs to watch for new files to upload. The connector contacts the platform every minute to pick up changes to upload rules. To configure upload rules for different projects, first switch into the desired project and select Connectivity. Choose the connector from the list and select Click to add a new upload rule and define the rule. The project field will be automatically filled with the project you are currently within.
When you schedule downloads in the platform, you can choose which connector needs to download the files. That connector needs some way to know how and where it needs to download your files. That’s what a download rule is for. The connector contacts the platform every minute to pick up changes to download rules. The following are the different download rule settings.
Select the chevron at the middle-bottom of the rule configuration to expand the following optional advanced download rule settings.
When you set up your connector for the first time, and your sample files are located on a shared drive, it’s best to create a folder on your local disk, put one of the sample files in there, and do the connector setup with that folder. When this works, try to configure the shared drive.
Transfer to and from a shared drive may be quite slow. That means it can take up to 30 minutes after you configured a shared drive before uploads start. This is due to the integrity check the connector does for each file before it starts uploading. The connector can upload from or download to a shared drive, but there are a few conditions:
The drive needs to be mounted locally. X:\illuminaupload or /Volumes/shareddrive/illuminaupload will work, \\shareddrive\illuminaupload or smb://shareddrive/illuminaupload will not.
The user running the connector must have access to the shared drive without a password being requested.
The user who runs the Illumina Service Connector process on the Linux machine needs to have read, write and execute permissions on the installation folder.
Illumina might release new versions of the Service Connector, with improvements and/or bug fixes. You can easily download a new version of the Connector with the Download button on the Connectivity screen in the platform. After you downloaded the new installer, run it and choose the option ‘Yes, update the existing installation’.
To uninstall the connector, perform one of the following:
Windows and Linux: Run the uninstaller located in the directory the connector was installed.
Mac: Move the Illumina Service Connector to your Trash folder.
The Connector has a log file containing technical information about what’s happening. When something doesn’t work, it often contains clues to why it doesn’t work. Interpreting this log file is not always easy, but it can help the support team to give a fast answer on what is wrong, so it is suggested to attach it to your email when you have upload or download problems. You can find this log file at the following location:
\<Installation Directory>\logs\BSC.out
Default: C:\Program Files (x86)\illumina\logs\BSC.out
/<Installation Directory>/Illumina Service Connector.app/Contents/java/app/logs/BSC.out
Default: /Applications/Illumina Service Connector.app/Contents/java/app/logs/BSC.out
/<Installation Directory>/logs/BSC.out
Default: /usr/local/illumina
| Field | Description |
|---|---|
| Field | Description |
|---|---|
| Field | Description |
|---|---|
| Operating system | Issue | Solution |
|---|---|---|
| General issues | Solution |
|---|---|
Unique name
Name of the upload rule.
Local folder
The folder path on the local machine where files to be uploaded are stored.
File pattern
Files with filenames that match the string/pattern will be uploaded.
Location
The location the data will be uploaded to.
Project
The project the data will be uploaded to.
Description
Additional information about the upload rule.
Assign Format
Select which data format tag the uploaded files will receive. This is used for various things like filtering.
Data owner
The owner of the data after upload.
Order of execution
If using multiple download rules, set the order the rules are performed.
Unique name
Name of the download rule.
Target Local folder
The folder path on the local machine where the files will be downloaded to.
Description
Additional information about the download rule.
Project
The projects the rule applies to.
Format
The format the files must comply to in order to be scheduled as downloaded.
Filename Expression
Edit the file names of your downloaded files for pipelines with auto-download enabled.
Windows
Service connector doesn't connect
First, try restarting your computer. If that doesn’t help, open the Services application (By clicking the Windows icon, and then typing services). In there, there should be a service called Illumina Service Connector. • If it doesn’t have status Running, try starting it (right mouse click -> start) • If it has status Running, and still does not connect, you might have a corporate proxy. Proxy configuration is currently not supported for the connector. • If you do not have a corporate proxy, and your connector still doesn’t connect, contact Illumina Technical Support, and include your connector BSC.out log files.
OS X
Service connector doesn't connect
Check whether the Connector is running. If it is, there should be an Illumina icon in your Dock. • If it doesn’t, log out and log back in. An Illumina service connector icon should appear in your dock. • If it still doesn’t, try starting the Connector manually from the Launchpad menu. • If it has status Running, and still does not connect, you might have a corporate proxy. Proxy configuration is currently not supported for the connector. • If you do not have a corporate proxy, and your connector still doesn’t connect, contact Illumina Technical Support, and include your connector BSC.out log files.
Linux
Service connector doesn't connect
Check whether the connector process is running with:
ps aux
Linux
Can’t define java version for connector
The connector makes use of java version 8 or 11. If you run the installer and get the following error “Please define INSTALL4J_JAVA_HOME to point to a suitable JVM.”: • When downloading the correct java version from Oracle, there are 2 variables in the script that can be defined (INSTALL4J_JAVA_HOME_OVERRIDE & INSTALL4J_JAVA_PREFIX), but not INSTALL4J_JAVA_HOME, which is printed in the above error message. Instead, export the variable to your env before running the installation script. You can export the variable to your env before running the script, like this: • Note that Java home should not point to the java executable, but to the jre folder. For example: export INSTALL4J_JAVA_HOME_OVERRIDE=/usr/lib/jvm/java-1.8.0-openjdk-amd64 sh illumina _unix_1_13_2_0_35.sh
Linux
Corrupted installation script
If you get the following error message “gzip: sfx_archive.tar.gz: not in gzip format. I am sorry, but the installer file seems to be corrupted. If you downloaded that file please try it again. If you transfer that file with ftp please make sure that you are using binary mode.” : • This indicates the installation script file is corrupted. Editing the shell script will cause it to be corrupt. Please re-download the installation script from ICA.
Linux
Unsupported version error in log file
If the log file gives the following error "Unsupported major.minor version 52.0", an unsupported version of java is present. The connector makes use of java version 8 or 11.
Linux
Manage the connector via the CLI
• Connector installation issues:
It may be necessary to first make the connector installation script executable with:
chmod +x illumina_unix_develop.sh
Once it has been made executable, run the installation script with:
bash illumina_unix_develop.sh
It may be necessary to run with sudo depending on user permissions on the system:
sudo bash illumina_unix_develop.sh
If installing on a headless system, use the -c flag to do everythign from the command line:
bash illumina_unix_develop.sh -c
• Start connector with logging directly to the terminal stdout) (in case log file is not present, likely due to the absence of java version 8 or 11). From within the installation directory run:
./illuminaserviceconnector run
• Check status of connector. From within the install location run:
./illuminaserviceconnector status
• Stop the connector with:
./illuminaserviceconnector stop
• Restart the connector with:
./illuminaserviceconnector restart
Connector gets connected, but uploads won’t start
Create a new empty folder on your local disk, put a small file in there, and configure this folder as upload folder. • If it works, and your sample files are on a shared drive, have a look at the Shared Drives section. • If it works, and your sample files are on your local disk, there are a few possibilities: a) There is an error in how the upload folder name is configured in the platform. b) For big files, or on slow disks, the connector needs quite some time to start the transfer because it needs to calculate a hash to make sure there are no transfer errors. Wait up to 30 minutes, without changing anything to your Connector configuration. • If this doesn’t work, you might have a corporate proxy. Proxy configuration is currently not supported for the connector.
Upload from shared drive does not work
Follow the guidelines in Shared Drives section. Inspect the connector BSC.log file for any error messages regarding the directory not being found. • If there is such a message, there are two options: a) An issue with the the folder name, such as special characters and spaces. As a best practice, use only alphanumeric characters, underscores, dashes and periods. b) A permissions issue. In this case, ensure the user running the connector has read & write access, without a password being requested, to the network share. • If there are no messages indicating the directory cannot be found, it may be necessary to wait for some time until the integrity checks have been done. This check can take quite long on slow disks and slow networks.
Data Transfers are slow
Many factors can affect the speed: • Distance from upload location to storage location • Quality of the internet connection • Hardlines are preferred over WiFi • Restrictions for up- and download by the company or the provider. These factors can change every time the customer switches from location (e.g. working from home).
The upload or download progress % goes down instead of up.
This is normal behavior. Instead of one continuous transmission, data is split into blocks so that whenever transmission issues occur, not all data has to be retried. This does result in dropping back to a lower % of transmission completed when retrying.