Transfer Files using Rsync Between Hosted Clarity Instance and Customer File Server For File-Based Integrations

In Clarity file-based integrations such as NextSeq 500/550, there may be a need for file transfer between hosted Clarity LIMS instance and Customer File Server.

Examples of file transfer scenarios:

• Transfer of sample sheet from Clarity LIMS instance to Customer File Server

• Transfer of sequencing/analysis run data from Customer File Server to Clarity LIMS instance

Rsync-based File Transfer

Rsync can be used as an alternative to VPN for transferring files between Clarity LIMS instance and Customer File Server. Rsync is a command-line tool to copy and synchronize files and directories between two locations and leverages SSH network protocol through port 22.

Prerequisites

1. Same version of Rsync should be installed on both Clarity LIMS instance and Customer File Server.

   • Install Rsync by running either commands as root user:

     apt install rsync
     yum install rsync

   • Check the rsync version and protocol version:

     rsync --version

2. SSH (port 22) on the Clarity LIMS instance must be accessible from the Customer's instance where the rsync script will be run.

3. Requires appropriate permissions:

   • Read permissions to the source directory and files in the Clarity LIMS instance

   • Write permissions to the destination directory in the Clarity LIMS instance

   {% hint style="danger" %} It is recommended to use /opt/gls/clarity/customextensions directory. {% endhint %}

Command

• Transfer file from customer file server to Clarity LIMS instance.

  {% hint style="danger" %} The destination directory to transfer the file must exist in the Clarity LIMS instance and the user must have write permissions to this directory. {% endhint %}

  rsync {File To Transfer} {user}@{Clarity LIMS Instance Name}.claritylims.com:{Target Path}

• Transfer file from Clarity LIMS instance to customer file server.

  {% hint style="danger" %} The file to transfer must exist in the Clarity LIMS instance and the executing user must have read permissions to the source directory containing the file and the file itself. {% endhint %}

  rsync {user}@{Clarity LIMS Instance Name}.claritylims.com:{File To Transfer} {Target Path}

Refer to https://download.samba.org/pub/rsync/rsync.1 for detailed information on rsync

Example

Transfer sample sheet from Clarity LIMS instance to customer file server

• transfer_samplesheet.sh:

  #!/bin/bash
  rsync -av --include='*/' --include='*.csv' "[email protected]:/path/to/samplesheet/" "/path/to/destination"

Transfer run data customer file server to Clarity LIMS instance

• transfer_run_data.sh:

  #!/bin/bash
  pushd "/path/to/instrument/run/root/dir"
  find . -type f -mtime -1 -print0 | rsync -aRv --include='*/' --include='*.bin' --include='*.xml' --exclude='*' --files-from=- --from0 . "[email protected]:/path/to/destination/"
  popd

transfer_samplesheet.sh:

489B

transfer_samplesheet.sh

Open

transfer_run_data.sh:

772B

transfer_run_data.sh

Open

Data Clean Up

The Clarity LIMS instance is able to perform at the same efficiency up till storage is full. However, it is recommended to clean up files before storage capacity is full. Upon reaching maximum capacity, Clarity LIMS UI will not be functional and will throw an error, prompting user to approach their admin.

Clean up old files in the Clarity LIMS instance

• cleanup_rundata.sh:

  #!/bin/bash
  find /path/to/destination -type f -mtime +7 -exec rm -f {} \;

Delete unwanted files in target folder during rsync file transfer

• The flag --delete can be added to the rsync command to delete any files in the target directory that is not in the source directory.

  Rsync --delete ...

• When unwanted files are cleaned up on the customer file server, Rsync will also clean up those files on the Clarity LIMS instance.

cleanup_rundata.sh:

316B

cleanup_rundata.sh

Open

Automated Bash Script

Crontab can be used to automatically schedule bash scripts to run at a given set interval.

Example

1. Run the following command to open and edit the current cron jobs for the current user.

   crontab -e

2. Add the following lines to run the transfer scripts every 15 minutes.

   */15 * * * * /bin/bash /usr/local/bin/transfer_run_data.sh
   */15 * * * * /bin/bash /usr/local/bin/transfer_samplesheet.sh

3. Add the following lines to run the clean up script every Sunday at midnight.

   0 0 * * 0 /bin/bash /usr/local/bin/cleanup_rundata.sh

Ensure that bash and script locations are correctly specified.

For more information on crontab, refer to the official documentations: https://man7.org/linux/man-pages/man5/crontab.5.html