arrow-left

All pages
gitbookPowered by GitBook
1 of 6

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Configure Multiple Identical netPathPrefixSearch Values

It is possible to provide support for multiple, identical seqservice.netPathPrefixSearch property values. This is achieved by using the seqservice.eventFileDirsToUseForSearchReplacePair properties. These properties can be used to specify the mappings between search replace pairs and their associated event directories.

  • The value of the seqservice.eventFileDirsToUseForSearchReplacePair property is a comma-separated list of event file directory suffixes to be applied to the search/replace pair.

  • If the property is not supplied (or the value is left empty), the behavior is unchanged. In this case, the search/replace pair is treated as global and applied to all active event file directory suffixes.

circle-info

By default, BaseSpace Clarity LIMS does not specify the eventFileDirsToUseForSearchReplacePair property. If you want the search/replace pair to apply only to a subset of the event file directories, you must supply the property manually.

The following example uses a MiSeq installation to show several configuration options, with comments. Similar configuration may be applied for installations of other Illumina instrument integrations (eg, MiSeqDx, NextSeq).

Note that the last part of this example shows configuration that results in an error.

miseq.v8.seqservice.eventFileDirectory.1 '/mnt/MiSeq1/data/gls_events'
miseq.v8.seqservice.eventFileDirectory.2 '/mnt/MiSeq2/data/gls_events'
miseq.v8.seqservice.eventFileDirectorySuffixes '1,2'

miseq.v8.seqservice.netPathPrefixSearch.1 'Y:\'
miseq.v8.seqservice.netPathPrefixReplace.1 '/mnt/Miseq1/data/'
miseq.v8.seqservice.eventFileDirsToUseForSearchReplacePair.1 '1' // just use search/replace pair 1 in dir 1
 
miseq.v8.seqservice.netPathPrefixSearch.2 'Y:\'
miseq.v8.seqservice.netPathPrefixReplace.2 '/mnt/Miseq2/data/'
miseq.v8.seqservice.eventFileDirsToUseForSearchReplacePair.2 '2' // just use search/replace pair 2 in dir 2
 
miseq.v8.seqservice.netPathPrefixSearch.3 'A:\'
miseq.v8.seqservice.netPathPrefixReplace.3 '/mnt/Miseq2/data/'
miseq.v8.seqservice.eventFileDirsToUseForSearchReplacePair.3 '1,2' // use search/replace pair 3 in dir 1 AND 2
 
miseq.v8.seqservice.netPathPrefixSearch.4 'B:\'
miseq.v8.seqservice.netPathPrefixReplace.4 '/mnt/Miseq2/data/'
miseq.v8.seqservice.eventFileDirsToUseForSearchReplacePair.4 '' // global, use this search/replace pair in all dirs
 
miseq.v8.seqservice.netPathPrefixSearch.5 'C:\'
miseq.v8.seqservice.netPathPrefixReplace.5 '/mnt/Miseq2/data/' // no replace pair dir prop specified = global, use this search/replace pair in all dirs
 
miseq.v8.seqservice.netPathPrefixSearch.6 'Y:\'
miseq.v8.seqservice.netPathPrefixReplace.6 '/mnt/Miseq2/data/'
miseq.v8.seqservice.eventFileDirsToUseForSearchReplacePair.6 '1' // duplicate search value (Y:) for event file dir 1 - ERROR
 
miseq.v8.seqservice.netPathPrefixSearchReplaceSuffixes '1,2,3,4,5,6'

Terminology

hashtag
Custom Field/User-Defined Field

As of Clarity LIMS v5.0.0, the term user-defined field (UDF) has been replaced with custom field in the user interface. However, the API resource is still called UDF.

There are two types of custom fields:

  • Master step fields — Configured on master steps. Master step fields only apply to the following:

    • The master step on which the fields are configured.

    • The steps derived from those master steps.

  • Global fields — Configured on entities (eg, submitted sample, derived sample, measurement, etc.). Global fields apply to the entire Clarity LIMS system.

hashtag
Run Mode

  • Samplesheet run mode - Only the sample sheet is generated and stored on Clarity LIMS. Run/Analysis configuration is imported manually into the instrument via a sample sheet to start the run. The downstream secondary analysis is performed on-premise. In NextSeq 1000/2000, this mode is also known as the Local Mode.

  • Hybrid/Cloud run mode - The planned run is created on ICA based on the run/analysis data from the integration service. The planned run contains all the run and analysis configuration required to start the run on the instrument. The sample sheet is also generated and stored on Clarity LIMS for reference purposes. In Cloud run mode, the analysis is performed on the cloud whereas in Hybrid run mode, the analysis is performed on-premise.

Configure Support for Samples Having Duplicate Names with Different Indexes

To support planned run creation for samples having duplicate names with different indexes, updates are required on the library preparation workflow that these samples go through.

Follow these steps to update the output Naming Convention of the library preparation workflow.

  1. Select the Configuration tab, and then select the Lab Work tab.

  2. Under Master Steps, select the master step of the library preparation workflow (e.g., Library Prep Validation).

Local run mode - The planned run is created on IRM based on the run/analysis data from the integration service. The planned run contains all the run and analysis configuration required to start the run on the instrument. The sample sheet is also generated and stored on Clarity LIMS for reference purposes. The analysis is performed on-premise.

Samplesheet, hybrid, cloud

MiSeq i00 Series (on-prem)

Local

Clarity Integration

Supported Run Mode

Remarks

NextSeq 1000/2000 (cloud)

Samplesheet, hybrid, cloud

NextSeq 1000/2000 (file-based)

Samplesheet

NovaSeq X Series (cloud)

Samplesheet, hybrid, cloud

NovaSeq X Series (on-prem)

Local

MiSeq i00 Series (cloud)

Navigate to the Step Type pane and update Naming Convention to

{SubmittedSampleName}_{OutputItemLIMSID}

References

Illumina Instrument Sample Sheets

Compatibility:

  • Clarity LIMS v5.1 & later, with NGS Extensions v5.17 and later

Use the Template File Generator tool to generate custom sample sheets for upload to Illumina instruments.

The driver_file_generator script is invoked in an automation command line, typically configured on the step prior to the instrument run. By default, the automation is triggered by a button on the Record Details screen.

For details on the parameters used by the script, template file contents, and how to create custom template files, see the Template File Generator documentation.

MiSeq Control Software and bcl2fastq2-compatible sample sheet template files are available from the Clarity LIMS Support team.

hashtag
MiSeq Sample Sheet Generation

This section summarizes the default configuration for MiSeq sample sheet generation. A template file that is compatible with MiSeq Control Software (MCS) is attached to this page.

¹ The version reflected in the master step may be different depending on the integration version you are using.

Automation command line:

For a description of the parameters used by the driver_file_generator script, see the article.

In Clarity LIMS v5.1 and later, the automation is configured on the Automation configuration screen. By default, the automation is triggered by a manual button on the Record Details screen. The automation trigger settings are configured on the master step configuration screen.

hashtag
bcl2fastq2 Sample Sheet Generation

This section summarizes the default configuration for bcl2fastq2 sample sheet generation.

Two bcl2fastq2 software-compatible template files are attached to this page. If you would like reverse complement sequences to be included in your sample sheet, use the BCL2FASTQ_Reverse_Complement_Samplesheet.csv template.

circle-info

The bcl2fastq2 sample sheet may be used with multiple instruments. The default configuration for the Illumina NextSeq integration is shown below.

¹ The version reflected in the master step may be different depending on the integration version you are using.

Automation command line:

or a description of the parameters used by the driver_file_generator script, see the article.

In Clarity LIMS v5.1 and later, the automation is configured on the Automation configuration screen. By default, the automation is triggered by a manual button on the Record Details screen. The automation trigger settings are configured on the master step configuration screen.

hashtag
Custom File Generation

To generate a custom sample sheet, complete the following steps:

  1. Download the desired template. (See section below.)

  2. Customize the template file as required. (See documentation.)

  3. Upload the customized template file to the appropriate step automation.

For instructions on uploading template files to an automation, and adding the token to the command line, see the following articles:

  • Adding & Configuring Automations (Attaching template files to step automations document section) in the .

hashtag
Attachments

MiSeqSamplesheetv3.csv:

BCL2FASTQ_Samplesheet.csv:

BCL2FASTQ_Reverse_Complement_Samplesheet.csv:

A token for the template file is automatically added to the Tokens list.
  • In the automation command line, provide the file token as the -t (-templatePath) parameter value:

    At run time, the token is replaced with the absolute path of the template file.

  • Automation name

    Enabled on master step/process type

    Triggered by

    Generate MiSeq SampleSheet

    Denature and Dilute (MiSeq v1.0)¹

    Manual button on Record Details screen

    Automation name

    Configured on master step

    Triggered by

    Generate bcl2fastq2 NextSeq Samplesheet

    Denature and Dilute (NextSeq 500/550 v1.0)¹

    Manual button on Record Details screen

    Creating Template Files
    Creating Template Files
    Attachments
    Template File Generator
    Clarity LIMS (Clarity & LabLink Reference Guide) documentation
    Creating Template Files
    file-download
    6KB
    MiSeqSamplesheetv3.csv
    arrow-up-right-from-squareOpen
    file-download
    1KB
    BCL2FASTQ_Samplesheet.csv
    arrow-up-right-from-squareOpen
    file-download
    2KB
    BCL2FASTQ_Reverse_Complement_Samplesheet.csv
    arrow-up-right-from-squareOpen
    bash -l -c "/opt/gls/clarity/bin/java -jar /opt/gls/clarity/extensions/ngs-common/v5/EPP/DriverFileGenerator.jar -i {processURI:v2} -u {username} -p {password} \
    script:driver_file_generator \
    -t {file:myTemplate.csv} \
    -o {compoundOutputFileLuid1}.csv \
    -q true \
    -destLIMSID {compoundOutputFileLuid1} \
    -l {compoundOutputFileLuid2}"
    bash -l -c "/opt/gls/clarity/bin/java -jar /opt/gls/clarity/extensions/ngs-common/v5/EPP/DriverFileGenerator.jar -i {processURI:v2} -u {username} -p {password} \
    script:driver_file_generator \
    -t /opt/gls/clarity/extensions/conf/driverfiletemplates/MiSeqSamplesheet.csv \
    -o {compoundOutputFileLuid1}.csv \
    -q true \
    -destLIMSID {compoundOutputFileLuid1} \
    -l {compoundOutputFileLuid2}"
    bash -l -c "/opt/gls/clarity/bin/java -jar /opt/gls/clarity/extensions/ngs-common/v5/EPP/DriverFileGenerator.jar -i {processURI:v2} -u {username} -p {password} \
    script:driver_file_generator \
    -t /opt/gls/clarity/extensions/conf/driverfiletemplates/{udf:Samplesheet Template} \
    -o {compoundOutputFileLuid1}.csv \
    -q true \
    -destLIMSID {compoundOutputFileLuid1} \
    -l {compoundOutputFileLuid2}"

    Transfer Files using Rsync Between Hosted Clarity Instance and Customer File Server For File-Based Integrations

    In Clarity file-based integrations such as NextSeq 500/550, there may be a need for file transfer between hosted Clarity LIMS instance and Customer File Server.

    Examples of file transfer scenarios:

    • Transfer of sample sheet from Clarity LIMS instance to Customer File Server

    • Transfer of sequencing/analysis run data from Customer File Server to Clarity LIMS instance

    hashtag
    Rsync-based File Transfer

    Rsync can be used as an alternative to VPN for transferring files between Clarity LIMS instance and Customer File Server. Rsync is a command-line tool to copy and synchronize files and directories between two locations and leverages SSH network protocol through port 22.

    hashtag
    Prerequisites

    1. Same version of Rsync should be installed on both Clarity LIMS instance and Customer File Server.

      • Install Rsync by running either commands as root user:

      • Check the rsync version and protocol version:

    hashtag
    Command

    • Transfer file from customer file server to Clarity LIMS instance.

      {% hint style="danger" %} The destination directory to transfer the file must exist in the Clarity LIMS instance and the user must have write permissions to this directory. {% endhint %}

    • Transfer file from Clarity LIMS instance to customer file server.

      {% hint style="danger" %} The file to transfer must exist in the Clarity LIMS instance and the executing user must have read permissions to the source directory containing the file and the file itself. {% endhint %}

    Refer to for detailed information on rsync

    hashtag
    Example

    chevron-rightTransfer sample sheet from Clarity LIMS instance to customer file serverhashtag
    • transfer_samplesheet.sh:

    chevron-rightTransfer run data customer file server to Clarity LIMS instancehashtag
    • transfer_run_data.sh:

    transfer_samplesheet.sh:

    transfer_run_data.sh:

    hashtag
    Data Clean Up

    triangle-exclamation

    The Clarity LIMS instance is able to perform at the same efficiency up till storage is full. However, it is recommended to clean up files before storage capacity is full. Upon reaching maximum capacity, Clarity LIMS UI will not be functional and will throw an error, prompting user to approach their admin.

    chevron-rightClean up old files in the Clarity LIMS instancehashtag
    • cleanup_rundata.sh:

    chevron-rightDelete unwanted files in target folder during rsync file transferhashtag
    • The flag --delete can be added to the rsync command to delete any files in the target directory that is not in the source directory.

    • When unwanted files are cleaned up on the customer file server, Rsync will also clean up those files on the Clarity LIMS instance.

    cleanup_rundata.sh:

    hashtag
    Automated Bash Script

    Crontab can be used to automatically schedule bash scripts to run at a given set interval.

    Example

    1. Run the following command to open and edit the current cron jobs for the current user.

    2. Add the following lines to run the transfer scripts every 15 minutes.

    3. Add the following lines to run the clean up script every Sunday at midnight.

    triangle-exclamation

    Ensure that bash and script locations are correctly specified.

    circle-info

    For more information on crontab, refer to the official documentations:

    SSH (port 22) on the Clarity LIMS instance must be accessible from the Customer's instance where the rsync script will be run.
  • Requires appropriate permissions:

    • Read permissions to the source directory and files in the Clarity LIMS instance

    • Write permissions to the destination directory in the Clarity LIMS instance

    {% hint style="danger" %} It is recommended to use /opt/gls/clarity/customextensions directory. {% endhint %}

  • https://download.samba.org/pub/rsync/rsync.1arrow-up-right
    file-download
    489B
    transfer_samplesheet.sh
    arrow-up-right-from-squareOpen
    file-download
    772B
    transfer_run_data.sh
    arrow-up-right-from-squareOpen
    file-download
    316B
    cleanup_rundata.sh
    arrow-up-right-from-squareOpen
    https://man7.org/linux/man-pages/man5/crontab.5.htmlarrow-up-right
    apt install rsync
    yum install rsync
    rsync --version
    rsync {File To Transfer} {user}@{Clarity LIMS Instance Name}.claritylims.com:{Target Path}
    rsync {user}@{Clarity LIMS Instance Name}.claritylims.com:{File To Transfer} {Target Path}
    #!/bin/bash
    rsync -av --include='*/' --include='*.csv' "[email protected]:/path/to/samplesheet/" "/path/to/destination"
    #!/bin/bash
    pushd "/path/to/instrument/run/root/dir"
    find . -type f -mtime -1 -print0 | rsync -aRv --include='*/' --include='*.bin' --include='*.xml' --exclude='*' --files-from=- --from0 . "[email protected]:/path/to/destination/"
    popd
    #!/bin/bash
    find /path/to/destination -type f -mtime +7 -exec rm -f {} \;
    Rsync --delete ...
    crontab -e
    */15 * * * * /bin/bash /usr/local/bin/transfer_run_data.sh
    */15 * * * * /bin/bash /usr/local/bin/transfer_samplesheet.sh
    0 0 * * 0 /bin/bash /usr/local/bin/cleanup_rundata.sh