About the Application Examples

The Clarity LIMS Application Examples use example scripts to help you learn how to work with REST and EPP (automation) scripts. Application examples are larger scripts that teach you how to do something useful. The purpose of these examples is to get you scripting quickly.

Be aware that the application examples differ from purchased Clarity LIMS integrations:

• Purchased LIMS integrations cannot be modified. Because Illumina guarantees specific functionality when using integrations, any code or script modifications must only be performed by the Illumina Integrations or Support teams.

• Application examples are learning tools that are intended to be modified. The application examples follow best practices in terms of over-all script construction; however, their exact functions are less critical than the integrations.

The best way to get started is to download the example script and try it out. Once you have seen how the script works, you can dissect it and use the pieces to create your own script.

The latest glsapiutil and glsapiutil3 Python libraries can be found on the GitHub page. Download links to these files will be removed from example documentation over time.

Additional reference information for Clarity LIMS.

This topic forms a natural partner to the Starting a Protocol Step via the API application example. When protocol steps are being initiated programmatically, we must know how to advance the step through the various states to completion.

Solution

Advancing a step is actually quite a simple task. It requires the use of the steps/advance API endpoint - in fact, little else is needed.

Let us consider a partially completed step with ID 24-1234. To advance the step to the next state, the following is required:

1. Perform a GET to the resource .../api/v2/steps/24-1234, saving the XML response.

2. POST the XML from step 1 to .../api/v2/steps/24-1234/advance, and monitor the returned XML for success.

3. If successful, the protocol step advances to its next state, just as if the lab scientist had advanced it via the Clarity LIMS interface.

The Python advanceStep (STEP_URI) method shown below advances a step through its various states. To achieve this, the URI of the step is passed to the method to be advanced/completed.

Assumptions and Notes

• The glsapiutil.py file is placed on the Clarity LIMS server, in the /opt/gls/clarity/customextensions folder. You will find the latest glsapiutil (and glsapiutil3) Python libraries on our .

• The example code is provided for illustrative purposes only. It does not contain sufficient exception handling for use 'as is' in a production environment.

• Illumina LIMS Integration

• Generating a Hierarchical Sample History

• Protocol-based Permissions

• Validating Process/Step Level UDFs

• Checking That Containers Are Named Appropriately

• Checking for Index Clashes Based on Index Sequence

• Repurposing a Process to Upload Indexes

• Adding Users in Bulk

• Moving Reagent Kits & Lots to New Clarity LIMS Server

Some steps produce data that you would like your collaborators to have access to.

This example provides an alternative method and uses a script to publish the files programmatically via the API.

Solution

In this example, suppose we have a protocol step, based upon a Sanger/capillary sequencing workflow, that produces up to two files per sample (a .seq and a .ab1 file).

An artifact is defined as something that is produced by a process or protocol step. It is the output of a process/protocol step run in Clarity LIMS. For example, analytes and result files are all artifacts.

Currently, the REST API does not support the following artifacts:

• Reagents—While reagents are produced by some processes in the Clarity LIMS client, they are not exposed as artifacts in the REST API. Reagents are an entity type of their own available through the list and single reagents resources.

• Subartifacts

There are often cases where empty containers are received and added into Clarity LIMS before being used in a protocol. This application example describes how to use the API to place samples into existing containers automatically. The application uses a CSV file that describes the mapping between the sample and its destination container.

Furthermore, the API allows accessioning into multiple container categories, something that is not possible through the web interface.

Prerequisites

• If you use Python version 2.6x, you must install the argparse package. Python 2.7 and later include this package by default.

• Also make sure that you have the latest glsapiutil.py Python API library on the Clarity LIMS server, in the /opt/gls/clarity/customextensions folder. You can download the latest glsapiutil library from our .

• Check the list of allowed containers for the step and make sure that all expected container categories are present. The API cannot place samples into containers that are not allowed for the step!

Structure of the Input File

The suggested input format is a four-column CSV with the following columns:

Sample Name, Container Category, Container Name, Well Position

The sample name should match the name as shown in the Ice Bucket/Queue screen.

Step Setup Screen Configuration

First, make sure that the Step Setup screen has been activated and is able to accept a file for upload:

EPP / Automation command line

Assuming the file is `compoundOutputFileLuid0`, the EPP / automation command line would be structured as follows:

The automation should be configured to trigger automatically when the Placement screen is entered.

Clarity LIMS v6.x automation trigger configuration

The Script

NOTE: The attached Python script uses the prerelease API endpoint (instead of v2), which allows placement of samples into existing containers.

The script performs the following operations:

1. Parses the file and create an internal map (Python dict) between sample name and container details:

   • Key: sample name

   • Value: (container name, well position) tuple

After the script runs, the Placement screen should show the placements, assuming there were no problems executing the script.

The attached script also contains some minimal bulletproofing for the following cases:

• Container was not found.

• Container is not empty.

• Well position is invalid.

• Sample in the ice bucket does not have a corresponding entry in the uploaded file.

In all cases, the script reports an error and does not allow the user to proceed.

Attachments

placeSamplesIntoExistingContainers.py:

Compatibility: API version 2

The Clarity LIMSweb interface provides an example Sample Import Excel file which can be manually uploaded to submit new samples to a selected project within the LIMS.

This application example shows how the Sample Import Sheet can be uploaded programatically with just one change to its format.

Solution

This example uses the same Sample Sheet with an additional column 'Project/Name'. The script processes the Sample Sheet, creates the samples and adds the samples to their respective projects.

This script leverages a python module xlrd, which is not included in the standard python library. It is used to extract data from .xls and .xlsx excel files.

Parameters

The script accepts the following parameters:

An example of the full syntax to invoke the script is as follows:

About the code

parseFile

This method carries out several operations:

• Opens the excel file and reads the text

• Stores the column headers in a dictionary variable called COLS

• Stores the row data accessibly in an array variable called ROWS

createProject

This method in turn carries out several operations:

• For each project name the script encounters it searches the LIMS to identify if a project with this name has already been created.

• If the project does not exist, the script will create the project. This example script is easily modifiable, however as written:

processRows

This method prepares the data needed to create a sample in LIMS:

• Assembles the UDF values, the project ID and container ID.

• For each non-tube container the script encounters it searches the LIMS to identify if a container with this name already exists.

• If the container does not exist the script will create the container.

The script contains additional supporting methods to generate XML which is POSTED to the API.

Assumptions & notes

• The UDFs in the Sample Sheet header have been configured in LIMS prior to submission.

• The following column headers are required: Sample/Name, Project/Name and any sample-level UDFs that are mandatory within your system.

• The script need not be run on the Clarity server, however it must have a connection to the Clarity LIMS API.

Attachments

_auth_tokens.py:

ClaritySampleSheetprojects.xlsx:

SampleSheetImporter.py:

Clarity LIMS can create Illumina-based MiSeq and HiSeq 'flavoured' sample sheets. However, if you are using algorithms or indexes outside of those suggested by Illumina, you may be required to produce your own 'custom' sample sheet.

This example script provides an algorithm that harvests the contents of a flow cell (or any container) that may contain pooled samples, and uses the resulting information to output a custom sample sheet.

Solution

The attached script uses aggressive caching in order to execute as quickly as possible. When extreme levels of multiplexing are involved, the cache size could consume considerable quantities of memory, which may be counter-productive.

The algorithm has been tested on the following unpooled analytes; pooled analytes; and 'pools of pools' — in which multiple homogeneous or heterogeneous pools are themselves combined to produce a new pool.

In these tests, the algorithm behaved as expected. If you find this is not the case please contact Illumina Support team.

1. The algorithm uses recursion to determine the individual analytes (samples) and their indexes that are located on the flow cell lane(s).

2. To determine whether an analyte constitutes a pool or not, the script looks at the number of submitted samples with which the analyte is associated.

   • If the answer is 1, the analyte is not a pool.

Parameters

The script is invoked with just three parameters:

An example of the full syntax to invoke the script is as follows:

Assumptions and Notes

• Both of the attached files are placed on the Clarity LIMS server, in the /opt/gls/clarity/customextensions folder.

• You will need to implement your own logic to gather the fields required for your specific sample sheet.

• You will need to update the HOSTNAME global variable such that it points to your Clarity LIMS server.

Attachments

flowcellContents.py:

The Container database table includes the stateid column. Unlike the states of artifacts, container stateids can be easily deciphered. (In Clarity LIMS, the stateids correspond to the states displayed in the BaseSpace Clarity LIMS Operations Interface).

As there is no table in the database that defines these stateid values, you will need to hard code the values in your queries based on the information provided in the following table:

*Since the concept of arrays does not apply to the BaseSpace Clarity LIMS Web Interface, only states 1-5 apply.

Example 1

To search for discarded and depleted containers, use the following example:

Example 2

To see all the arrays in the system that are new and still need to be scanned, use the following example:

Some of the more common LIMS ID prefixes are listed in the following table.

NOTE: QC steps do not have a special LIMS ID prefix.

In addition, there are also some esoteric and deprecated LIMS ID prefixes. The complete list of prefixes can be found in the classindex database table.

Translating a Step in the Web Interface to a Step URI in the API

Assuming you have a started a step, your browser address will look something like this:

https://demo.claritylims.com/clarity/work-placement/ 21816

The important piece is the suffix - 21816.

To convert to a step URI:

1. Note the type of step you are running. Is it a Standard step, an Indexing (Add Labels) step, or a Pooling step?

2. Add the corresponding hyphenated prefix using the previous list, to construct the step ID. For example, if it is an Indexing step, 21816 becomes 151-21816.

3. Build the step URI: https://hostname/api/version/steps/151-21816.

Translating a Step URI in the API to a Step in the Web Interface

To perform the reverse action and find a step in the web interface for a given a step URI (or a process URI):

1. Get the step ID. Using the example above, the step ID would be 151-21816.

2. Strip out the hyphenated prefix to give: '21816'.

3. Build the web address: https://hostname/clarity/work/21816.

Clarity LIMS will translate the word 'work' to the correct address for the screen the step is currently on. For example, if the step is completed, 'work' will become 'work-complete'.

When a lab takes in samples, they are named by the lab scientist who supplies them. As such, samples may be named in any way imaginable. Conversely, the research facilities processing the samples often have strict naming conventions.

When these two situations occur, there is a need for the sample to renamed with the strict nomenclature of the processing lab. However, in order for the final data to be meaningful for the scientist who supplied the samples, the original name must also be retained.

Solution

In this example, the attached script may be used to rename samples, while retaining their original names in a separate field.

• The original name of the sample is saved in a user-defined field (UDF) called Customer's Sample Name, .

• The Sample Name for the submitted sample is overwritten, using the specific naming convention of the customer.

It is recommended that the script be launched by a process/protocol step as early as possible in the sample lifecycle.

Parameters

The script is invoked with just three parameters:

An example of the full syntax to invoke the script is as follows:

About the Code

Once the command-line parameters have been harvested, and the API object set up ready to handle API requests, the renameSamples() method is called.

This method implements the following pseudo code:

Assumptions and Notes

• Both of the attached files are placed on the Clarity LIMS server, in the /opt/gls/clarity/customextensions folder.

• A user defined field, named Customer's Sample Name has been created at the analyte (sample) level.

• The Customer's Sample Name field is visible within the LabLink Collaborations Interface.

Attachments

renameSamples.py:

Often, workflows do not have a linear configuration. Even in those cases, samples progressing through workflows may be re-queued, replicated, or submitted into a number of parallel workflows.

Attempts to align downstream results with submitted samples may get hindered when trying to account for sample replicates or the dynamic decisions made in the lab.

A visual representation of a samples complete history presented in a clear hierarchical format allows for a digestible report of the work done on the sample. This format provides at-a-glance understanding of the any of the branching or re-queuing of a sample.

Solution

This example describes a python script which, given an artifact, recursively finds all the processes for which that artifact was an input, then finds all the associated output artifacts of that process. This continues for all processes all the way down to the most downstream artifact.

A clear visual representation of the entire history of work on a sample, similar to what was available in the Ops interface, can allow a user to see all the processes and derivations of a sample. This is especially of use for troublesome samples that have branched into numerous downstream replicates which may end up in the same or different sequencing protocol.

Parameters

The script accepts the following parameters:

An example of the full syntax to invoke the script is as follows:

Script output example:

Sibling artifacts will appear aligned vertically with the same indentation. In the above example, 122-1650 Library Pooling (MiSeq) 5.0 created two replicate Analytes, 2-4110 and 2-4111. Analyte 2-4111 was the input to the subsequent step ( 24-1952 ) and no additional work was performed on 2-4110.

Processes performed on an artifact will appear underneath with a tab indentation. In the above example, the first 4 processes ( 3 QC processes and Fragment DNA ) are all using the Root Analyte (CRA201A1PA1) as an input.

Adding Colours with termcolor

Install package termcolor for colour printing support. Entity colours can be configured within the script. Globally turn off colours by changing the variable use_colours to False ( line 16 ).

Assumptions and Notes

• Your configuration conforms with the script's requirements, as documented in .

• You are running a version of Python that is supported by Clarity LIMS, as documented in the Clarity LIMS Technical Requirements.

• The glsapiutil.py file is placed in the working directory.

Attachments

sample_history_colours.py:

sample_history.py:

To help those who prefer learning at their own pace, we have created video materials that supplement our API training materials. These videos correlate directly to the materials covered in the slides for each module, and in most cases we have also provided videos that illustrate the solution to the exercises.

These videos refer to resource definitions and the API Cookbook:

• Resource Definitions can be accessed through the API Portal.

• The cookbook is a set of example scripts intended to help you learn how to work with REST and EPP (automation) scripts.

Module 1: Overview of the API

This module outlines the course structure of the API Training Series of modules and provides a general strategy for being successful using the API.

Module 2: General Concepts of the API

This module explores the basic principles of interacting with the API. A browser plugin such as XHR for Chrome or Poster for Firefox is used and illustrates where specific LIMS data are accessible via the API. The PUT, POST, GET,and DELETE actions are explained as to how these operations are supported through the API.

Understanding the concept of a genealogy and how user defined fields are configured and used in the system is uncovered. Completing the exercises for this module is critical before proceeding to Module 3.

Module 2: Example Exercise 1

Module 2: Example Exercise 2

Module 2: Example Exercise 3

Module 3: Groovy

This module explores setting up Groovy and interacting with data in the Clarity LIMS database via the API. Common script examples from the Cookbook section are used to show how to use a scripting language to interact with the API. API example videos are also provided as part of module 3.

Module 3: Example Exercise 1

Module 3: Example Exercise 2

Module 4: Process Inputs and Outputs

This module focuses on how the data for processes that have been carried out in the LIMS are represented in XML format.

Module 4: Example Exercises

When processing samples, there can be circumstances in which you must add downstream samples to additional workflows. This sample addition is not easy to achieve using the Server Name interfaces, but is easy to do via the API.

Solution

This example provides a Python script that can be used to add samples to an additional workflow step. The example also includes information on the key API interactions involved.

Samples progressing through workflows can branch off and must be directed to different workflows or stages within a workflow.

Example: If it is not known at the initiation of a workflow if a sample is to be sequenced on a HiSeq or MiSeq. Rerouting the derived samples could be necessary.

Solution

This example provides the user with the opportunity to route samples individually to the HiSeq, MiSeq, or both stages from the Record Details screen.

Laboratories may want to limit what steps Researchers can start. At the time of writing, BaseSpace Clarity LIMS does not natively support protocol-based permissions. However, with an EPP at the beginning of the step we can check to see if the technician/researcher starting the step has been given approval to start the step, and halt the step from starting if they do not have permission. There are several ways this can be done, but special considerations to how these permissions are administered need to be made.

Solution

In order to allow an administrator to easily maintain permissions we will assign users to groups in a config file and our EPP will consume this information. One parameter of the EPP is the groups that are permitted to run the step. When the script is triggered at the start of the step, it will look for the name of the technician starting the step in the config file and determine if the technician is:

Sequential numbers are sometimes needed for naming conventions and require self-incrementing counters be created and maintained. We do not recommend using the BaseSpace Clarity LIMSdatabase for this. However the Unix “(n)dbm” library provides an easy way to create and manage counters by creating Dbm objects that behave like mappings (dictionaries).

Solution

They way this would work is the attached script ( and the counters file it creates / manages ) would live on the Clarity server and other scripts would depend upon it, and use code similar to below whenever a sequential number was needed. While the script is written in python and uses the dbm module there is nothing inherently Pythonic about this code that couldn’t be reimplemented in another language. However, more information on the Python dbm module can be found at: https://docs.python.org/2/library/dbm.html

While not a common scenario, it is possible to have two molecular barcodes/indices in BaseSpace Clarity LIMS that share the same sequence.

Within Clarity LIMS, there is logic that can be enabled to prevent pooling of samples with the same index name. However, this check is based upon the index name, not the sequence. In the case of identical indexes with differing names, additional logic is required to prevent index clashes based upon the index sequence.

This example provides a solution that looks for index clashes based upon sequence.

Solution

Compatibility: API version 2

Uploading indexes (reagent types) into BaseSpace Clarity LIMScan be done 3 different ways:

1. Manually adding the indexes one at a time using the Operations interface.

2. Uploading an XML data file using the config slicer.

Stakeholders are interested in the progress of samples as they move through a workflow. E-mail alerts of events can provide them with real-time notifications.

Some possible uses of notifications include the following:

• Completion of a workflow for billing department

• Manager review requests

Currently, the REST API does not support the following process types:

• Batch Gel Image Analysis Run (known at Gel Analysis Experiment in the client)

• Batch Gel Run (multiple 1D Gel Run or 2D Gel Run processes run at the same time)

Workflows do not always have a linear configuration. There are situations when samples progressing through a workflow must branch off and be directed to different stages, or even different workflows. You can add samples to a queue using the /api/{version}/route/artifacts endpoint.

In the lab, decisions are made dynamically. At the initiation of this example workflow, it is not known whether the sample is destined to be sequenced on a HiSeq or MiSeq instrument. As a result, the derived samples must be routed to a different workflow stage midworkflow.

This programmatic approach to queuing samples can be used many times throughout a workflow. This approach eliminates the need to write multiple complex scripts - each of which must be maintained over time.

This example describes a Python script that takes instruction from a .csv template file. Removing any hard-coded reference to specific UDFs / custom fields or workflow stages from the script allows for easy configuration and support. All business logic can be implemented solely through generation of a template and EPP / automation command.

The template contains UDF / custom field names and values. If samples in the active step have fields with matching values, they are queued for the specified workflow stage.

Step Configuration

The step is configured to display the two checkbox analyte UDFs / derived sample custom fields. These fields are used to select the destination workflow stages for each derived sample / analyte. You can queue the sample for HiSeq, MiSeq, or both.

In the preceding example of the Sample Details screen, the user selected:

• Two samples to be queued for HiSeq

• Two samples for MiSeq

• Two samples where routing is not selected

In the preceding example of the Step Details screen, the user selected:

• A step level UDF / custom field to assign all samples to a given step. All output samples are routed to the associated step. Samples can be duplicated in the first step of the following protocol if:

• Samples are routed as the last step of a protocol and;

• The action of the next step is Mark Protocol as Complete.

The duplication is due to the Next Steps action queuing the artifact for its default destination in addition to the script routing the same artifact.

The solution here is to set the default Next Steps action to Remove from Workflow instead. This action can be automated using the Lab Logic Toolkit or the API.

LIMS v6.x

On the protocol configuration screen, make sure that Start Next Step is set to Automatic.

Template Configuration

Each UDF name and value pair correspond to a workflow and stage combination as the destination to route the artifact.

• The UDF name in the template could be configured as a step or analyte level UDF. If a step level UDF has a value that is specified in the template, all analytes in the step are routed.

• The UDFs can be of any type (Numeric, Text, Checkbox ). If a checkbox UDF is used, the available values are true or false.

• UDF values in the template are not case-sensitive.

The template requires four columns:

• UDF_NAME

• UDF_VALUE

• WORKFLOW_NAME

• STAGE_NAME

For this example, the template values would be:

• UDF_NAME, UDF_VALUE, WF_NAME, STAGE_NAME

• Route all samples to: Step A, Workflow A, Stage A

• Route all samples to: Step B, Workflow B, Stage B

This script might be used numerous times in different EPP / automation scripts, with each referencing a different template.

Using a Template String Instead of a Template File

Due to restrictions on file server access, this script accepts routing template instructions using a string EPP parameter with lines separated by a newline character ('\n'). The following example shows how this parameter string would be added to represent the previous template:

Required Parameters

The EPP/automation that calls the script must contain the following parameters:

*Either --template or --template_string is required. If both are provided, --template_string is used.

Optional Parameter

When the --input parameter is used, the script routes input artifacts instead of the default output artifacts. UDF values of the input artifacts (instead of outputs) are against the template file.

An example of the full syntax to invoke the script is as follows:

Or, if you wish to route the inputs instead of outputs:

User Interaction

When the Record Details screen is entered, the UDF / custom field checkboxes or drop-down options specify to which workflow/stage combination each derived sample is sent.

About the Code

The first important piece of information required is the URI of the destination stage. There is a unique stage URI assigned to each workflow and stage combination.

A stage URI can change across LIMS instances (such as switching from Dev to Prod). Therefore, the script gathers the stage URI from the workflow and stage name. This process occurs even when the workflows are identical.

The main method in the script is routeAnalytes() and it carries out several operations:

1. Gathers the information for the process that triggered the script, including output (or input) analytes.

2. For each analyte, evaluates which UDFs have been set, and adds the analyte to a list of analytes to route.

3. Creates the XML message for each stage.

This example, while in itself useful, also serves to demonstrate a more general concept. Routing artifacts is valuable in any situation where a sample must be queued for a stage outside of the usual order of a workflow. This routing is applicable even when routing newly submitted samples to the first stage in a workflow.

For more information, see the artifact/route REST API documentation.

Assumptions and Notes

• You are running a version of Python supported by Clarity LIMS, as documented in the Clarity LIMS Technical Requirements.

• The attached files are placed on the LIMS server, in the /opt/gls/clarity/customextensions folder.

• The Python API Library (glsapiutil.py) is placed on the Clarity LIMS server, in the /opt/gls/clarity/customextensions folder. You can download the latest glsapiutil library from the page.

Attachments

routing_template.csv:

route_by_template.py:

The Clarity LIMS interface offers tremendous flexibility when placing the outputs of a protocol step into new containers. Consider if your protocol step always involves the placement of samples using the plate map of the input plate. If yes, it makes sense to automate sample placement.

This example provides a script that allows sample 'autoplacement' to occur. It also describes how the script can be triggered, and what the lab scientist sees when the script is running.

For an example script that automates sample placement using multiple plates, see the Automatic Placement of Samples Based on Input Plate Map (Multiple Plates) example.

Solution

In this example, samples are placed according to the following logic:

• The step produces one output sample for every input sample.

• The output samples are placed on a 96 well plate.

• Each occupied well on the source 96 well place populates the corresponding well on the destination 96 well plate.

Step Configuration

In this example, the step is configured to invoke the script on entry to the Sample Placement screen.

Parameters

The EPP / automation command is configured to pass the following parameters:

An example of the full syntax to invoke the script is as follows:

User Interaction

When the lab scientist enters the Sample Placement screen, a message similar to the following appears:

When the script has completed, the rightmost Placed Samples area of the Placement screen will display the container of auto-placed samples:

At this point the lab scientist can review the constituents of the container, and complete the step as normal.

About the Code

The main method in the script is autoPlace(). This method in turn carries out several operations:

1. A call to createContainer() prompts the creation of the destination 96 well plate.

2. The method harvests enough information so that the objects required by the subsequent code can retrieve the required objects using the 'batch' API operations. You add additional code to build and manage the cache of objects retrieved in the batch operations:

   • cacheArtifact()

Assumptions and Notes

• The attached file is placed on the Clarity LIMS server, in the /opt/gls/clarity/customextensions folder.

• The Python API Library (glsapiutil.py) is placed on the Clarity LIMS server, in the /opt/gls/clarity/customextensions folder. You can download the latest glsapiutil library from our .

• The HOSTNAME global variable must be updated so that it points to your Clarity LIMS server.

Attachments

autoplaceSamplesDefault.py:

This example provides a script that can be used to parse lanebarcode.html files from demultiplexing. This script is written to be easily used with the out of the box Bcl Conversion & Demultiplexing (HiSeq 3000/4000) protocol.

• Result values are associated with a barcode sequence as well as lane.

• Values are attached to the result file output in Clarity LIMS, with matching barcode sequence (index on derived sample input) and lane (container placement of derived sample input).

• Script modifications may be needed to match the format of index in Clarity LIMS to the index in the HTML result file.

Parameters

The script accepts the following parameters:

An example of the full syntax to invoke the script is as follows:

Configuration

Defining the UDFs / Custom Fields

All user defined fields (UDFs) / custom fields must first be defined in the script. Within the UDF / custom field dictionary, the name of the field as it appears in Clarity LIMS (the key) must be associated with the field from the result file (the value).

The fields should be preconfigured in Clarity LIMS for result file outputs.

Modifying individual UDFs / Custom Fields

The UDF / custom field values can be modified before being brought into Clarity LIMS. In the following example, the value in megabases is modified to gigabases.

Checking for matching flow cell ID

The script currently checks the flow cell ID for the projects in Clarity LIMS against the flow cell IS in the result file.

NOTE: The script will still complete and attach UDF / custom field values. You may wish to modify the script to not attach the field values if the flow cell ID does not match.

Assumptions and Notes

• Your configuration conforms with the script's requirements, as documented in the Configuration section of this document.

• You are running a version of Python that is supported by Clarity LIMS, as documented in the Clarity LIMS Technical Requirements.

• The attached Python file is placed on the LIMS server, in the /opt/gls/clarity/customextensions folder.

Attachments

demux_stats_parser.py:

demux_stats_parser_4.py:

When Clarity LIMS is replacing an existing LIMS, or if the LDAP integration is not being utilized, a list of current LIMS users and collaborators often already exists in a file.

In this scenario, it is useful to use this file to create users in Clarity LIMS. This topic provides an example script and outlines a strategy for the bulk import of users from a simple CSV file.

Solution

The attached script parses a CSV file containing user information. Based on the columns of data present in the file, the script then creates users and their associated labs in Clarity LIMS.

Parameters

Since this operation is independent of workflow and sample processing, the parameters used to invoke the script differ from those typically used:

An example of the full syntax to invoke the script is as follows:

File format

The format of the file is very flexible. The order of the columns is not relevant.

If the names of your columns do not match the column names in the example file, modify the script so that the column names match yours.

Attached to this topic, you'll find a CSV file containing test data that illustrates the format further. The structure of this file is shown below.

Notice that the first two rows do not contain data of interest, and so the script ignores them. This is controlled by line 38 of the script, which specifies the location of the header row:

About the Code

The main method of interest is importData(). After parsing the file into data structures (COLS and DATA), the data is processed one line at a time using the following pseudocode:

To reduce the number of queries the script makes back to the server, each time a new lab is created it is added to the cache of existing labs that is stored in the LABS dictionary.

Finally, if you set the DEBUG variable to true, the script will stop processing the file after the first successful creation of a user. This is useful as it allows you to test your script using just one line of the CSV file at a time.

Assumptions and Notes

• Both of the attached *.py script files are placed on the Clarity LIMS server, in the /opt/gls/clarity/customextensions folder.

• The users will be created with the same hard-coded password (abcd1234). It is possible to have the script create a unique password for each user. If this is required, consider having the script e-mail the user with a 'welcome' e-mail outlining their username and password.

• The users will be created with permissions to access the Collaborations Interface only. This can be modified as required.

Attachments

user_List_Test.csv:

userImport.py:

Compatibility: API version 2

Many different types of CSV files are attached to BaseSpace Clarity LIMS. This example provides a template for a script that can parse a wide range of simple CSV files into Clarity LIMS. The user can change parameters to read the format of the file to be parsed.

Solution

Protocol step configuration

CSV files are attached to a protocol step. Artifact UDFs, where data will be written to, need to be configured for artifacts and for the protocol step.

Parameters

The script accepts the following parameters:

An example of the full syntax to invoke the script is as follows:

About the Code

The script contains an area with a number of configurable variables. This allows a FAS or bioinformatician to customize the script to parse their specific txt file. The following variables within the script are configurable:

MAPPING MODES

There are many attributes of samples which can be used to map the data in the text file with the corresponding derived samples in Clarity LIMS. The script should be configured such that one of these modes is set to True.

Three modes are available:

For any of the three modes, a mapping column value must be explicitly given. The value is the index of the column containing the mapping data (either artifact name, well location, or UDF value).

If using the mode MapTo_UDFValue, a UDFName must be given. This is the name of the UDF in clarity which will be used to match the value found in the mapping column.

artifactUDFMap

This Python dictionary maps the name of columns in the txt file to artifact UDFs for the outputs of the step. The data from these columns in the file will be written to these UDFs for the output artifacts. The dictionary can contain an unlimited number of UDFs. The dictionary keys, (left side), are the names of the columns in the txt file, and the dictionary values, (right side), are the names of the UDFs as configured for the artifacts.

Assumptions and Notes

• You are running a version of Python that is supported by Clarity LIMS, as documented in the Clarity LIMS Technical Requirements.

• The attached files are placed on the LIMS server, in the /opt/gls/clarity/customextensions folder.

• The example code is provided for illustrative purposes only. It does not contain sufficient exception handling for use 'as is' in a production environment.

Attachments

genericParser.py:

glsfileutil.py:

In some facilities, when samples are initially submitted into BaseSpace Clarity LIMS, it has already been determined which samples are combined to give pooled libraries. In such cases, it is desirable to automate the pooling of samples within the protocol step. Doing so means that the lab scientist does not have to manually pool the samples in Clarity LIMS interface. This automatic pooling saves time and effort and reduces errors.

Solution

This example provides a script that allows 'autopooling' to occur. It also describes how the script can be triggered, and what the lab scientist sees when the script is running.

The attached script relies upon a user-defined field (UDF) / custom field, named Pooling Group, at the analyte (sample) level.

This UDF is used to determine the constitution of each pool. This determination makes sure that samples combined to create a pool having the name of the Pooling Group value all have a common Pooling Group value.

For example, consider the Operations Interface (LIMS v4.x & earlier) Samples list shown below. The highlighted samples have a common Pooling Group value. Therefore, we can expect that they will be combined to create a pool named 210131122-pg1.

Step Configuration

In this example, the Pooling protocol step is configured to invoke the script as soon as the user enters the step's Pooling screen.

Parameters

The EPP / automation command is configured to pass the following parameters:

An example of the full syntax to invoke the script is as follows:

User Interaction

When the lab scientist enters the Pooling screen, a message similar to the following displays:

When the script has completed, the rightmost Placed Samples area of the Placement screen displays the auto-created pools:

At this point, the lab scientist can review the constituents of each pool, and then complete the protocol step as normal.

About the Code

The main methods of interest are autoPool() and getPoolingGroup().

1. The autoPool() method harvests just enough information so that the objects required by the subsequent code in the method can retrieve the required objects using the 'batch' API operations. This involves using additional code to build and manage the cache of objects retrieved in the batch operations, namely:

   • cacheArtifact()

   • prepareCache()

Assumptions and Notes

• The attached file is placed on the Clarity LIMS server, in the /opt/gls/clarity/customextensions folder.

• The Python API Library (glsapiutil.py) is placed on the Clarity LIMS server, in the /opt/gls/clarity/customextensions folder. You can download the latest glsapiutil library from our .

• The example code is provided for illustrative purposes only. It does not contain sufficient exception handling for use 'as is' in a production environment.

Attachments

autopoolSamples.py:

Whereas config slicer is a useful tool for moving configuration data from one server to another and, strictly speaking, reagent lot data is not considered configuration, there is a need for an easy way to transfer reagent lot data to a new Clarity LIMSserver.

Solution

The preventative solution is to not create production reagent kits on the development server. However, if you're reading this it might be too late for that.

This example demonstrates how to use the Clarity LIMS API to extract reagent kit and reagent lot data from one server, transfer the data as an XML file and create the equivalent reagent kits and/or lots on the new server.

This example contains two scripts which can be run on the CLI, one which exports data from the dev server and generates a txt file. The second which uses the exported data to create the kits/ lots on the prod server.

Export

The first script accepts the following parameters:

An example of the full syntax to invoke the first script is as follows:

Import

The second script accepts the following parameters:

An example of the full syntax to invoke the first script is as follows:

About the Code

reagents_export.py

The main method in the script searches the reagent kits and reagent lots endpoints for all the kits and lots or, if the -k parameter is used, for the kits and lots belonging to kits specified in the -k parameter.

The script writes the XML data for each kit and lot to the XML data file.

reagents_import.py

The main method of this script creates the reagent kits and reagent lots in CClarity LIMS. In the case of duplicate reagent kits, the reagent lots are associated with the newly created kits. If the --checkforKits parameter is included, the script does not create kits with duplicate names, and associates the reagent lots with the preexisting kits with the matching reagent lot name.

Assumptions and Notes

• You are running a version of Python that is supported by Clarity LIMS, as documented in the Clarity LIMS Technical Requirements.

• The example code is provided for illustrative purposes only. It does not contain sufficient exception handling for use 'as is' in a production environment.Attachments

• Script Updates

  Jan 2018: reagents_export_jan2018.py and reagents_import_jan2018.py

Attachments

reagents_import.py:

reagents_export.py:

reagents_import_jan2018.py:

reagents_export_jan2018.py:

reagents_import_Aug2018.py:

The BaseSpace Clarity LIMS interface offers tremendous flexibility when placing the outputs of a step into new containers. Consider if your protocol step always involves the placement of samples using the plate map of the input plate. If yes, it makes sense to automate sample placement.

This example provides a script that allows sample 'autoplacement' to occur, and describes how the script can be triggered.

This script is similar in concept to the Automatic Placement of Samples Based on Input Plate Map example. The difference being that this updated example is able to work with multiple input plates, which can be of different types.

Solution

In this example, samples are placed according to the following logic:

• The process type / master step is configured to produce just one output analyte (derived sample) for every input analyte.

• The output analytes are placed on the same type of container as the corresponding source plate.

• Each occupied well on the source plate populates the corresponding well on the destination plate.

Step Configuration

In this example, the step is configured to invoke the script on entry to the Sample Placement screen.

Parameters

The EPP / automation command is configured to pass the following parameters:

An example of the full syntax to invoke the script is as follows:

About the Code

The main method in the script is autoPlace(). This method executes several operations:

1. The creation of the destination plates is effected by calls to createContainer().

2. It harvests just enough information so that the objects required by the subsequent code can retrieve the required objects using the batch API operations. This involves using some additional code to build and manage the cache of objects retrieved in the batch operations, namely:

   • cacheArtifact()

Assumptions and Notes

• The attached file is placed on the Clarity LIMS server, in the /opt/gls/clarity/customextensions folder.

• The Python API Library (glsapiutil.py) is placed on the Clarity LIMS server, in the /opt/gls/clarity/customextensions folder. You can download the latest glsapiutil library from our .

• The HOSTNAME global variable must be updated so that it points to your Clarity LIMS server.

Attachments

autoplaceSamplesDefaultMulti.py:

An easy way to get sample data into BaseSpace Clarity LIMS is to import a Microsoft® Excel® sample spreadsheet. However, manually configuring the spreadsheet can be time-consuming and error-prone.

Use this Python application example to generate a custom Excel sample submission spreadsheet that meets the specific needs of your lab.

Solution

Parameters

The script accepts the following parameters:

The SampleSubmissionXLSGenerator.py application script is run as follows:

An XLS workbook containing the following worksheets is generated:

• Sample Import Sheet: Contains columns for all Sample UDFs/UDTs.

• Container Type Names: Contains a list of all container type names.

User interaction

1. In the Clarity LIMS Operations Interface, the lab manager configures Container Types and Sample UDFs/UDTs.

2. In the Clarity LIMS Web Interface, the lab manager or lab scientist runs the SampleSubmissionXLSGenerator.py application script, providing the required parameters.

3. An Excel workbook containing the Sample Import Sheet and Container Type Names worksheets is generated.

Using the generated spreadsheet

1. The spreadsheet will contain red and green column headers. Populate the spreadsheet with sample data:

2. • Red headers: These columns must contain data.

Example modifications

You can edit the Python application to include supplementary functions. For example, you may want to use other attributes from the resulting XML to generate additional data entry columns.

Inserting additional non-required & non-configured columns

• The SampleSubmissionXLSGenerator.py Python application script adds Sample/Volume and Sample/Concentration columns to the spreadsheet.

• The script includes a 'commented' section. You can remove the ### NON-REQUIRED COLUMN MODIFICATION ### comments and use this section to add your own columns.

Assumptions & notes

• You have downloaded the attached zip file to the server. On a non-production server use the glsai user account.

• Unzip the file to the following directory: /opt/glsclarity/customextensions. The contents of the zip file will be installed within that directory, to /CookBook/SpreadSheetGenerator/

• Python 2.7 is installed and configured on the system path.

All dependencies are preloaded if you install on a non-production server.

Attachments

python-xls-generator-2.0-bundle.zip:

Illumina sequencing protocols include a BCL Conversion and Demultiplexing step. This stage allows you to select the command options for running bcl2fastq2. The bcl2fastq must be initiated through a command-line call on the BCL server.

This example allows you to initiate the bcl2fastq2 conversion software by the selection of a button in BaseSpace Clarity LIMS.

Step Configuration

The "out of the box" step is configured to include the following UDFs / custom fields. You can select these options on the Record Details screen. You can also configure additional custom options.

About the Code

The main method in the script is convertData(). This method performs several operations:

1. The script determines the run folder. The link to the run folder is attached as a result file to the sequencing step.

   • The script searches for the appropriate sequencing step and downloads the result file containing the link.

   • The script changes directories into the run folder.

Script Configuration

This script must be copied to the BCL server because the script is executed on the BCL server remote Automated Informatics (AI) / Automation Worker (AW) node.

By default, the remote AI / AW node does not come with a custom extensions folder. Therefore, if this script is the first script on the server you can create a customextensions folder in /opt/gls/.

When uploading the script, ensure the following:

• The path to the bcl2fastq application is correct (line 17)

• The sequencing process type matches exactly the name of the process type / master step the artifact went through (the -d parameter)

• The customextensions folder contains both glsapiutil.py and glsfileutil.py modules. See .

Parameters

The script accepts the following parameters:

An example of the full syntax to invoke the script is as follows:

Assumptions and Notes

• You are running a version of Python supported by Clarity LIMS, as documented in the Clarity LIMS Technical Requirements.

• The attached files are placed on the LIMS server, in the /opt/gls/clarity/customextensions folder.

• The Python API Library (glsapiutil.py) is placed on the Clarity LIMS server, in the /opt/gls/clarity/customextensions folder. You can download the latest glsapiutil library from the page.

Attachments

kickoff_bcl2fastq2.py:

glsfileutil.py:

The following resources provide additional information on the tools and technologies used in the API and in the Cookbook examples.

REST

The Clarity LIMS Rapid Scripting API uses a RESTful web service with XML over HTTP. For more information about REST, see the following:

Linux and Common File Formats

Many of the scripts used in the Cookbook examples assume a knowledge of Linux and text file formats.

Review bash shell execution command syntax and file formats such as comma-separated values (CSV) and Extensible Markup Language (XML):

Groovy

For its clarity and readability, we have used Groovy for the Cookbook examples. The website provides useful resources for learning and working with Groovy:

Additional Resources

The capacity for completing a step programmatically, without having to open the BaseSpace Clarity LIMS web interface, allows for rapid validation of protocols. This method results in streamlined workflows for highly structured lab environments dealing with high throughput.

This example uses the /api/v2/steps endpoint, which allows for more controlled execution of steps. In contrast, a process can be executed using the api/v2/processes endpoint with only one POST. This ability is demonstrated in the example.

It may at times be desirable to take key data derived during a workflow and copy it to the submitted sample. There are several reasons why this could be useful:

• All key data is combined with all of the submitted sample's data, and becomes available on a single object.

• Key data determined during a workflow can be made immediately available to external collaborators via the LabLink Collaborations Interface, since these users have access to their submitted samples.

Often, a protocol step will have just a single 'next action' that is required to continue the workflow. In such cases, it can be desirable to automatically set the default action of the next step.

This example script shows how this task can be achieved programmatically.

Solution

Once a sequencing run has occurred, there is often a requirement to store the locations of the FASTQ / BAM files in Clarity LIMS.

For paired-end sequencing, it is likely that the meta-data file that describes the locations of these files will contain two rows for each sample sequenced: one for the first read, and another for the second read.

Such a file is illustrated below:

Column 2 of the file, Sample ID, contains the LIMS IDs of the artifacts for which we want to store the FASTQ file values listed in column 3 (Fastq File).

This example discusses the strategy for parsing and storing data against process inputs, when that data is represented by multiple lines in a data file.