If a BaseSpace Clarity LIMS script is run in an automation context, it is easy to obfuscate usernames and passwords by choosing the appropriate tokens ({username} or {password}) to be passed in as run-time arguments.

However, this type of functionality is not easily available outside of automations, and it is often necessary to store various credentials on machines that need to interact with the LIMS API, database, or some other protected resource. This article explains how to use cryptography in Python to protect and obfuscate these important authentication tokens.

Background

Many of the API Cookbook examples use a simple auth_tokens.py file that has usernames and passwords stored in plain text. This file can be compiled in Python, simply by importing it at a Python console:

import auth_tokens

print auth_tokens.username #just for sanity check

Importing this file creates an auth_tokens.pyc file—a byte-compiled version of the source file. The source file can now be deleted, providing the first rudimentary level of security. However, the credentials can still quite easily be retrieved. Even if the permissions on this file are restricted, this solution does not present a suitable level of security for most IT administrators. It does, however, allow us to easily prototype our code, hence its use in Cookbook examples.

Assumptions

• You have pycrypto installed (either through the OS package manager or pip).

• You have generated a secret key of random ASCII characters (the easiest way to do this is to button-mash on a US-layout keyboard and include a lot of symbols).

• You already have a plain-text auth_tokens.py file. An example is attached at the bottom of this article.

• You have access to the Python or iPython command line console.

Cryptography in Python

Python provides the pycrypto library that can easily be installed using the operating system's package manager, or the pip installation tool. It contains myriad different encryption algorithms and gives us a straightforward interface to wrap our own encryption objects and accessor functions.

Towards a Better auth_tokens.py

The goal is to be able to create a flat text file containing obfuscated usernames, passwords, hostnames, and so on. To do this, use a utility class called ClarityCred that provides encryption and decryption functionality using the ARC4 cipher from pycrypto. The ClarityCred class is provided in cred.py, attached at the bottom of this article.

While the use of ARC4 is considered deprecated in favor of stronger encryption algorithms, such as AES, the ARC4 example lends itself to easier understanding. ARC4 simply requires a secret key and a salt size to be specified. The secret key can be generated at random using any preferred method and is hard-coded in cred.py, along with the salt size purely for ease of demonstration. Ideally, the secret key and salt size should be stored externally.

After applying the ARC4 encryption, the ClarityCred class wraps base64 encoding around it to obfuscate the data further.

Assume that you need to store a username, password, and hostname inside our auth_tokens.py, and we have this information in plain-text stored in another file called auth_tokens_plain.py. The usage is as follows.

1. Open a Python console, and import ClarityCred from cred.py.

2. Call the ClarityCred.encrypt() static function on the plain text username, password, and hostname strings.

3. Copy-paste these values into auth_tokens.py.

The following image illustrates steps 1 and 2, using an existing auth_tokens_plain.py file:

The old auth_tokens_plain.py looked like this:

username = 'testuser' password = 'testpass' hostname = 'https://encryptiontest.claritylims.com'

The new auth_tokens.py looks like this:

username = 'zq1AwnqIkfA=$YFY1UuO1r6edu7qPnN9/l3kMI15ZG1JAsH7IhnxnNvYulMndhYh6lxjVBfFwjN9sZEqPM0Qlx6kjq3fbht/FlRrgklDL79H7NiUP6uYM2qVltPloRA4g8SiphF3KHx4gVTE93Ku58sFCgu1rnH5u6tkCz98v0R7PsuIOW1CDMi9zSToIu+IkcYDPPYcD1b4z8ojez/7lczunaDfrmPhwopyyUiETu9BR49Bwp5fz4XSWICZFGCd9AjoEg/FTE+/X18f+0pIz0viXQyN+JjE3vJkpNsRY2Z3d72sPgQmFFZhd48m+POUtD1UXLXhaijdxp78QTcEp7AHY+TiM8hsXT7BX1Q=='

password = '9qW5BftGyXY=$6GL1t/Zl1CbSmB7Qq54uf2TJ5fI8GUlW9NdBnumkTtF/X27WLEsr1+C0ilXQX6jnLm4kzR+5pCVgnz4xz6/80/dMLMlTll6tOvCJgPU4ZkRpkUYmcPVbrp+X3azR7I024O8UjV/JeJYV869h3kvdPyWJGXRH4oJgs5NTJKI2y6URBs0wlrlgBuZ2YkO855ZGPw9J07UMM606q9xERRzQ+LT1XLRzSCuFnuSoDVEhshhYqZ/jpYWDHvA6Z5+YTYI/i099iYZ+WQdJAiU9hcgkUnWCybjcwivvHG6vAIROroLqlOefo+hrJsVFBA3uDaPS8pkgMVsKMPUGeft6vx4NgN/jaw==

hostname = 'Q+oyq2m9Nv8=$rhgeJOMdm/M+dDNlSbBA3RCsUoo0Ts65G7lePvuajRmsLSNC5Qo5bwagRuyat0ztpeZrUmD8xTxTvhUBvZYDlM6GBLsq5drBP6PFh/lplxb6O8YiSRXrboFov8tRnu6GbaTfGR8WV7s8vBZsXhrhlPn67p7yalJLnHWb9VOKhx8AgCTtytQkkEwmpm2vbDwDha9kMdK63IrOSp2jmRaI/9X3xsd4upqaxvX7zrEJ8ruGU/szN0ITxTK1rprnowpyXfBRiOEcrI7uh1bg73oqOETn3pB/uTrGkhGETKYB2aHaewwWMccbeZTgEPT0kDmuJdpoGYy+p+gxSoR9Arh3JtREIA=='

Examples of the plain-text auth_tokens_plain.py and encrypted auth_tokens.py are attached at the bottom of this article.

Using the New auth_tokens.py in Your Scripts

Now that the new auth_tokens.py is ready to use, you can import it and create the corresponding PYC file to provide that extra level of security, as previously discussed. You can remove the PY file and ship the PYC file everywhere it is required.

It may also be a good idea to restrict the read/write/execute permissions on the file to the system user that is calling the file (usually glsai in Clarity LIMS installations).

To use the values in this file in the code, we need to use the decrypt() function in ClarityCred. Look at the simple example of initializing a glsapiutil api object. For reference, the example current directory listing looks like this:

Notice the .py source files are removed wherever possible.

Using a Python console, the normal api invocation (using a plain-text auth_tokens file) would look as follows.

import glsapiutil import auth_tokens_plain api = glsapiutil.glsapiutil2() api.setHostname( auth_tokens_plain.hostname ) api.setVersion( 'v2' ) api.setup( auth_tokens_plain.username, auth_tokens_plain.password )

Now, however, with our encrypted tokens, we decrypt the values on-the-fly (changes shown in italicized red text):

import glsapiutil import auth_tokens from cred import ClarityCred api = glsapiutil.glsapiutil2() api.setHostname( ClarityCred.decrypt( auth_tokens.hostname ) ) api.setVersion( 'v2' ) api.setup( ClarityCred.decrypt( auth_tokens.username ), ClarityCred.decrypt( auth_tokens.password ) )

This method provides a relatively robust solution for encrypting and obfuscating sensitive data and can be used in any Python context, not just for Clarity LIMS API initialization. By further ensuring that only the auth_tokens.pyc file is shipped and copied with restricted read/write/execute permissions, this method should help satisfy IT security requirements.

However, the matter of storing the secret key externally remains. One idea is to store the secret key in a separate file and encrypt that file using openssl or an OpenPGP key. While the problem of storing each piece of information in encrypted format likely never fully goes away, the use of multiple methods of encryption can offer better protection and peace of mind.

Attachments

auth_tokens.py:

auth_tokens_plain.py:

auth_tokens_plain.py:

Automations (formerly referred to as EPP triggers or automation actions) allow lab scientists to invoke scripts as part of their workflow. These scripts must successfully complete for the lab scientist to proceed to the next step of the workflow.

EPP automation/support is compatible with API v2 r21 and later.

The API documentation includes the terms External Program Integration Plug-in (EPP) and EPP node.

As of Clarity LIMS v5.0, these terms are deprecated.

• EPP has been replaced with automation.

• EPP node is referred to as the Automation Worker or Automation Worker node. These components are used to trigger and run scripts, typically after lab activities are recorded in the LIMS.

Automations have various uses, including the following:

• Workflow enforcement—Makes sure that samples only enter valid protocol steps.

• Business logic enforcement—Validates that samples are approved by accounting before work is done on them. This automation can also make sure that selected samples are worked on together.

• Automatic file generation—Automates the creation of driver files, sample sheets, or other files specific to your protocol and instrumentation.

• Notification—Notifies external systems of lab progress. For example, you can notify Accounting of completed projects so that they can then bill for services rendered.

Configuration

You can enable automations on master steps in two configuration areas of Clarity LIMS:

• On the Automations tab, when adding/configuring an automation. See the Adding and Configuring Automations article in the Automations section of the Clarity LIMS documentation.

• On the Lab Work tab, on the master step configuration form. See the _Adding & Configuring Master Steps and Step_s article in the Steps and Master Steps section of the Clarity LIMS documentation.

After it is enabled on a master step, the automation becomes available for use on all steps derived from that master step.

You can configure the automation trigger on the master step, or on the steps derived from that master step.

Script messages

Progress message

While executing a script, if more than one script would be triggered for a single user action, they are reported in sequence. This reporting continues until all scripts complete, or one of them fails.

An example scenario would be a step that is configured to execute the following:

• One script upon exit of the Placement screen.

• A second script upon entry of the Record Details screen.

In this scenario, when the lab scientist advances their protocol step from the Placement screen to the Record Details screen, the scripts are executed in sequence.

The parameter string/automation name configured on the master step is displayed in a progress message. You can use this feature by giving your parameter strings/automations meaningful names that provide you with context about what the script is doing. The following is an example of a progress message.


You cannot proceed until the script completes successfully.

Non-responsive scripts

You can request to cancel a script that is not responsive. While canceling abandons the monitoring of script execution, it does not stop the execution of the script.

After canceling a script, follow up with the Clarity LIMS administrator to determine if the AI node/automation worker must be restarted.

Non-Blocking Success and Warning Message

The scientific programmers in your facility can provide you with a message upon successful execution of a script. There are two possible non-fatal messages: OK and WARNING. These messages can be set using the step program status REST API endpoint.

Message boxes display the script name, followed by a message that is set by the script using the step program status REST API endpoint. Line breaks are permitted in the custom message. The following is an example of a success message:

After you select OK, you are permitted to proceed in the workflow.

Blocking Script Failure Message

When a script fails, a message box displays. There are two ways to produce fatal messages:

• By using the step program status REST API endpoint (informing FAILURE as the status)

• By generating output to the console and returning a non-zero exit code.

For example, when beginning a step, if the script does not allow you to work on the samples together in Ice Bucket, the samples will be returned to Ice Bucket after acknowledging the error message. In this case, the step is prevented from being tracked. The following is an example of a failure message:

If you attempt to advance a step from the Pooling screen, but an error is detected, the error state prevents you from continuing. The following is an example of this type of message:

After you select OK, you are prevented from proceeding in the workflow. Instead, you must return to the Pooling screen and address the problem before proceeding.

REST and automation are the key interfaces for scripting. A language-agnostic application programming interface (API) is important to scientists as it allows for broad and diverse integration. Together, REST and automation provide powerful and easy-to-use scripting. However, you first need to understand the conceptual structure and design of these interfaces.

Within the Clarity LIMS Rapid Scripting API, REST technology is used to provide data specifically structured for life science research.

NOTE: If you are new to the REST Web Service, we recommend that you read Development Prerequisites and REST Web Services.

REST: a web service for data access

The REST Web Service is the fundamental data access interface using XML over HTTP. It is agnostic to programming languages as most languages support HTTP and XML with libraries or built-in methods.

In life science research labs, tracking samples and the data associated with biology, research, and lab work is complex. The REST resources return information that is human-readable and interpretable. Use a web browser to explore the XML returned.

REST represents real laboratory items and activities in self-contained groups of data called resources. It provides access to recorded lab steps and to sample test results, and it provides this access using resources. For example:

• The process and steps resources track the steps in the lab in terms of who did what and when.

• The sample and artifact resources contain information on the submitted sample and test results on sample derivatives (also referred to as derived samples).

The REST resources and their relationships are explained in Structure of REST Resources.

The full details of each resource are described in the API Portal.

General concepts

Requests are made to the API by sending XML messages:

• POST is used to create an item.

• GET is used to read an item.

• PUT is used to update an item.

• DELETE is used to delete an item.

Note: HEAD requests are not supported.

The full URL to which requests should be sent will vary depending on the specific installation, but will generally follow this format:

  http[s]://<hostname>:<port>/api/<version.of.api> 

Automation / EPP: GUI trigger for calling scripts

Automation / EPP is used to trigger scripts from within the Clarity LIMS interface.

Script-triggering is often used because the data collected needs to be dispatched for further processing. Automating data processing and returning information, in the appropriate format, to the lab for immediate use increases efficiency and quality.

File handling and file management are fundamental elements in life science scripting. When triggered, scripts can issue a command, transfer files for processing, and collect and transfer files back to the server. To enable triggering of scripts in any programming language, the information and files are provided for batch processing at the operating system command line level.

For Clarity LIMS (v5 and later)

To use automation, administrators complete the following steps:

1. In Clarity LIMS, create and configure master steps.

2. Configure automations that trigger scripts. Enable those automations on the master steps.

3. Use the configured master steps as building blocks to create and configure steps to be run by lab scientists.

Related Resources

• Automation

• Work with EPP/Automation and Files

The REST API methods attempt to return appropriate HTTP status codes for every request. To use the REST API effectively, a good understanding of HTTP and status codes is required. A complete list of HTTP status codes and definitions can be found at the following website: HTTP/1.1 Status Code Definitions

The primary status codes used by the REST API are as follows:

• 200 OK: Success.

• 201 Created: A resource was successfully created.

• 400 Bad Request: Invalid data was supplied for the relevant resource type.

• 401 Unauthorized: The requested resource cannot be loaded until valid logon credentials have been entered. If this error is received after logon credentials have been entered, this indicates that the credentials are not valid.

• 403 Forbidden: Access to the requested resource has been denied. (Make sure that the authorized user has administrative privileges.)

• 404 Not Found: The URI requested is invalid or the resource requested does not exist.

• 413 Request Entity Too Large: The request is larger than the server is willing or able to process.

• 500 Internal Server Error: A generic error message, given when there is no suitable specific message.

Error message format

Error messages are returned as exception elements with a message element containing a user-facing error message.

The exception may also include a suggested-actions element with more detail on how to resolve the error.

User-facing XML error messages are not returned for 401 and 403 errors. In these cases, the HTTP error must be resolved.

Together, REST and External Program Integration Plug-ins (EPP)/automation provide powerful and simple-to-use scripting. Before working with the REST API, understand the conceptual structure and design of these interfaces.

The links below provide overview information to help you get started, a self-training Cookbook guide with example scripts, and videos that supplement the API training materials.

Current API Version:

- v2 r34

Previous API Versions:

- v2 r33

Reagent labels are artifact resource elements and can be applied using a PUT. To apply a reagent label to an artifact using REST, the following steps are required:

1. GET the artifact representation.

2. Insert a reagent-label element with the intended label name.

3. PUT the modified artifact representation back.

You can apply the reagent label to the original analyte (sample) artifact or to a downstream sample or result file.

Prerequisites

Before you follow the example, make sure that you have the following items:

• Reagent types that are configured in Clarity LIMS and are named index 1 through index 6.

• Reagents of type index 1 through index 6 that have been added to Clarity LIMS.

• A compatible version of API (v2 r14 to v2 r24).

Code Example

In this example, you can adjust the following code:

def toLabel = GLSRestApiUtils.httpGET(artifactToLabelURI, username, password)
println '*** Before labeling'
println GLSRestApiUtils.nodeToXmlString(toLabel)
println
 
new Node(toLabel, 'reagent-label', [name: 'Index 1'])
println '*** After labeling'
println GLSRestApiUtils.nodeToXmlString(toLabel)
println
 
GLSRestApiUtils.httpPUT(toLabel, toLabel.@uri, username, password)

By inserting the reagent-label element, you end up with the following code.

*** Before labeling
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<art:artifact xmlns:udf="http://genologics.com/ri/userdefined"
    xmlns:file="http://genologics.com/ri/file" xmlns:art="http://genologics.com/ri/artifact"
    uri="http://yourIPaddress/api/v2/artifacts/RCY1A97PA1?state=301" limsid="RCY1A97PA1">
    <name>Sample-1</name>
    <type>Analyte</type>
    <output-type>Analyte</output-type>
    <qc-flag>UNKNOWN</qc-flag>
    <location>
        <container uri="http://yourIPaddress/api/v2/containers/27-12" limsid="27-12" />
        <value>A:1</value>
    </location>
    <working-flag>true</working-flag>
    <sample uri="http://yourIPaddress/api/v2/samples/RCY1A97" limsid="RCY1A97" />
</art:artifact>
 
*** After labeling
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<art:artifact xmlns:udf="http://genologics.com/ri/userdefined"
    xmlns:file="http://genologics.com/ri/file" xmlns:art="http://genologics.com/ri/artifact"
    uri="http://yourIPaddress/api/v2/artifacts/RCY1A97PA1?state=301" limsid="RCY1A97PA1">
    <name>Sample-1</name>
    <type>Analyte</type>
    <output-type>Analyte</output-type>
    <qc-flag>UNKNOWN</qc-flag>
    <location>
        <container uri="http://yourIPaddress/api/v2/containers/27-12" limsid="27-12" />
        <value>A:1</value>
    </location>
    <working-flag>true</working-flag>
    <sample uri="http://yourIPaddress/api/v2/samples/RCY1A97" limsid="RCY1A97" />
    <reagent-label name="Index 1" />
</art:artifact>

Although it is not mandatory, it is recommended that you name reagent labels after reagent types using the Index special type. This allows you to relate the reagent label back to its sequence.

The Clarity LIMS Rapid Scripting™ API provides scientific programmers with self-descriptive, yet flexible, data access. It uses a RESTful model for data access because this model is well suited to these requirements. This article provides a high-level introduction to REST concepts and technologies.

Introduction to REST Web Service Technology

Representational State Transfer (REST) is a style of software architecture for distributed information retrieval systems, most commonly observed by using the web.

• REST governs proper behavior. It is not a methodology or a design principal, but rather a set of rules to which a system should conform.

• REST allows a uniform interface between clients and servers that is simple and decoupled, enabling each system to evolve independently.

• REST is referred to as stateless because each new API request contains all the information required to complete it, without relying on previous requests. Conforming to these REST principals is referred to as being RESTful.

• REST was developed in parallel with HTTP and makes use of this protocol. It is an elegant way to programmatically access resources over HTTP. It is very flexible because you can use it with any language or tool that supports HTTP.

Other uses of REST

The web is probably the largest known RESTful system. Its behavior is very simple:

1. When you click a link in a web browser, your system requests information by sending a GET request to the specified URL. This URL is a resource.

2. The server that hosts the URL responds, typically with one of two things:

   • If the page exists, the server sends the browser an HTTP 200 response code and the contents of the page.

   • If the page does not exist, the server sends an HTTP 404 response code and an error message indicating that the page cannot be found.

Many software development groups use RESTful APIs. Google, Yahoo, and many public web sites use the RESTful model for information access.

Communicating with the REST API

The REST API allows you to retrieve and update information using HTTP operations. This ability provides some flexibility in how to communicate with the system.

While REST requests and responses can be in a variety of formats, we chose XML. Each resource and XML element is detailed in the API Portal.

Authenticating with the API

To use the REST API, sign in using HTTP BASIC authentication. The method used to authenticate will depend on how you use the API:

• When using a browser to retrieve information from the API, sign in to the browser with a user name and password. When signing in using a browser, the session remains open until the browser is closed.

• When using an HTTP request tool to retrieve, add, update, or remove data using the API, the tool asks for a user name and password each time you submit a request to the system.

• When using a script to communicate with the API, the script must first authenticate with the API. The session remains open for as long as the script is being actively read by the system.

The account you use to sign in to the API must have System Administrator or Facility Administrator privileges.

Understanding URIs used by the API

The API allows self-discovery of an object. When you request information about an object, the system typically returns URIs to its children and, sometimes, its parent. Use one URI to find the next URI in a hierarchy.

When viewing XML in a browser, tools can automatically create links from the URIs returned by the system. Examples of such tools are the Firefox Text Link or Linkificator add-ons. This way, you can select URIs to browse through the API.

Requests are made to the API by sending XML in HTTP calls:

• GET is used to read an item.

• POST to create an item.

• DELETE to delete an item.

• PUT is used to update an item.

In its simplest form, use a browser to enter and read the content of a URI, which allows browsing through the system. When using this method, a GET request is issued to the API for a specified object (referred to as a resource). The request returns XML containing the metadata about that resource. See the following section for details.

If you want to add, update, and delete small amounts of data using the API, use an HTTP request tool, such as the Firefox RESTClient add-on.

Resources and Namespaces

When working with REST, there are references to resources and namespaces.

For references:

A RESTful API groups related information into resources, each of which is referenced with a global identifier (URI).

In the API, for example, every sample in the LIMS has a resource for its information. When scripting, the resource is created or updated with POST and PUT HTTP calls. There are two types of resources: single and list types.

• A list resource is used to access a collection of single resources (such as a listing of all samples).

• The single resource type is used to access details on just one resource (a sample, for example).

It's important to understand how the information in the LIMS has been grouped and structured into resources. To learn more, see Structure of REST Resources.

For namespaces:

An API that uses XML relies on namespaces. In XML, namespaces define the vocabulary of elements and attributes in an XML document. Each REST resource references the XML structure defined by a particular namespace.

When scripting, we use namespaces to look up specific details related to the XML data elements, attributes, and formats that represent a resource. Namespaces also order the subelements of the XML document.

In the current revisions of the API, the PUT and POST methods read the subelements of the XML independent of order, but the namespace still defines the order of the XML provided in GET calls.

Additional Information

For sophisticated write operations and automation of work, you must use a script to communicate with the API. The Cookbook contains examples that demonstrate how to use scripts to perform your work.

Overview

The internal Clarity LIMS API (eg, https://example.claritylims.com/clarity/api) is the API used to deliver the Clarity LIMS web interface. This interface is not typically meant for public consumption. However, some customers use it for troubleshooting and to mitigate system issues.

Preventing CSRF Attacks

As of Clarity LIMS v5.1, access to the internal Clarity LIMS API changed to enhance security and prevent Cross Site Request Forgery (CSRF) attacks. Two new HTTP headers must now be present when issuing PUT, POST, DELETE, and PATCH requests:

• Origin—This header must be set to the scheme and authority of the server being accessed (eg, https:// example.claritylims.com).

• X-Requested-With—This header must be set to XMLHttpRequest.

The attached cURL, Python, and Java examples demonstrate how to authenticate and issue internal API requests. These examples assume a Clarity LIMS server at https://example.claritylims.com.

csrf headers.sh:

csrf headers.py:

csrf headers.java:

The REST API only returns 500 results per request. Because of this feature, with certain resources, you can use the start-index parameter and previous-page and next-page elements to work with large amounts of data.

For example, the following request is submitted to the API:

The response looks like this:

Note the presence of the previous-page and next-page URIs, which allow moving within the pages of results.

Viewing Results from a Specified Point

Use the start-index parameter to view results from a specified point in a list. The first record in a list is index 0, and you can use values that are positive, whole numbers. If the value specified is greater than the number of results available from a resource, the system returns an empty list.

Configuring the Maximum Number of Requests

By default, the REST API only returns 500 results per request. To change the default number, contact the Illumina Support Team.

Removing Fields and Types

To remove a UDF or UDT value, submit a PUT request with the desired UDF or UDT omitted from the XML.

Updating Resources

When submitting a PUT, it is critical to update all information. The submitted XML must include all the current UDFs and UDTs for the resource. If the field and type elements for a UDF and UDT are not included, the system removes those fields and types.

To update the UDF information for an item, the PUT request can add new UDF values and update or remove current UDF values. When working with UDTs, replace the current UDT with another UDT, or add or remove fields within the current UDT.

• Update all UDFs and UDTs in a PUT

• Even if the current user-defined values are not changing, include the current UDF and UDT values in the XML representation for a PUT request.

Filtering with UDF and UDT Values

Data formatting of the UDF and UDT values is important when filtering a resource list with a query parameter. When using UDF or UDT values as a query parameter, all nonalphanumeric characters must be URL encoded.

Data Type Formats

UDFs and UDTs are presented as fields and types in the XML. The following example shows a representation of fields and types returned by a GET request for a sample:

Data Type Values in XML

XML resource representations do not render UDF values in the same format as views in the client user interface. The table later in this section compares images taken from the client user interface and the XML from an http GET.

The differences are intentional, to remove ambiguity and aid script writers when handling the data values.

The following table compares the values displayed by the user interface and the API for UDF data types.

Numerics and Significant Digits

Trailing zeros are removed to support both integer and real numerics. Determine the significant digits by looking up the display-precision element of the /configuration/udfs/{udfid} resource.

EPP UDF Date format

When Date UDFs are expanded on the EPP command line, their format differs from the one used in the Clarity LIMS GUI and the REST API.

The internal Clarity LIMS API (eg, https://example.claritylims.com/clarity/api) is the API used to deliver the Clarity LIMS web interface. This interface is not typically meant for public consumption. However, some customers use it for troubleshooting and to mitigate system issues.

Preventing CSRF Attacks

As of Clarity LIMS v5.1, access to the internal Clarity LIMS API changed to enhance security and prevent Cross Site Request Forgery (CSRF) attacks. Two new HTTP headers must now be present when issuing PUT, POST, DELETE, and PATCH requests:

• Origin—This header must be set to the scheme and authority of the server being accessed (eg, https:// example.claritylims.com).

• X-Requested-With—This header must be set to XMLHttpRequest.

The attached cURL, Python, and Java examples demonstrate how to authenticate and issue internal API requests. These examples assume a Clarity LIMS server at https://example.claritylims.com.

Attachments

csrf headers.sh:

csrf headers.py:

csrf headers.java:

Clarity LIMS software is a powerful laboratory information management system (LIMS) designed to optimize genomics sample and workflow management. It enables labs to track samples, streamline complex tasks, generate sample sheets, and identify poor-quality samples before they reach the sequencing system.

• Saves time and minimizes errors in sample handling through an automated workflow.

• Out-of-the box integration with Illumina instruments. Accelerate adoption of Illumina NGS and array protocols with preconfigured workflows that require no coding experience.

• Designed with compliance features including data entry validation, workflow enforcement, audit trails, electronic signatures and role-based permissions.

• Easily collect and share data in real-time with external clients via LabLink. Collaborate on sample submission, status, and results delivery in a single, secure environment.

• Scales with laboratory needs, accommodating third-party instruments and software through a robust RESTful Application Programming Interface (API).

• Flexible deployment options with cloud and local implementations supported.

What's New

Security Bulletin

The following table shows how API terminology maps to terminology used in the Clarity LIMS v5.x interface.

See also the Terms and Definitions section.

When generating XML, explicitly set the document to UTF-8 character encoding.

If using other encoding methods (eg, MacRoman for OS X), special characters such as μg/ml are stored incorrectly. This could cause data integrity issues.

In Groovy, set the encoding attribute on the StreamingMarkupBuilder object as shown in the following example:

When submitting a request to the REST API, specify the version of the API being used. The version number is a path parameter in each resource URI. The desired API version is substituted into the request URI as follows:

Changes to the API are tracked with a version number (version major) and a revision number (version minor).

• The version number indicates forwards and backwards compatibility.

• The revision number within the version describes features added to the API that will not negatively affect current functionality.

Only the version number is referenced as part of the request. The revision number simply tracks incremental enhancements to the API.

When a new version of the API is released, update the scripts and code as soon as possible.

Check API Version

To find out what version of the API is available from a given server, submit a GET request to the base API URI. For example, in a web browser, browse to:

The system will return the version:

Exception for Legacy Systems

The API was originally intended for internal use or for just a few customers. In those early days, API versioning was different. If working with legacy scripts, this older functionality can be maintained. For example, if scripts were written before v2 and have nondefault system configuration properties for api.prefix and api.rewrite on the server, the …/api/ URI lists the resources and does not provide version information.

Traversing Up a Genealogy

The artifacts resource includes a parent-process element that provides a URI to the process that created an artifact.

To facilitate walking up the genealogy, the processes resource exposes the parent process for an input artifact in the input-output-map of a process:

The parent-process element does not display if the parent process is not supported by the API.

Traversing Down a Genealogy

The processes resource supports an inputartifactlimsid query parameter. This parameter limits the list of processes to those processes with one of the specified artifacts as an input.

1. Start with the initial sample.
2. The processes for the sample can be queried, as follows.
3. The process contains an input-output-map for the input artifact.
4. The steps can then be repeated using the LIMS ID of each output artifact that is associated with the input artifact.

Understanding lab information management in a scientific context is one of the more powerful skills in genomics research today. The Clarity LIMS Rapid Scripting™ API is designed to use these skills, allowing a knowledgeable scientific programmer to adapt lab informatics with scripts and automation.

NOTE: Based on experience working with bioinformaticians and scientific programmers, assumptions about your background, setup, and skills have been made.

Before you begin

Before using the API Cookbook, set up a #h_19503004-55bb-48e6-a8ad-af73e66a1d54.

If any of the topics covered on this page are a concern, contact the IlluminaSupport team for additional training or custom scripting services.

Terminology

Within the Cookbook, the term scripting refers to programs running independently of the client and server that direct the input and output of information. Use scripts and the API for file handling and text processing in the context of biological samples, containers, and instruments.

Skills and Training

• This API Cookbook assumes that you can program in modern computer languages, and are comfortable with scripting and bioinformatics.

• The topics are best understood by those users who can program small applications and are experienced with experimental processes in molecular biology.

• The topics assume that you have received administrator-level training or know how to configure the system. The topics also assume that a nonproduction server is set up to play with cookbook examples, develop real scripts, and test before deploying in production.

• Be comfortable with the following skills:

  • XML

  • System file handling

  • General-purpose scripting languages

  • Working on the command line

Non-production scripting sandbox servers

Illumina provides multiple server licenses for API users: a production server license and one or more non-production server licenses for developing and testing.

To allow developers to design, build, test, and upgrade efficiently, it is recommended to install at least two servers. Installing three is even better.

The non-production server licenses serve the following purposes:

• To provide a sandbox in which to experiment with the API and the system configuration.

• To provide a verification platform for upgrading scripts, software components, and overall system integration before deploying to production.

All the examples in the Cookbook are intended to be used with the nonproduction scripting sandbox server. See Useful Tools.

Clarity LIMS version 4.0 introduced architectural changes that enforce SSL-based security. As a result, the structure of the URIs that reference the Clarity LIMS API was modified, and scripts written before Clarity LIMS v4.0 may require updating.

Details

Scripts that use the API do so by using RESTful methods on specific URIs. The base portion of the URI references the server on which the Clarity LIMS application is running.

Before Clarity LIMS v4.0 the base portion of the URI took the following form:

http[s]://<your_server_name>:<your_port_number>/api

Where:

• The protocol could either be HTTP or HTTPS, depending on whether the application was SSL-enabled or not.

• <your_server_name> represented the fully qualified domain name (or IP address) relating to the server on which the Clarity LIMS application was running.

• <your_port_number> represented the port number (typically 8080) on which the Clarity LIMS application was listening.

In Clarity LIMS v4.0 and later, the base portion of the URI is in the following form:

https://<your_server_name>/api

Where:

• The protocol must be HTTPS, because the Clarity LIMS application is now installed with SSL enabled.

• The server name must match the certificate that was purchased and installed into Clarity LIMS.

• The port number (and the colon) is no longer required. Do not provide it.

When to Update Existing Scripts

The following information should help determine if updates to the scripts are needed.

Scripts generally determine the API URI in one of the following ways:

1. The URI is passed to the script by the automation or External Program Plugin (EPP) component, as a parameter or command-line argument.

2. The URI is passed to the script by another script or a command line embedded in a crontab file.

3. The script contains the URI as a hard-coded string literal.

4. The script determines the fully qualified domain name of the server and adds the prefix (http://) and suffix (:8080) accordingly.

5. The script imports, or includes a file that contains, the URI.

Most scripting uses methods one or three. However, other methods may be used in the facility.

• If method one is used, it is not necessary to update the scripts because Clarity LIMS passes in the new form of the URI.

• If other methods are used, you likely need to update the scripts to convert the URI to the new format. Often, a search and replace tool is able to make these changes.

To make sure that the correct locations are searched, keep in mind that scripts are often stored in the following locations:

1. In the /opt/gls/clarity/customextensions folder (and subfolders) on the server where Clarity LIMS is running. This location is the domain of the default Automation Worker (AW)/Automated Informatics (AI) node, which listens on the channel name of limsserver.

2. If there are additional AW/AI nodes on the server, in the folders used by these nodes.

3. If there are additional AW/AI nodes external to the Clarity LIMS server, within the folders used by these nodes.

4. If scripts are launched by cron, or other mechanisms, they could be stored anywhere and may not even be on the Clarity LIMS server itself.

For points 1–3, query Clarity LIMS (via either the API or the database) to produce a listing of all the scripts it is configured to use. As a result, determine on which node they run and their location.

For point 4, there is no easy answer. Hopefully, if the script is important, the location has been documented.

When submitting a GET request to certain REST API resources (also known as list resources), the system returns a list of records. For example, submitting a GET request to the samples resource returns a list of all submitted samples stored in the system. Depending on the resource being used, use various query parameters to filter the records based on certain criteria. For more information about the parameters that are available, refer to the reference documentation for the desired resource.

To filter a list, the resource and parameter must be separated with a question mark (?). The parameter and the value you want to base the query on must be separated with an equal sign (=).

When filtering a list of artifacts, combine parameters within the same query statement. You can also repeat certain parameters, specifying a new value with each occurrence of the parameter.

The first parameter must be preceded with a question mark (?). Add additional parameters by separating each parameter with an ampersand (&).

Repeating a parameter with new values:

Combining parameters:

When combining or repeating parameters, each record returned matches one of the parameter values, or all the parameter values, depending on the usage:

• If the query statement contains multiple values for the same parameter, the ampersands are treated as an OR.

• If the query statement contains values for multiple parameters, the ampersands are treated as an AND.

For example, if a project LIMS ID and a process type are provided as parameters, the system returns only the files that match both the project LIMS ID and the process type. To see the files that match the project LIMS ID or the process type, issue two separate GET requests and combine the results.

• /api/v2/processes—This URI returns all processes run in the system.

• /api/v2/processes?type=MALDI—This URI returns all MALDI processes run in the system.

• /api/v2/processes?type=Sample Prep&type=MALDI—This URI returns all MALDI or Sample Prep processes run in the system.

• /api/v2/containers—This URI returns all containers in the system.

• /api/v2/containers?type=Tube—This URI returns all tubes in the system.

• /api/v2/containers?type=Tube&name=27-111&name=27-112—This URI returns the tubes in the system that are named 27-111 OR 27-112.

Using the Last-Modified Parameter

Certain resources include the last-modified query parameter.

When used, the system displays only the results that have been modified because the last-modified date. The last-modified date is represented in ISO 8601 Complete date including hours, minutes, and seconds format: YYYY-MM-DDThh:mm:ssTZD.

Viewing Paginated Results

Lists of records are often large, spanning multiple pages. Many list resources include parameters that are used to work with paginated results.

Using UDF and UDT parameters

Certain resources include parameters that can be used to filter the results displayed based on UDF information that is associated with the results:

• udf.UDFNAME[.OPERATOR]=UDFVALUE—This parameter filters the results based on a specified value for a specified UDF. Any item that contains the value for the UDF is returned, unless parameters include an optional operator filter ( [.OPERATOR] in the expression provided). The lowercase filter operators of min or max are described later.

• udt.name=UDTNAME—This parameter filters the results based on a specified UDT. Any item that has the UDT selected is returned.

• udt.UDTNAME.UDFNAME[.OPERATOR]=UDFVALUE—This parameter filters the results based on a specified value for a specified UDF that resides within a specified UDT. Any item that contains the value for the UDF is returned, unless parameters include an optional operator filter ( [.OPERATOR] in the expression provided). The lowercase filter operators of min or max are described later.

To To filter results using UDF information, use the following query structure:

To filter results using the name of a UDT, use the following query structure:

To filter results using UDF information that is part of a specific UDT, use the following query structure:

When filtering lists and using date or numeric UDF or UDT values, use operators to restrict a query. The following operators are supported:

• .min—This operator displays results that are greater than or equal to the specified value.

• .max—This operator displays results that are less than or equal to the specified value.

Examples:

• /api/v2/processes?type=Sample Prepandudt.name=Plasma—This URI returns all Sample Prep processes run in the system that have the Plasma UDT selected.

• /api/v2/processes?type=Sample Prepandudt.Plasma.Platelet Count.min=50—This URI returns all Sample Prep processes that have a UDF named Platelet Count with a value of 50 or greater, within a UDT named Plasma.

• /api/v2/processes?type=Sample Prepandudf.Sample=Serumandudf.Sample=Tissue—This URI returns all Sample Prep processes that have a UDF named Sample with a value of Serum OR a UDF named Sample with a value of Tissue.More examples of filtering exist in the Cookbook.

For more examples of filtering, see the Cookbook.

When filtering with UDT or UDF parameters, all special characters in the parameter string must be URL encoded. The Pipe ( | ) or the URL-encoded pipe ( ) cannot be used.

When filtering on a UDF that is configured as a Multiline Text UDF, if a value contains a hard return, the value must include the URL-encoded line feed () at the appropriate location. Depending on how API requests are issued (via a browser or a script), spaces in names or values may require URL encoding, and trailing spaces in a name or value always require encoding. For example, for results to be returned, ‘name ‘ requires ‘name’.

As of BaseSpace Clarity LIMS v5.0, several terms have been deprecated:

• External Program Integration Plug-in (EPP) has been replaced with automation

• EPP/AI node has been replaced with automation worker / AW node

• Parameter has been replaced with token

• User defined field (UDF) has been replaced with custom field

When a job is dispatched to the AI node/automation worker, the following steps occur:

1. A temporary working directory is created on the AI node / automation worker:

   • In AIInstallDirectory/temp/

   • With a unique name including the client process LIMS ID.

2. The command configured and selected as part of the step run in the LIMS is then sent to the AI node / automation worker, with any specified parameters / tokens replaced with actual values.

3. The command is executed on the AI node / automation worker, spawning step execution using the temporary working directory as the working directory context.

   • Script processing can use stdout, stderr, and return codes following standard shell programming packages.

4. When the script exits, the AI node/automation worker automatically retrieves any files with matching LIMS IDs from the temporary working directory. The files are attached to the appropriate output file placeholders.

How the API Infrastructure is Used

The automation API infrastructure can be used alone or with the REST API infrastructure.

For example:

• Simple scripts can use automation parameters/tokens and data files directly from the current working directory. They can write results back to the current working directory, associating them back to the relevant placeholders in Clarity LIMS.

• More advanced scripts can also use the REST API infrastructure to retrieve additional required information and place relevant data back into UDFs/custom fields. Advanced scripts can also attach and associate data files to placeholders, which may be in different locations, while the script is still running.

When multiple users are working on multiple plates in high-throughput labs, programmers may find that the large number of HTTP method calls to the REST API can slow down their scripts.

To improve performance, Illumina has created the following batch resources:

• artifacts.batch.retrieve

• artifacts.batch.update

• containers.batch.create

• containers.batch.retrieve

• containers.batch.update

• files.batch.retrieve

• files.batch.update

• samples.batch.create

• samples.batch.retrieve

• samples.batch.update

Use the batch resources to access a group of artifacts or a group of containers using a single batch method call. Using these resources to iterate a list of items significantly improves script execution times.

Key Concepts

Batch resources are best thought of as unordered collections or lists of items accessed. A POST to batch/create, batch/update or batch/retrieve, therefore, is a request to create, update, or retrieve those items. There is no guaranteed order to batch responses.

Batch resources are nonbreaking additions to the existing REST API. Updated scripts can still use their existing nonbatch methods.

For example, the resources may have URIs (Universal Resource Identifiers) such as:

Batch operations do not require sophisticated HTTP client or server methods. The only HTTP method for batch resources is POST.

What is sent in a batch resource POST?

To update a group of artifacts, use a POST operation to the /artifacts/batch/update resource. The XML input payload consists of a series of elements, as follows.

What is returned from a batch resource POST?

As large data transfers can affect performance, it is important to return concise XML in response to a batch resource request. Therefore, except for retrieve resources, the XML output payload consists of a list of created or updated URI links, such as the following:

Return codes

The batch resources use common HTTP return codes:

• An HTTP 200 (OK) code is returned when batch resources have been successfully created or updated.

• An HTTP 400 error code is returned if the input payload details included incorrect, mixed, or duplicate URI links. For example, if the details of an artifacts.batch.update (list) request included a container resource.

As of BaseSpace Clarity LIMS v5, the Operations Interface Java client, which is used by administrators to configure processes, consumables, user-defined fields, and users, has been deprecated. All configuration and administration tasks are now executed in the Clarity LIMS web interface.

In addition, several terms have been deprecated:

• External Program Integration Plug-in (EPP) has been replaced with automation

• EPP/Automated Informatics (AI) node has been replaced with automation worker / AW node

• Parameter has been replaced with token

Use step automations to trigger a command-line call on a process/step or a file attachment event. The steps required differ depending on the LIMS version.

This article provides an overview of the steps required to configure automations and automation triggers. For detailed version-specific instructions, see the following documentation:

• Clarity LIMS v6 reference guide > Configuration > Automations

Configuration (Clarity LIMS v5 and later)

1. On the main menu bar, click Configuration, and then click the Automation tab.

2. On the Automation configuration screen, on the Step Automation tab, add a new automation:

   • Name the automation.

   • Set the channel name.

   • Define the command line.

   • Enable the automation on the desired steps.

3. On the Master Step Settings or Step Settings screen of the related step, set the following:

   • Trigger Location—The stage at which the script it is to be initiated (beginning of step, end of step, on entry to/exit from a screen, etc.).

   • Trigger Style—How the script is to be initiated (automatically or manually when the user selects a button in the interface).

Additional resources

Overview

To trigger scripts and third-party programs from the BaseSpace Clarity LIMS user interface, use command-line calls configured in step automations.

There are many ways to use automation in Clarity LIMS. Consider the following examples:

• Automate sample tracking and enhance the information recorded.

• Generate and attach specially-formatted text files.

• Simplify data entry.

• Automate the population or updating of data fields and data files.

Before using automation, become familiar with the topics discussed in this article and understand how automation functionality interacts with users, Clarity LIMS, and REST.

As of BaseSpace Clarity LIMS v5.0, several terms have been deprecated:

• External Program Integration Plug-in (EPP) has been replaced with automation

• Automated Informatics (AI) node has been replaced with Automation Worker (AW) node

• Parameter has been replaced with token

Common applications of automation

Automation scripts are often used to automatically create and attach specially-formatted text files, such as the following:

• Files containing sample lists (sometimes called instrument driver files). Users can import these driver files into control software, saving time and ensuring accurate sample processing.

• Barcode or label files. These are specially-formatted files that can be supplied to barcode software systems to allow users to print out container labels.

• Summary analysis results - for example, from alignment or molecular identification and quantification algorithms.

The two common applications of automation are:

• To create files and attach them to process output placeholders.

• To update data fields with information created during data analysis.

Scripts triggered by automation also use REST to update information directly within the REST resources.

How Users Trigger Scripts

Record lab work in Clarity LIMS by running steps on samples. These steps may be configured in Clarity LIMS by an administrator.

Most steps can be configured with an automation trigger that invokes an external script. The script may include fixed and variable information parameters/tokens on the command line.

NOTE: As of Clarity LIMS v5.0, the term command-line parameter has been replaced with token.

Configuring Processes / Steps to Trigger Scripts

After an AI/AW node is installed, processes (in Clarity LIMS v4.2 and earlier) or steps (Clarity LIMS v5 and later) must be configured to call out to it.

This configuration is executed by the Clarity LIMS administrator:

• In Clarity LIMS v4.2 and earlier, execute configuration in the Operations Interface process configuration dialog on the External Programs tab.

• In Clarity LIMS v5 and later, execute configuration on the Automation configuration screen.

When configuring an automation, the following must be defined:

• Name

• Channel

• Command line call

NOTE: Only a brief summary of automation configuration is provided here. This material should be familiar from the Clarity LIMS administration training.

Configuring Scripts that are Called by Automation

Scripts or third-party programs are called using the operating system command line. They must meet the following requirements:

• Be callable on the command line and, preferably, be able to read and respond to command-line parameters.

• Be accessed by the user account running the automation, with appropriate permissions and disk locations.

• Exit with appropriate exit codes, otherwise the automation may record the script completed with an error.

How Information Flows Between a User and a Script

The following diagram illustrates what happens when a user runs a step in the LIMS.

Step 1 - A User Runs a Step

The lab scientist tracks activities by running a step in Clarity LIMS. The step is configured to display a button that invokes the configured automation script.

Step 2 - Server Creates the New Process / Step

NOTE: As of Clarity LIMS v5.0, the term parameter has been replaced with token.

Common applications of automation

The application server creates a new step, which is much like POSTing to the processes resource of the REST API. The server resolves any parameters/tokens found in the string and sends the resolved command-line string to the automation.

In this example, two of the most common parameters/tokens used when working with automation are discussed:

• {processURI:version:scheme}—This parameter/token passes the REST API URI of the step that issued the command-line string.

• {ouputFileN}—This parameter/token passes the LIMS ID of the specified expected output file of the step that issued the command-line string. You can use 0 (zero) for the first file, 1 (one) for the second file, etc.

For more information about the parameters/tokens available for use, refer to the articles in the following API documentation sections:

NOTE: As of API v1 r12, the version and scheme values for {processURI:version:scheme} are automatically populated, based on the REST version and protocol of the deployed server.

Step 3 - The Automation Invokes the Operating System Shell Command

The automation program receives the command-line string from the application server. It may also receive other process(step)-related information, such as temporary files. The command-line string is executed by the operating system of the host computer.

Steps 4 to 7 - The External Program Executes (Optionally Calling REST Resources)

The automation can work with any third-party program that supports command-line parameters/tokens. The program may simply create files or it may manipulate information directly via the REST API (steps 5 and 6).

Simple automation operation does not require anything of the REST API. If a third-party program creates files that users would like brought back into Clarity LIMS, scripts should use the outputFileN parameter/token to specify that the program create file names that are expected by the client. The files are placed in a temporary local working directory and automatically imported into the client. With this method, the automation automatically handles many of the things that would need manually scripts using the REST API processes, artifacts, files, and glsstorage resources.

For more complicated scenarios, you may want to use automation with the REST API. This situation is where the processURI parameter/token is used. A GET request on the URI of the step that issues the command-line string provides all the information recorded by the user. This information includes links to the analytes (samples) used as inputs. The script can then use other REST API resources to create or update information.

On completion, the third-party program exits (step 7). Standard shell exit codes apply: zero (0) equals successful completion.

Step 8 - The Automation Returns (Optionally Attaching Created Files)

On exit of the third-party program, the automation software updates the application server. If the system finds files with names that match the file placeholders produced by a process/step, the files are uploaded to the file server and attached to the appropriate placeholders.

A nonzero exit code sets a flag on the step, indicating that there is an error.

Step 9 - The User Accesses Updated Results

With updates complete, the application server sends refresh events to the Clarity LIMS. The user will see that files have been uploaded.

Verifying Automation Scripts

Verifying and testing scripts is an important part of working with automation. Remember that there are three software components:

• The server

• The automation instance (calling scripts)

• The script

The best way to debug scripts is to unit test each component separately. For example a logical order to work on a script is as follows:

1. Define and test the REST calls required in a web browser.

2. Define the command-line parameters / tokens sent to the automation at step completion.

3. Test the script running just from the command line.

4. Test automation calls to the script at step completion by running the step from the LIMS interface.

Installing an Automation Worker / AI Node

Before the system can use the automation, a system administrator installs one or more automation workers / AI nodes within the lab network. The installation typically occurs on the server that contains the script program, or third-party application, to be integrated.

The installer program is contained in the Automated Informatics / Automation Worker software package.

The information recorded in BaseSpace Clarity LIMS is organized into resources within the REST API. Each resource refers to an XML schema associated with a namespace. Before working with the REST Web Service, understand how the information recorded in Clarity LIMS translates to the REST resources.

The following diagram highlights the major REST resources. Each resource is discussed further in the following sections.

• Samples are the objects that are entered into the LIMS before processing begins. Every sample belongs to a single project and has a related analyte (sample) artifact. Every project must have an associated researcher.

• When you add a sample to the system, it is classified as a submitted sample. This allows the original samples, and any related data, to remain separate and distinct, even as processing and aliquoting occurs. Every sample or file created by running a step from the LIMS user interface can be traced back to a submitted sample.

• In Clarity LIMS, processes (know as steps in the user interface) are run on analyte (derived sample) artifacts. Samples must always be in containers.

• Clarity LIMS v4.x and earlier: In the Clarity LIMS Operations Interface processes are run on analyte (sample) or result file artifacts. Samples must always be in containers.

  • As of BaseSpace Clarity LIMS v5, the Operations Interface Java client used by administrators to configure processes, consumables, user-defined fields, and users have been deprecated. All configuration and administration tasks are now executed in the Clarity LIMS web interface.

  • To understand how API terminology maps to terminology used in the Clarity LIMS v5 interface, see .

Samples

Within the REST Web Service, the samples resource is key.

The samples resource represents submitted samples and contains information about those samples, including:

• The dates samples are entered and received.

• Any user-defined data related to the samples.

While the artifact associated with a submitted sample is only seen at the database or REST level, and is never exposed in the LIMS interface, the system uses this artifact when running protocol steps.

When running a step on a submitted sample, the artifact is used as an input to the step, and not the submitted sample itself. All artifacts reside within the artifacts resource.

When a submitted sample is processed, the system generates output artifacts. Depending on the configuration of the process, many types of artifacts - including result files - can be generated. Any downstream sample created by running a process is considered an analyte artifact (referred to as a derived sample in the user interface).

Projects

Projects are used to group samples based on the originating lab (account) or study. Projects collect all records related to that sample in the LIMS.

A project stores information about:

• The client (researcher) who owns it

• Significant dates

• The status of the project

• Any user-defined information that the lab needs to collect

After creating a project, you can add samples to it. Samples can then be added to workflows, and steps (processes) are run on those samples to reflect the analysis performed in the lab.

In the REST Web Service, a submitted sample can only belong to one project. You can use the projects resource to return projects.

Note the following details regarding projects:

• Every submitted sample must belong to a project.

• Every project must be assigned to a researcher (an owner) that corresponds to a client in the system.

• NOTE: In the LIMS user interface, the term Contact has been replaced with Client. However, the API permission is still called contact.

The researchers resource represents clients in the system.

When working with projects, each project must list a client as the owner of the project. This role generally represents the person who submitted the original samples.

The client does not need to have a user account.

Processes

• Clarity LIMS v5 and later—Created in the Clarity LIMS web interface, master steps model and track the work performed on the samples in the lab. These master steps are then used as building blocks to create and configure steps. These steps are known as processes in the API.

Different interfaces may allow you to run steps/processes on different artifacts.

In the API view, a process takes in one or many analytes and/or result files and creates one or many analytes and/or result files.

When running a step in Clarity LIMS, lab scientists record information about the step, the instruments used, and the properties and characteristics of the samples.

Depending on the configuration of the process/master step on which it is based, the step can generate another sample analyte and/or placeholders to which result files can be attached for storage in the system.

With the REST Web Service, the processes resource is used to track these activities.

Note the following details regarding processes:

• Processes are used to represent work that occurs in the lab or in silico.

• Processes take inputs and create outputs. With the REST processes resource, this is modeled using the input-output-map element.

In addition to tracking historical work via the processes resource in the REST Web Service, use the service to POST new processes to the system.

POSTing a process to the REST Web Service creates the process itself, along with the outputs of the process. However, all the input and output containers must exist in the system already.

• For a simple example of the XML required to POST a process, see the processes (list) section of the REST resources space.

• For basic details about POSTing processes, see Working with Processes/Steps in the Cookbook section.

• For examples of process POSTing, see Pooling Samples with Reagent Labels and Demultiplexing in the Cookbook section.

• To find out how to integrate automation with process POSTing to set quality control flags, see the Setting Quality Control Flags application example.

Query the processes resource using input artifact LIMS IDs. This query allows you to find the processes that were run at each step in the workflow or on each artifact generated during processing.

Artifacts

All inputs and outputs of a process are artifacts, and can be returned via the artifacts resource.

Note the following details about artifacts:

• An artifact is a derivative of a sample and is used as an input to a process.

• An artifact may be a sample analyte or a result file.

• The artifacts resource includes artifacts for the submitted sample and all process outputs, both file- and sample-based.

Artifacts are categorized by type, to distinguish between pure information results (file-based artifacts, such as result files) and the biological material created by processing the sample (analyte artifacts).

In Clarity LIMS, the term artifact is used to describe items needing to be processed. Think about artifacts as the intellectual property added by the lab.

For example, applying reagents to change the nature of a sample creates an artifact, as does generating and analyzing data files by running a sample on a NextGen or microarray instrument.

Anything created by a process in the system is an artifact. In the REST Web Service, there are several types of artifacts, but this article focuses on two:

• Computer-generated files called result files

• Physical sample derivatives called analytes.

The high-level relationship between artifacts, analytes, and result files is shown in the following diagram.

An artifact references data elements, which vary depending on the type of artifact you are working with. For example, a result file has an attached-to URI that links to a files resource, whereas an analyte has a location URI that links to the containers resource.

Artifacts are key to tracking lab process activities and also link to a submitted sample.

All artifacts include one or more sample URI data elements, which make it easy to trace any lab-generated product or result directly back to its original sample.

When working with artifacts in the REST API, their URIs often include a numeric state. The state is used to track historical QC, volume, and concentration values.

Unless you are interested in a historical state, it is best practice not to include state when using an artifact URI. When state is omitted, the API defaults to the most recent state.

Containers

When samples are processed in the lab, they are always placed into containers of some sort (tubes, 96-well plates, flow cells, etc.) and moved into new containers as processing occurs.

For many kinds of processing, the container placement is a critical piece of information. Further processing of the sample, and data files created by analyzing the sample, are often linked based on the placement of the sample in the container.

Containers are central to processing in the lab. In Clarity LIMS, therefore, the samples (analyte artifacts) must also always be placed into a container resource.

When working with the REST Web Service, analyte artifacts include a URI that links to the container housing the artifact. Use the containers resource to view all the containers registered in the system.

Note the following details about containers:

• Containers represent the tubes, plates, flow cells, and other vessels that can be populated with a sample.

• All samples/analytes must reside in a container or they will not be visible in the LIMS client.

All containers include a name and a LIMS ID.

• The name is a text element over which the scientific programmer has full control.

• The LIMS ID is a unique identifier generated by the system in a fixed format.

For assistance, the Illumina Consulting team can recommend various settings, such as uniqueness constraints, based on your requirements.

Files

A lab produces various files: large scientific result data files, summary result files, image files, label files, equipment and robotic setup files, and software logs.

These files are stored in different locations and it can be challenging to manage the relationship between a file on a computer or hard disk and the sample, step, or project with which it is associated.

Clarity LIMS lets you store files related to a project or sample and files generated during a step in a workflow. These files can be imported in various locations within the client and are stored on the file server.

To model this feature within the REST Web Service, there are two resources:

• files resource

• glsstorage resource

Within the REST Web Service, files are represented by the files resource. This resource manages files and the resources or artifacts to which they are related, and stores information about:

• The sample, project, or process output with which the file is associated, referenced by the attached-to URI.

• Where the file was imported from, and its original name, referenced by the original-location URI.

• The location of the file, referenced by the content-location URI. It also specifies the transfer protocol that can be used to retrieve the file. The following transfer protocols are supported:

  • ftp

  • sftp

  • HTTP

Files added using the LIMS client

If you are using REST to view a file that was added through the LIMS client, the content-location URI will reference a location on the file server. This location is where the system stores all files that are imported through the Clarity LIMS client.

Files added using REST

If you are using REST to import a file into the system, do one of the following:

Store the file on the file server:

1. Use the glsstorage resource to create a unique storage location and file name on the file server.

2. After this step is complete, the system returns a location and file name using the content-location URI element.

3. Then do as follows.

   • Provide the URI to the files resource.

   • Put the file in the specified location.

Store the file somewhere other than on the file server:

• Use the files resource and reference the name and location of your file with the content-location URI element.

• This feature must be configured by Illumina. For more information, contact the Illumina Support team.

Not the following key concepts:

• Files: The files resource defines the location of a file and its relationship with other REST resources, such as artifacts and projects.

• Glsstorage: The glsstorage resource allocates space on the file server.

• XML elements: Within the XML used by the files and glsstorage resources, the attached-to and content-location URIs are used to link disk files to file-based artifacts produced by a process, or to link disk files to projects or samples.

The following diagram outlines how the XML elements link files to system resources and artifacts:

Using the REST Web Service to Work with Files

In the lab, one of the most important associations that must be made is between:

• A file that is the result of an instrument run

- and -

• The sample that was analyzed to produce that file.

In Clarity LIMS, this association is represented by creating a process that takes a sample analyte and produces a result file.

When you run a process configured to create a result file, the process generates a placeholder for a file. To populate the placeholder, simply import the result file generated by the instrument into Clarity LIMS.

While working in the lab, lab scientists can upload result files that are used or produced while samples are processed. However, it may sometimes be more appropriate to automate this work. In these cases, you can use the REST files and the glsstorage resource.

Depending on the file storage needs and how the files are generated, there are two ways to do this process.

• Import a file and store it on the file server.

• Import a file and store it on a different server.

Import a result file and store it on the file server:

1. POST conforming XML to the glsstorage resource.

2. This action returns XML that includes a name and storage location for the file.

3. Place the file into the specified location using the file name provided in the XML.

4. POST the returned XML to the files resource, which links the file on disk to the result file placeholder.

Import a result file and store it on a different server:

1. Make sure that the file exists in the desired location.

2. POST conforming XML to the files resource, referencing the name and location of your file with the content-location element. The file path must contain the transfer protocol supported by the server. For example: sftp://192.168.13.247/home/glsftp/Process/2010/10/SCH-RAA-101013-87-1/ADM53A1PS3-40-1.dat NOTE: It is not necessary to POST to the glsstorage resource.

Attaching Reference Information

If you have files that were not generated during the analysis of a sample, you can also attach reference information to projects and samples.

For example, suppose you receive an e-mail when a sample is submitted to the lab, you may want to store that information in the LIMS. In this case, when you POST XML to the files resource, the XML links the file to the desired submitted sample – instead of to a result file placeholder.

Clarity LIMS v5.x and later:

In Clarity LIMS, the file is attached to the Sample Details section of the Sample Management screen.

1. On the Projects and Samples screen, select the project containing the sample for which you have posted a result file.

   • Scroll down to the Samples and Workflow Assignment section of the screen and select the appropriate sample.

   • Select Modify 1 Sample.

2. On the Sample Management screen, scroll to the bottom of the Sample Details section to find the attached file.

Before Clarity LIMS v5:

In the Clarity LIMS web interface, the file is attached to the Sample Details section of the Sample Management screen.

• For details on accessing the file, see the previous content on Clarity LIMS v5.x and later.

In the Clarity LIMS Operations Interface, the file is attached to the Files tabbed page of the applicable submitted sample.

1. In the Clarity LIMS Explorer, click Opened Projects.

2. In the Opened Projects list, double-click the project containing the sample for which you have posted a result file.

3. On the project details page, click the Samples tab.

4. At the bottom of the tab, in the Containers pane, double-click the appropriate sample.

5. On the sample details page, click the Files tab to find the attached file.

The REST Web Service separates the resources needed for files and file storage.

This action allows for greater control and the flexibility to apply various tracking and storage strategies. The content-location element can be used to define the file location without having to move the file. This ability is key in next-generation sequencing, which requires the management of large files, such as assemblies.

The content-location element needs to reference the file location in a storage system using a specific file transfer protocol. Currently only FTP, SFTP, and HTTP protocols are supported.

This mechanism makes file management flexible, but it maintains access to the file from REST with a single link. However, this feature must be configured by Illumina. For more information, contact the Support team.

User-Defined fields (UDFs)

Note the following key concepts about UDFs and UDTs:

• UDFs and UDTs are configured to collect information that is important to the lab.

• With the REST Web Service, you can include UDF and UDT values in the XML representation of any individual resource that has a UDF or UDT defined.

• Not all artifacts have both UDFs and UDTs.

In Clarity LIMS v5 and later:

• The API still uses the term udf the term. However, in the user interface, this term has been replaced with custom field.

• UDTs are not supported.

You can configure the system to collect user-defined information. Consider the following examples:

• You can create UDFs to add options and fields to the user interface when working with samples, containers, artifacts, processes/protocol steps, and projects.

• You can also create User-Defined Types (UDTs), which are organized subsets of related UDFs. As you add and process samples, you can add information to these options and fields.

In the following example, UDFs are added to submitted samples, processes, and sample analytes (derived samples).

• For the submitted sample named Goo, there are UDFs named Type, Color, and Source.

• For the Prepare Goo process/step, there are UDFs named Reagent Lot ID, Temperature, and Cycle Time.

• The output of the Prepare Goo process/step is an analyte named Prepared Goo, which contains UDFs named Quality and Category.

To record information for these UDFs:

1. Add Goo to the system and populate the sample-level UDFs.

2. In Clarity LIMS, run the Prepare Goo step and complete the following actions:

   • Populate the Step Details fields (the process-level UDFs).

   • Populate the Sample Details table (the analyte-level UDFs).

Using the REST Web Service to collect UDF information

You can also use the REST Web Service to collect user-defined information for samples, containers, artifacts, and processes.

After you have configured UDFs, the XML of the appropriate resource expands with data elements for the field values. For example, the Prepared sample of goo artifact would have the following XML:

Configuring UDFs / Custom Fields

UDFs/custom fields are useful for collecting data at various stages of your workflow. In next-generation sequencing, it is important to record information, such as who submitted a sample, the tested concentration of a library, the reagents that were used during library prep.

As illustrated in the previous Goo example, collect this information by adding UDFs/custom fields for the samples, artifacts, and processes resources:

• A submitted sample UDF / custom field named 'Type'

• An artifact-level UDF / derived sample custom field named 'Validated Concentration'

• A process-level UDF / master step field named 'Reagent Name'

Artifact UDFs/custom fields are flexible.

• You can configure different sets of UDFs / custom fields for the analyte artifact type and the result file artifact type.

• You can configure different sets of UDFs / custom fields based on the process type.

This flexibility means that:

• Process type / master step A can display fields 'm' and 'n' on a result file, and fields 'q' and 'r' on an analyte

• Process type / master step B can display fields 'm' and 'o' on a result file, and fields 'q' and 's' on its output analyte.

Control how users access artifact-level UDFs/custom fields by configuring the type of artifact or process type/master step to which they apply.

Not every detail tracked and recorded needs a UDF. To optimize lab efficiency, it is recommended that you define an essential UDF set.

Increasing the complexity of information collected and managed does not necessarily improve operations or scientific quality. It may be more effective to store files, because the complete details are then available and secure within the attached file.

As an API programmer, it is important to understand the difference between steps and stages. This distinction is especially important because the concept of stages is hidden from the end user. As such, when receiving requirements from end users, steps sometimes means steps. At other times, steps mean stages. This article highlights the differences between these two entities.

The Life Cycle of a Sample

We tend to think of a protocol as being a linear collection of steps, as shown below.

Figure 1

However, this illustration is not complete as the life cycle of a sample modeled within Clarity LIMS reflects what happens in reality. The workflow is broken into periods of activity and inactivity. If a workflow is comprised of three steps (A, B, and C—as shown in Figure B), Step B does not begin at the exact time that Step A is complete.

Stages and Steps

To reflect these inactive periods, Clarity LIMS uses the concept of stages in addition to steps. A more complete representation of a workflow is shown below, with the stages occurring between the steps.

Figure 2

The following phrase simplifies this concept:

If the sample isn't active in a step, it's waiting in a stage.

NOTE:

The Clarity LIMS concept of the virtual ice bucket is another state that occurs when a sample leaves a stage, but work on the step has not started. This scenario is represented in the Figure 3, with the virtual ice bucket appearing between stages and steps, as the sample moves from left to right. However, virtual ice buckets are largely irrelevant to this discussion. While recognizing their existence, they are discounted from further explanation.

Figure 3

Having simplified our model of a sample passing through a workflow to resemble Figure 2, we can now add the next layer of complexity.

Protocols are components of workflows. As such, it is easy to imagine two or more workflows sharing a protocol. This detail leads to the following summary:

Steps belong to protocols, whereas stages belong to workflows.

This summary means that the stages that exist between steps are part of the workflow (represented in Figure 4 below). For example, samples passing through Workflow O proceed through Step A, Stage X, Step B, Stage Y, and Step C.

Samples passing through Workflow P (which shares the protocol with Workflow O) pass through the same steps. However, samples pass through a different set of stages (Stage X' and Stage Y').

Figure 4

Why Stages Are Important

Looking at the counts of samples associated with steps in a protocol (for example, in the Lab View dashboard in Clarity LIMS), the number of samples awaiting a particular step is actually the total number of samples across all relevant stages that feed into the step.

For bioinformaticians and programmers who are using the Clarity LIMS API, stages have an additional function. Route samples in ways that vary from the expected, linear route by manipulating which stages the artifacts are in. For example, using the API via a script, do the following actions:

• Implement a forking workflow by assigning artifacts to one (or more) additional stages.

• Create iterative (or looping) workflows by routing artifacts to an earlier stage for additional work.

The BaseSpace Clarity LIMS Rapid Scripting™ API adapts lab informatics using the Clarity LIMS platform.

It is important to integrate scripting into the overall processes. Begin by identifying any areas that may require adaptation to fit the lab workflow. It also helps if users are involved in the early stages of the software system analysis process.

Most scripts in an implementation are finalized towards the end of the process, as the full impact and benefits of the new system become clear.

Take some time to become familiar with the user interface, learn how to configure the product, and work with the tools that the lab uses. Also, establish the workflows and the configuration of the system before investing in API scripts and automation.

Start with Administrator Training

New customers receive administrator-level training before working with the API.

If you are not comfortable configuring steps, custom fields, containers, etc., in Clarity LIMS, you may find the API material difficult to understand. Contact Illumina for more information on administrator training and training materials.

If you are not comfortable configuring steps, custom fields, containers, etc., in the LIMS, you will find the API material difficult to understand. Contact Illumina for more information on administrator training and training materials.

Defining the Lab Solution

Before committing time and resources to using the API, it is important to define what you would like to accomplish. Understanding the key outcomes, use cases, users, and constraints of the lab helps with learning the API more quickly and improves efficiency.

If you require assistance, Illumina can provide expert resources to audit and analyze the laboratory users, processes, workflows, instrumentation, data production, and environment. This careful and focused analysis results in a requirements specification that provides extensive value to the facility.

Related Resources

Clarity LIMS automations typically call scripts or third-party programs written for a shell or command-line interpreter, of either a Linux or Windows operating system (OS). Although the use of any system shell if acceptable, Bash is recommended.

Depending on the systems that integrate with the given automations, various restrictions apply to the string parameters/tokens and formatting used in the automation command line.

As of Clarity LIMS v5, several terms have been deprecated:

• External Program Integration Plug-in (EPP) has been replaced with automation

• EPP/AI node has been replaced with automation worker / AW node

• Parameter has been replaced with token

Environment variables

Environment variables can be used to aid in configuration. However, automation commands are generated with a limited shell. For full access to environment variables, the recommended practice is to start the command to instantiate a 'full' user shell. For example, for Bash use the following command:

This procedure provides the following advantages:

• Ensures updates of the environment variables, removing the need for repeated AI node/automation worker restarts.

• Ensures access to all environment variables, including the full path to Groovy.

• Allows certainty of the shell being used.

Operating system shell formatting, spaces, & special characters

The various operating system (OS) shells each have their own rules and regulations. When creating command-line strings, be aware of the considerations described in the following sections.

Operating System Shell Formatting Differences

Windows shell command-line interpreters require different syntax and formatting than Linux shell variants. For example, the following scripts are identical, but are formatted for different AI nodes/automation workers running on different operating systems.

On Linux:

On Windows:

Most of the examples in this specification use Windows formatting, because Windows is the most common platform found in the lab.

Spaces

Spaces in paths, file names, or parameter/token data can cause commands to be misinterpreted as information passes between systems.

Many OS shells automatically parse command-line contents by space, which cannot be what is intended. Enclose commands in double quotes " " to avoid misinterpretation of spaces by the OS shell command-line interpreter.

Special characters

OS shell command-line interpreters can attempt to interpret and act upon certain special characters, rather than passing them along as textual information. A character can have a rule applied to it within one OS shell environment, and a different rule under another environment. To use a character in its literal form, escape the characters. The escape character used varies depending on your OS shell. The most common escape character is the backslash character.

The most common OS shell characters that require escaping are:

This section provides information to help you work with Clarity LIMS automation tokens in Clarity LIMS v5 and later.

• Derived Sample Automation Tokens

• Step Automation Tokens

• Project Automation Tokens

Scripts produce a numeric code on exit. By convention and by default, a successful exit has a code of 0 (zero). Within error handling, select different nonzero exit codes to indicate various error conditions.

NOTE: For Clarity LIMS v5.0, the term External Program Integration Plug-in (EPP) is deprecated and replaced with automation.

Script Error Logging and Standard Streams

Though logging information in the Clarity LIMSuser interface is useful, scripts can also write debugging/troubleshooting information into the automatedinformatics.log file using stderr (standard error). When a line is printed to stderr, a [WARN] line is written to this log, which is useful for troubleshooting interactions between the script and the automation programs. For more information on the automatedinformatics.log file, see Troubleshooting Automation. For more information on using stderr on the command line, refer to Unix and Windows documentation on standard I/O streams, especially standard error.

Error handling

In Clarity LIMS, the last line written to stdout (standard out) is automatically captured and shown in the interface.

If the exit code is zero, the message displays in green. If the exit code is nonzero, the message displays in red.

The Operations Interface Java client is deprecated in Clarity LIMS v5. All configuration and administration tasks are currently executed in the LIMS web interface.

If the script exits with a nonzero code, a sample genealogy flag is automatically added to the process outputs, with a standard error message indicating there is an External Program Error.

If the script is complex, or includes several error conditions, configure an additional result file process output (eg result.log) designed to capture status and error information. Make the external script write additional information to this file. If an error occurs, the file is still captured in the client, and is available to view for troubleshooting purposes.

The automation and integration of the day-to-day work in the lab requires different Automated Informatics (AI) nodes/automation workers to perform different tasks.

For BaseSpace Clarity LIMS v5.0, several terms are deprecated:

• Automation replaced External Program Integration Plug-in (EPP).

• Automation Worker (AW) node replaced EPP/AI node.

Specifying Channels

Channels are manually named, and ideally clearly represent the task performed (eg. Type_2_Analysis). To make sure that dispatched automation work is routed to the correct destination, specify a channel in the following places:

• On the AI node / automation worker

• Clarity LIMS v5 and later: When configuring step and derived sample automations on the Automation tab

How Channels Work

1. When the automation trigger conditions are met in the LIMS, the automation job first enters a channel-specific 'first in, first out' (FIFO) queue of work for completion.

2. Jobs queue in this channel until one of the AI nodes/automation workers operating on the channel completes its previous work, and indicates it is free to accept more.

3. The next job is then dispatched from the channel queue to the node. This strategy allows a single channel queue to receive service by one, or many, AI nodes/automation workers servicing the specified channel.

It is possible to have multiple AI nodes/automation workers performing the same type of work all configured on the same channel, allowing a simple but effective way to increase throughput of a particular analysis bottleneck, or to ensure redundancy during a single node failure.

When configuring automations in BaseSpace Clarity LIMS, copy tokens from the Tokens list and paste them into the Command Line field. These tokens are available for use in step automations. If using multiple variables, add a space between each entry. All tokens and parameters are case-sensitive.

When configuring automations in the BaseSpace Clarity LIMS, copy tokens from the Tokens list and paste them into the Command-Line field.

These tokens are available for use in derived sample automations. If using multiple variables, add a space between each entry. All tokens and parameters are case-sensitive.

When configuring automations in BaseSpace Clarity LIMS, copy tokens from the Tokens list and paste them into the Command Line field.

These tokens are available for use in project automations. If using multiple variables, add a space between each entry. All tokens and parameters are case-sensitive.

This section outlines several strategies to enable this feature.

In all cases, assume that a UDF called Batch ID that was on Step A, and you want to access it on Step D:

NOTE: If the samples in Step D do not have a homogeneous lineage, expect multiple values for the Batch ID.

Scenario 1: Crawl Back

This method involves crawling backwards from Step D to Step A.

The general form is as follows.

1. Examine the inputs to Step D.

   Each input (I) has a parent-process element with a URI to the step that created the artifact. In this case, it is the URI to Step C.

2. Get the input-output maps for Step C (from the /details resource) and find the input (I') that produced output I. Each input (I') has a parent-process element with a URI to the step that created the artifact. In this case, it is the URI to Step B.

3. Get the input-output maps for Step B (from the /details resource) and find the input (I'') that produced the output I'. Each input (I'') has a parent-process element with a URI to the step that created the artifact. In this case, it is the URI to Step A.

4. Get the value of the UDF (Batch ID) from Step A: 1234.

This method is computationally slow, but it is safe. As the number of steps that need to be crawled back through increases, so does the duration of the script to retrieve the value.

Scenario 2: Jump Back

This method tried to jump straight to Step A, without passing through Steps B and C.

The general form is as follows.

1. Examine the inputs to Step D. Each input (I) has a sample element that contains the limsid (S) of the related submitted sample.

2. https://<your_hostname>/api/v2/artifacts?samplelimsid=Sandprocess-type=Step%20A

   This query should give an XML response containing the URI to Step A. From there, get the value of the UDF (Batch ID): 1234.

This method makes two assumptions:

1. That Step A produces analytes (derived samples). Thus, if Step A is a QC process, or does not produce analyte outputs, this method fails.

2. That the analytes (derived samples) resulting from S only passed through Step A one time. If this assumption is not true, you receive multiple URIs to the individual instances of Step A that relate. Also, you cannot be certain which Batch ID to rely upon.

This method is computationally fast, and its duration is not reduced if there are many steps between Step A and Step D.

Scenario 3: Pay it Forward

This method works well, but it involves making configuration changes to the steps. As such, this method is useless for legacy data resulting from samples that passed through the steps before the configuration was applied.

Its general form involves:

• In Step A: Add a script that copies the value of the Batch ID UDF (1234) to every input and output of type analyte in the step.

• In Step B: Add a script that copies the value of the Batch ID UDF (1234) to every output of type analyte in the step.

• In Step C: Add a script that copies the value of the Batch ID UDF (1234) to every output of type analyte in the step.

• In Step D: The inputs contain the value of the Batch ID.

This method relies on propagating the Step UDF through Steps A, B, and C to Step D. It is safe and fast. However, if the protocol is edited and a new step is inserted between B and C, add the script that propagates the value. This addition is so the chain does not break. This method is safe if any of the steps are QC steps or do not produce analyte outputs.

Scenario 4: Along for the Ride

This method is a niche solution, but it works well. It assumes that the samples from Step A proceed to Step D as an intact group, and they are joined by a control sample.

This method involves making configuration changes to the steps. As such, this method is useless for legacy data resulting from samples that passed through the steps before the configuration was applied.

• In Step A: Identify the control sample for the group, then copy the value of the Batch ID to the control sample.

• In Step D: Identify the control sample for the group, then retrieve the value of the Batch ID from it.

This method is the least work, but it does make several assumptions that might make it impracticable.

Use the setExitStatus.py Python script, attached to this page, to test and simulate the use of the automation triggers within Clarity LIMS.

The setExitStatus.py script uses the following command-line parameters:

An example of a parameter string that invokes this script from Clarity LIMS is provided in the following. Note the use of the stepURI token in the -l parameter.

python /opt/gls/clarity/customextensions/setExitStatus.py -l {stepURI:v2:http} \

-u {username} -p {password} -s "OK" -m "successful"

Attachments

glsapiutilv2.py:

setExitStatus.py.txt:

This section discusses methods for integrating BaseSpace Clarity LIMS with upstream sample accessioning systems.

The following illustration shows a typical architectural overview:

Creating a Sample in Clarity LIMS

Required:

• A sample must have a Name / ID

• A sample must be associated with a Case / Patient / Study / Project

• A sample must be associated with a Container (Tube / Plate etc)

Optional (but expected):

User-defined fields (UDFs)/custom fields (defined by your LIMS configuration)

Typical flowchart of actions within the broker:

The following animation illustrates the elements of an XML sample-creation message to Clarity LIMS.

Common options for the broker

Build your own:

• Pro: Not too difficult

• Con: Stability as number of messages increases

• ?: Maintainable over the long-term

Use a commercial / open-source offering (e.g. Mirth Connect)

• Pro: Quicker than build

• Pro: Robust, multi-threaded support for millions of messages per day

• ?: May prove to be an excessive or over-complicated means to accomplish something relatively simple

Does the broker need to carry out other business logic?

For example, one customer added logic to their broker that dealt with medical billing and was able to distinguish between physicians ordering duplicate tests for a subject (not reimbursable, therefore the duplicate sample wasn’t submitted to Clarity LIMS), versus a temporal study that was reimbursable.

The best practice is to take advantage of as many legacy systems as possible, rather than creating samples in Clarity LIMS, then reinventing business logic to remove unwanted ones.

This section provides tips and tricks to help you work efficiently with the API. For example, learn how to copy and update field values, create and rename samples, work with files and QC flags, and automate BCL conversion.

Compatibility

Clarity LIMS v4 and later

Automation is powerful and simple in design. However, its applications can quickly become complex. We recommend you keep your scripts simple. When troubleshooting, the best practice is to isolate the issue to determine the source. The Automated Informatics (AI)/automation worker log file, automatedinformatics.log, is useful for isolating system components and diagnosing problems.

Isolate System Components

Isolate the behavior of each component in the system. In particular, determine if the following components can be ruled out as causing of the problem:

• The script or program—Running the custom logic, REST calls, and file handling.

• The AI nodes/automation workers—Calling the command line and invoking the script or program.

• The network—Providing reliable and timely TCP/IP packet transfers.

• The client—Completing the process/step and notifying the server.

• The server—Responding to client notifications and dispatching to AI nodes/automation workers.

Start with the Script

The script provides many options for troubleshooting. For example, increase logging to rule out unexpected behavior.

• Printing to stderr in the script writes a line to the automatedinformatics.log file. This file is a great source of information.

• The records in the log file allow for emulating the command-line call for unit testing the script, and calling it manually on the command-line prompt (see ).

• helps verify the automation program on the AI node/automation worker. If the script and the AI node/automation worker are functioning, review the log file entries for any warning (WARN) or error (ERR) lines near the time-stamp of the process completion event sent from the client.

• If the issue is related to the client or server software, contact the Illumina Support team, providing:

  • The automatedinformatics.log file

  • The server log

  • The results of the isolation tests (in the previous section).

Validate AI Node/Automation Worker

Use the following steps to test and verify the setup.

Test and Verify Setup on a Windows AI Node/Automation Worker

1. Create a process/step that generates a result file.

2. Configure an automation on the process/step. Associate it with the channel on which the AI node/automation worker is configured to communicate.

3. Add the following command line string.

   cmd /c "C:\ai\ai.bat {outputFile0}"

4. On the AI node/automation worker machine, create an ai folder in C:\ so the system has an C:\ai path. Create a new file named ai.bat.

5. Edit the ai.bat file and add the following line:

   echo Data for Output File LIMS ID %1 > %1.txt

6. Run the process/step created on an existing attached result file.

   • The step passes the LIMS ID of its output file placeholder to the script.

   • The script creates a file in the working directory.

   • When the script exits, this file transfers to the LIMS and associated with the step. The file contains a single line of text that includes the LIMS ID of the output file for easy verification.

Test and Verify Setup on a Linux AI Node/Automation Worker

1. Create a process/step that generates a result file

2. Configure an automation on the process/step. Associate it with the channel on which the AI node/automation worker is configured to communicate.

3. Add the following command line string:

   bash -c "echo Automation Test > {outputFile0}.txt"

4. Run the process/step created on an existing attached result file.

   • The step passes the LIMS ID of its output file placeholder to the script.

   • The script creates a file in the working directory with a file name that contains this LIMS ID.

   • When the script exits, this file transfers to the LIMS and associated with the step. The file contains the text "Automation Test". When open, the file opens in the default program associated with *.txt files.

Review Automated Informatics Log File

AI nodes/automation workers are installed using the Automated Informatics (Automation Worker for LIMS v5 and later) software package.

In the installation directory:

1. Find the /log directory, which contains an automatedinformatics.log file.

2. Use this file to locate log lines near the time of the process/step completion event.

   • Locate the log line containing the parameter/token command string, and manually run and test the script.

   • Locate the working directory to review temporary files created.

Step 1: Locate log lines near time of the process completion event

This step can indicate if the cause of the error lies with a network or other issue external to the computer running the AI node / automation worker.

Step 2: Locate log line containing parameter/token command string to manually run/test script

Running the script manually forms a unit test. The script is run on the command line, without being invoked by automation.

To locate the line containing the command-line string, search for the Command string, or externalprogram.runExternalProgram.

An example is shown in the following abridged log file section. Copy the line to a text file and modify it for script testing.

2014-02-28 21:24:48,688 INFO ... definitions.behaviour.automatedinformatics.plugins.externalprogram.runExternalProham as ...

2014-02-28 21:24:49,323 INFO ... (ExternalProgramBehaviour.java:133)... Command string: bash -c "~/scripts/HelloWorld.sh http://###.###.###.###/api/v2/processes/A30-MXX-110228-24-2320 > 92-2869.txt"

Step 3: Locate working directory to review temporary files created

Used when temporary files are left by the script, this automation script removes the temporary working directory, unless there was an error. In case of an error, the directory provides clues to the root cause of the error.

To locate the working directory, search for "Working directory:" in the log file.

An example is shown in the following abridged log file section. Use the recorded directory to list and review temporary files.

2014-02-28 21:24:49,324 INFO ... Working directory: /home/gls/GenoLogicsAutomatedInformatics/temp/runExternalProgram-28022014-432350866632555775.A30-MXX-110228-24-2320

2014-02-28 21:24:49,324 INFO ... Retrieved files. Executing command.

Assumptions

The incoming message contains the following:

• Project ID or Name

• Sample ID or Name

• Container ID or Name

• Container type (plate / tube type)

• Container well position (if sample is on a plate) eg G:2

• Sample user-defined fields (UDFs) / custom fields

Main Logic

Create a Sample:

POST to https://your_server/api/v2/samples:

We receive something like the following:

Create a Project:

POST to https://your_server/api/v2/projects

We receive something like the following:

Create a Container (tube):

POST to https://your_server/api/v2/containers:

We receive something like the following:

Create a Container (96 well plate):

POST to https://your_server/api/v2/containers:

We receive something like the following:

Create a Sample in the 96 Well Plate Rreated, and the Project Created:

POST to https://your_server/api/v2/samples:

We receive something like the following:

Confirm the Project Exists

GET: https://your_server/api/v2/projects?name=Week%2039

If the project exists, we receive something like the following:

If the project does not exist, we receive something like the following:

Confirm the Container Exists

GET: https://your_server/api/v2/containers?name=Example%20Container%2020140910

If the container exists, we receive something like the following:

If the container does not exist, we receive something like the following:

In a highly automated workflow, a lab gains little value from manually selecting samples into the ice bucket and then transitioning them through a step. Ideally, upon completion of one step, a following step could be automated such that the output analytes were transitioned through to the Record Details screen.

The Clarity LIMS External Program Plugin (EPP)/automation system cannot aid in this transition. The last point at which an automation can be triggered is before the step completion.

This scenario requires a stand-alone API application, which can be run by an automation at the end of a step.

Using this approach, a standalone app would poll the API until each of the output analytes from the previous step were queued for the next step. After they are queued, they can be walked through to the Record Details stage.

The steps are as follows:

1. EPP / automation triggers at step completion and launches an API app as a new Linux process and then finishes. The parameter for the API app is the URL for the current process.

2. API app polls to see if each output analyte is queued.

   • Use the artifacts batch endpoint (api/v2/artifacts/batch/retrieve) to poll.

   • Check the last workflow-stage node within workflow-stages and look for status="QUEUED".

3. API app moves the output analytes through the step to Record Details.

   • Use the /api/v2/steps endpoints to start the step and then move the analytes forward.

This article explains how to make files that were produced by / attached to the LIMS in an earlier step, visible in a subsequent step.

Solution

Consider a simplified workflow / protocol containing just two steps: Produce Files and Display Files.

• The first step, Produce Files, will take in analytes (derived samples), and generate individual result files (one per input).

• The subsequent Display Files step will allow us to view the files associated with the analytes from the previous step.

After the files have been generated by and attached to the Produce Files step, the Record Details screen of the step displays the files.

The key to displaying these files in any subsequent step involves producing a hyperlink to the file and displaying it as a user-defined field (UDF)/custom field in subsequent steps.

You may be familiar with creating and using text, numeric, and checkbox UDFs/custom fields. However, you may be less familiar with the hyperlink option. Fields of this type are used less frequently, but they are perfect for this solution.

NOTE: As of Clarity LIMS v5.0, the term user-defined field (UDF) has been replaced with custom field in the user interface. However, the API resource is still called UDF.

This solution involves a script that runs on the Record Details screen on the subsequent Display Files step and populates the fields. See the following figure.

As you can see, the structure of the hyperlink is straightforward and includes:

• The IP address / hostname of the server.

• The port.

• A link to the LIMS ID of the file to be linked to.

The Script

To populate these fields, there are numerous methods available within an API-based script. The method discussed here works for the two-step protocol described earlier (namely that we want the files displayed in the next step of the protocol). It also works when the steps in which the files are uploaded and displayed are separated by several intermediate steps.

Assuming that the script will run just as the Record Details screen of the Display Files step is being displayed, use pseudocode to produce the hyperlinks.

For each output:

1. Determine the LIMS Unique ID (LUID) of the output artifact.

2. Determine the LUID of the submitted sample associated with the output artifact.

3. Determine the LUID of the resultfile artifact produced by the earlier process, derived from the common submitted sample.

4. Determine the LUID of the file associated with the resultfile artifact.

5. Update the hyperlink UDF / custom field on the output artifact (from step 1) with the specific hyperlink value.

To illustrate these pseudocode steps, XML from a demo system is provided.

1. Gather the LIMS Unique ID (LUID) of the Output Artifact

<prc:process uri="http://192.168.8.10:8080/api/v2/processes/24-24452" limsid="24-24452">
    <type uri="http://192.168.8.10:8080/api/v2/processtypes/1555">display files</type>
    <technician uri="http://192.168.8.10:8080/api/v2/researchers/1">
    <first-name>System</first-name>
    <last-name>Administrator</last-name>
    </technician>
    <input-output-map>
        <input post-process-uri="http://192.168.8.10:8080/api/v2/artifacts/ADM1301A2PA1?state=48636" 
            uri="http://192.168.8.10:8080/api/v2/artifacts/ADM1301A2PA1?state=48636" 
            limsid="ADM1301A2PA1"/>
        <output uri="http://192.168.8.10:8080/api/v2/artifacts/2-81806?state=49106" 
            output-generation-type="PerInput" output-type="Analyte" 
            limsid="2-81806"/>
    </input-output-map>
    <input-output-map>
        <input post-process-uri="http://192.168.8.10:8080/api/v2/artifacts/ADM1301A3PA1?state=48632" 
            uri="http://192.168.8.10:8080/api/v2/artifacts/ADM1301A3PA1?state=48632" 
            limsid="ADM1301A3PA1"/>
        <output uri="http://192.168.8.10:8080/api/v2/artifacts/2-81805?state=49105" 
            output-generation-type="PerInput" output-type="Analyte" 
            limsid="2-81805"/>
    </input-output-map>
    <input-output-map>
        <input post-process-uri="http://192.168.8.10:8080/api/v2/artifacts/ADM1301A1PA1?state=48650" 
            uri="http://192.168.8.10:8080/api/v2/artifacts/ADM1301A1PA1?state=48650" 
            limsid="ADM1301A1PA1"/>
        <output uri="http://192.168.8.10:8080/api/v2/artifacts/2-81804?state=49104" 
            output-generation-type="PerInput" output-type="Analyte" 
            limsid="2-81804"/>
    </input-output-map>
</prc:process>

From the XML representation of the Display Files process/step. we see that we have three output artifact LUIDS: 2-81806, 2-81805 and 2-81804.

2. Determine the LUID of the Submitted Sample Associated with the Output Artifact

By examining the XML representation of the first output artifact (2-81806), we see the LUID of the associated submitted sample is ADM1301A2:

<art:artifact uri="http://192.168.8.10:8080/api/v2/artifacts/2-81806?state=49106" limsid="2-81806">
    <name>WG-23476-2</name>
    <type>Analyte</type>
    <output-type>Analyte</output-type>
    <parent-process uri="http://192.168.8.10:8080/api/v2/processes/24-24452" limsid="24-24452"/>
    <qc-flag>UNKNOWN</qc-flag>
    <location>
        <container uri="http://192.168.8.10:8080/api/v2/containers/27-11353" limsid="27-11353"/>
        <value>1:1</value>
    </location>
    <working-flag>true</working-flag>
    <sample uri="http://192.168.8.10:8080/api/v2/samples/ADM1301A2" limsid="ADM1301A2"/>
</art:artifact> 

3. Determine the LUID of the resultfile Artifact Produced by the Earlier Process/Step, Derived from the Common Submitted Sample

After the common ancestor is found, ask Clarity LIMS for the output artifacts produced by our step of interest (Produce Files) directly.

For example:

http://192.168.8.10:8080/api/v2/artifacts?samplelimsid=ADM1301A2&process-type=Produce%20Files&type=ResultFile

Yields the following XML:

<art:artifacts>
    <artifact limsid="92-81803" uri="http://192.168.8.10:8080/api/v2/artifacts/92-81803"/>
</art:artifacts> 

The resultfile with LUID 92-81803 is associated with the current output artifact (2-81806), even though these entities may be separated by several steps.

If the process/step produces multiple resultfiles, you may need to further constrain the search using the name= parameter. For example:

http://192.168.8.10:8080/api/v2/artifacts?samplelimsid=ADM1301A2&process-type=Produce%20Files&type=ResultFile&name=<name of ResultFile>

4. Determine the LUID of the File Associated with the resultfile Artifact

By gathering the XML representation of artifact 92-81803, the associated file has LUID 40-3652:

<art:artifact uri="http://192.168.8.10:8080/api/v2/artifacts/92-81803?state=49103" limsid="92-81803">
    <name>WG-23476-2</name>
    <type>ResultFile</type>
    <output-type>ResultFile</output-type>
    <parent-process uri="http://192.168.8.10:8080/api/v2/processes/24-24451" limsid="24-24451"/>
    <qc-flag>UNKNOWN</qc-flag>
    <sample uri="http://192.168.8.10:8080/api/v2/samples/ADM1301A2" limsid="ADM1301A2"/>
    <file:file limsid="40-3652" uri="http://192.168.8.10:8080/api/v2/files/40-3652"/>
</art:artifact> 

5. Update the Hyperlink UDF/Custom Field on the Output Artifact (from Step 1) with the Specific Hyperlink Value

Now that you know the LUID of the file associated with output artifact 2-81803, set the value of its hyperlink field in the following form:

http://192.168.8.10:8080/clarity/api/files/3652

When constructing the value for the hyperlink, the 40- prefix should be removed from the LUID of the file.

A lab may receive samples submitted from various sources. This can pose a problem with regards to sample names.There may be duplicate sample names, and/or various name formats, all of which make it hard for lab scientists to recognize a sample.

Clarity LIMS programmers often rename all incoming samples to a certain naming convention.

This section provides an example to address this problem.

Recommendations

• When accepting a project and its samples, the receiving lab scientist runs a Clarity LIMS step named Receive Samples.

• The underlying Receive Samples process type / master step is configured with analyte (sample) inputs, and no analyte outputs.

• A shared result file output is configured to capture logging from the script.

• The sample name could be a derivative of the Sample LIMSID, with a prefix:

  • Because the LIMSID is guaranteed to be unique, this approach mitigates any need to maintain an external sequence of numbers.

  • The Sample LIMSID is derived from the Project LIMSID, which is configurable.

Proposed solution

• The Receive Samples process is configured to trigger a script that renames the samples that are input to the process.

• This trigger also passes the OriginatingProcessURI to the script. This example assumes that the original submitted sample name must be preserved, and so it is saved in a sample UDF.

The following pseudo code shows how one might implement the sample-renaming script:

1. Connect to the API, using the OriginatingProcessURI.

2. Retrieve the OriginatingProcessXML and store it in a variable.

3. Iterate through the inputoutput map of the OriginatingProcessXML, and for each InputArtifact:

   • GET the InputArtifactURI and store the input ArtifactXML in a variable.

   • From this ArtifactXML, GET the SourceSampleXML and store it in another variable.

   • Modify the SourceSampleXML. To do this:

   • Rename the SampleName to a desired name (see Recommendations section, above).

   • Finally, PUT the Sample XML back.

Use the API to update the preset value of a user-defined field (UDF)/custom field configured on a step.

From your test server:

1. GET a chosen UDF/custom field.

2. Do a PUT and include a new line.

For example, to add 'My new preset', insert the preset (My new preset), after your last value in your XML:

........

<preset>Your last preset</preset>

<preset>My new preset</preset>

<is-required>Your original value</is-required>

<attach-to-category>Your original value</attach-to-category>

</cnf:field>

This tool is powerful when integrating with external systems and combined with the Begin Work trigger. For example, it can be used to reach out to an external source with a script, initiated with the Begin Work trigger. The script makes sure that the presets for the Step Details UDFs/custom fields are always up to date and in sync with the server—before entering the Record Details screen.

Within a script, you may sometimes need to know to which workflow the current sample is assigned.

However, in Clarity LIMS, the XML payload that relates to the sample does not provide information about the workflow associations of the sample.

For example, consider a sample (artifact), picked at random, from a demo system:

<art:artifact uri="http://192.168.8.10:8080/api/v2/artifacts/2-42201?state=25871" limsid="2-42201">
    <name>Sanger Sample 49</name>
    <type>Analyte</type>
    <output-type>Analyte</output-type>
    <parent-process uri="http://192.168.8.10:8080/api/v2/processes/24-13704" limsid="24-13704"/>
    <qc-flag>UNKNOWN</qc-flag>
    <location>
        <container uri="http://192.168.8.10:8080/api/v2/containers/27-6899" limsid="27-6899"/>
        <value>E:1</value>
    </location>
    <working-flag>true</working-flag>
    <sample uri="http://192.168.8.10:8080/api/v2/samples/KUZ407A145" limsid="KUZ407A145"/>
</art:artifact>

It is evident that this XML payload does not provide the workflow information.

This following solution shows how to use the Clarity LIMS API to determine the association of a sample to one or more workflows.

Solution

The XML payload that corresponds to each sample artifact contains a link to the related submitted sample (or samples, if it is a pooled artifact).

Follow that link to see what it yields:

<smp:sample uri="http://192.168.8.10:8080/api/v2/samples/KUZ407A145" limsid="KUZ407A145">
    <name>Sanger Sample 49</name>
    <date-received>2013-07-15</date-received>
    <project limsid="KUZ407" uri="http://192.168.8.10:8080/api/v2/projects/KUZ407"/>
    <submitter uri="http://192.168.8.10:8080/api/v2/researchers/4">
        <first-name>Jill</first-name>
        <last-name>Hesse</last-name>
    </submitter>
    <artifact limsid="KUZ407A145PA1" uri="http://192.168.8.10:8080/api/v2/artifacts/KUZ407A145PA1?state=25080"/>
    <udf:field type="String" name="Progress">Initial sample QC complete</udf:field>
    <udf:field type="String" name="Sanger Primer">ITS1</udf:field>
    <udf:field type="String" name="Template (uL)">6.9</udf:field>
</smp:sample>

The XML corresponding to the submitted sample has a link to an artifact. This artifact is special for several reasons:

• It is known as the 'root artifact'.

• It has an unusual LIMS ID for an artifact. LIMS IDs start with '2-' for Analytes, and '92-' for ResultFiles. This one appears to be derived from the LIMS ID of the sample: KUZ407A145PA1

• A root artifact is created 'behind the scenes' whenever a submitted sample is created in the system.

• The sample history in Clarity LIMS makes it appear as if the first step in the workflow is run on the submitted sample. However, it is actually the root artifact that is the input to the first process.

• When a submitted sample is assigned to the workflow, it is the root artifact that is assigned to that workflow.

Therefore, if gathering the XML payload corresponding to the root artifact, you should see the workflow assignment:

<art:artifact uri="http://192.168.8.10:8080/api/v2/artifacts/KUZ407A145PA1?state=25080" limsid="KUZ407A145PA1">
    <name>Sanger Sample 49</name>
    <type>Analyte</type>
    <output-type>Analyte</output-type>
    <qc-flag>UNKNOWN</qc-flag>
    <location>
        <container uri="http://192.168.8.10:8080/api/v2/containers/27-6754" limsid="27-6754"/>
        <value>1:1</value>
    </location>
    <working-flag>true</working-flag>
    <sample uri="http://192.168.8.10:8080/api/v2/samples/KUZ407A145" limsid="KUZ407A145"/>
    <udf:field type="String" name="Conc. Units">ng/uL</udf:field>
    <udf:field type="Numeric" name="Concentration">100</udf:field>
    <artifact-group name="Sanger Sequencing" uri="http://192.168.8.10:8080/api/v2/artifactgroups/851"/>
</art:artifact>

The key element is as follows.

<artifact-group name="Sanger Sequencing" uri="http://192.168.8.10:8080/api/v2/artifactgroups/851"/>

The name of the artifact-group (Sanger Sequencing) should match the name of the workflow in which the root artifact (and by inference, artifacts derived from the root artifact) is assigned.

Troubleshooting

If you find that the artifact-group node is missing from some of the root artifacts, there are several potential reasons:

• The workflow has been completed, causing the root artifact to be unassigned from the workflow.

• The derived samples / artifacts have been removed from the workflow intentionally, because of a sample processing issue.

• An API script has intentionally removed the derived samples / artifacts from the workflow.

• The assigned workflow has been marked as 'Archived'.

How to copy the value of a UDF/custom field from source to destination (typically from the inputs of a process/step to the outputs) is a frequently asked question.

For example, suppose a process/step takes in libraries and tracks their normalization. In such a case, the input samples have a UDF/custom field that is used to track the library ID. Since the library ID changes, it is desirable for the output samples to also have this ID.

Solution

Use the API to gather the XML for the inputs, then copy the XML node relating to the UDF/custom field to the outputs.

Alternatively, use the out-of-the-box copyUDFs script, which Illumina provides as part of the NextGen Sequencing configuration.

Script

The copyUDFs script is available in the ngs-extensions.jar archive*, and can be called from the EPP / automation parameter string.

The archive file may be named differently, depending upon the version you are running.

Usage:

Defining the UDF / custom field values

The UDF / custom field values to be copied are defined in the -f portion of the syntax. These values must be present on both the inputs and outputs of a process.

For example, suppose you wanted to use this script to copy the value of a UDF called Library ID:

• The Library ID field must be defined on both inputs and outputs.

• The -f flag is defined as follows:

To copy multiple UDF values from source to destination, list them in comma-separated form as part of the -f flag.

• To copy Library ID and Organism from source to destination, use the following example:

When running the Aggregate QC step in Clarity LIMS, the QC pass and fail flags for the samples display in the Record Details screen.

This section explains how to use the API instead to find the samples that passed or failed QC aggregation.

Query the API and filter the results list based on the qc-flag parameter value. For more on filtering, see section.

• To filter the list by QC flag with a value of PASSED, use the following example:

• To find an individual QC flag result for an individual sample, use the LIMS ID of the sample:\

• Then search for the value of the element of the endpoint payload for the artifact.

The <qc-flag> element of the input analyte (sample) artifact is sent into the Aggregate QC step.

To demonstrate this detail, review the following steps:

1. In the API, find a single analyte artifact (derived sample) that has passed QC. The XML QC flag value is PASSED.

2. In Clarity LIMS, find the same sample and change the value of the element from PASSED to FAILED. Save the change.

3. In the API, find the sample again. See that the XML QC flag value is set to FAILED.

The QC flag parameter qc-flag can be set on an input or output analyte (derived sample) or on an individual result file (measurement) with a few lines of Groovy code.

In the following example, the qc-flag value of the analyte artifact is set based on the value of the bp_size variable when compared to the threshold1 and threshold2 variables.

The following code determines whether a qc-flag value is previously set, such that a flag is only set if one does not exist.

When a sequencing run is complete, it is often desirable to pass data to CASAVA for BCL conversion automatically rather than manually. This section proposes a method to configure this automation.

NOTE: This solution is not tested end-to-end on an instrument.

The proposed approach involves adding an automation trigger to the Sequencing step, such that it invokes a script that launches the BCL Conversion step.

However, because the BCL Conversion step does not run immediately, it is launched in a dormant state until the Sequencing step is complete.

The key event here is the Run Report that is created and attached to the Sequencing step. As the last event to occur in the step, the creation of this report is used to prompt the BCL Conversion step to 'wake up' from its dormant state and begin processing.

The following pseudocode describes the work that must occur within the script:

Solution details

Step 1. Create Script to Launch BCL Conversion Process / Step

A required script that launches the BCL Conversion step via the API might be absent. The creation of such a script is covered in . This example only covers the functionality of the script rather than code.

In addition to the expected processURI, username, and password parameters/tokens, the script should accept another parameter (the LIMSID of the Run Report from the Sequencing step).

For example, the script can be invoked as follows:

Use this syntax when configuring the command line on the Sequencing process/step.

Step 2. Configure the Clarity LIMS Web Interface

Configure the automation so the script is automatically triggered when exiting the Record Details screen.

Example configuration details

1. The BCL Conversion process is configured:

   • To take in a ResultFile input and generate a non-shared ResultFile output

   • With a process parameter of 'Standard,' which initiates the actual BCL conversion.

2. The script is passed the value '92-3771' as the -r parameter.

3. This is then converted to a full URI and became the input element of the following XML, which is POSTed to the /processes API resource:

Notes

• Update all URIs in the XML to point to the hostname and API version for the system.

• Provide a valid URI for the lab scientist. There might be a user in the system with LIMS ID of '1'.

• If the POST is successful, the API returns the valid XML for the created process.

Note: This scenario is one of the few occasions where the POST succeeds, yet returns XML that differs from the input XML. The results can be confusing, because a standard approach for validating whether POSTs are successful is to compare the output XML with the input. If they differ, assume that the POST failed. However, in this scenario it did not fail.

This topic explains how to:

1. Detect when files have have been uploaded.

2. Extract the key information that might comprise a notification.

The Files API Resource

The key resource to investigate is the files resource, which provides a listing of files within the system.

Step 1. Access the Files Resource

On a test system accessing the files resource as follows:

produces the following output:

Step 2. Filter the Files URI

Although not particularly useful in itself, the files URI becomes more interesting when we filter it to only include files uploaded after a specified date-time, and also only those files that have a published status of 'true'.

For example, the following URI:

produces this output on a test system:

This outcome is much more manageable. Because they are uploaded via the Collaborations Interface, they inherently have a published status of 'true'. We use this status to exclude regular files uploaded to the LIMS via other methods and interfaces.

Step 3. Retrieve the XML Representations of the Files

By following the URIs to retrieve the full XML representations of these files, the output is similar to the following:

and:

Step 4. Retrieve Project/Sample Information

Retrieve the associated project/sample, and extract the names and/or IDs to embed into the notification, by following the URI in the 'attached-to' elements.

In this case, the following result is produced:

and:

Proposed Solution

A script must be run periodically (hourly/daily) that queries the files resource for files that have a published status of true, and are last modified in the period of interest.

After this list of files is retrieved, the following pseudocode can be applied:

An example derived from the above XML could lead to the following notifications:

${baseName}.dirServer-side configuration allows for configuration of multiple filestores to be associated to entities (samples, projects, processes/steps, and so on) in BaseSpace Clarity LIMS.

This feature allows for linking to large data files on a different server, eliminating the need to move large files on the Clarity LIMS filestore. Large files can include results, images, and searches, and so on.

For example, sequencing instruments typically produce large result files. Attaching these files to the Sequencing step in Clarity LIMS results in the following drawbacks.

• Involves transferring the files to the Clarity LIMS filestore. The larger the file, the slower the transfer speed.

• Requires a large amount of space as runs build up.

An alternative solution is to set up a remote filestore to be used as the results directory from which Clarity LIMS accesses the files directly.

To do this setup, three steps are required:

1. Set up HTTP, HTTPS, FTP, or SFTP access to the files and folders you wish to share.

2. Configure the Clarity LIMS server to recognize the URI of a file on the remote filestore.

3. POST information to Clarity LIMS, via the REST API, to reference the file from a Clarity LIMS entity (project, sample, process/step, result file, and so on).

Setting Up Access to the Files

BaseSpace Clarity LIMS can operate with many different forms of file servers – HTTP, HTTPS, FTP, and SFTP access are all supported.

It is your responsibility to set up this access. For HTTP, you may be interested in or for HTTP file serving.

Configuring the LIMS Server

Properties Required for Tracking a New Remote Filestore

To track a new remote filestore, Clarity LIMS requires four database properties: directory, host, port, and scheme.

The four properties share a base name, but have different suffixes attached (dir, host, port, scheme). These suffixes are summarized in the following table.

• The base name can be anything. Clarity LIMS finds any base names that end in .scheme and uses that base name to find the other information.

• If necessary, add the last three properties listed in the table (with the .domain, .user, and .password suffixes) to specify a domain, username, and password to be used when accessing files.

• Clarity LIMS v5 and later—For the property changes to take effect, Tomcat must be restarted.

Working with the Database Properties

Use the omxprops-ConfigTool.jar to create, update, and retrieve values of the database properties. This tool is found at the following location: /opt/gls/clarity/tools/propertytool

To create a property, use the following examples:

NOTE: These properties may not be global properties. Do not use the -g property here.

To get the value of an existing property:

To update the value of an existing property:

To encrypt a password:

NOTE: To set a property to the encrypted result, set the value as ENC().

Example: Mapping a Remote HTTP URI

In this case, the base name for the properties is http-lims-files.

Steps

1. As the glsjboss user, access the omxprops property tool in /opt/gls/clarity/tools/propertytool.

2. Add the following dir, host, port, and scheme properties to the server from the command line:

In the example above, the <http-lims-files.dir> parameter value is /limsdata. Any file in http://<YourHTTPHost/limsdata/ is available to be referenced by BaseSpace Clarity LIMS.

For all files on the web server to be available, use the / parameter value, for example:

Posting Information to the LIMS Referencing Files on the Remote Filestore

After the filestore properties are added to Clarity LIMS (and JBoss/Tomcat has been restarted, as applicable), you can attach the files to Clarity LIMS.

To attach the files to Clarity LIMS:

• POST to http://hostname/api/v2/files, with the content-location tag pointing to the remote filestore.

An example XML POST is provided, using the filestore created in the previous example:

Results

• The file is now downloadable directly from Clarity LIMS.

• Any entity that can have a file attached to it may be referenced in the parameter.