The Clarity LIMS Rapid Scripting™ API provides scientific programmers with self-descriptive, yet flexible, data access. It uses a RESTful model for data access because this model is well suited to these requirements. This article provides a high-level introduction to REST concepts and technologies.

Introduction to REST Web Service Technology

Representational State Transfer (REST) is a style of software architecture for distributed information retrieval systems, most commonly observed by using the web.

• REST governs proper behavior. It is not a methodology or a design principal, but rather a set of rules to which a system should conform.

• REST allows a uniform interface between clients and servers that is simple and decoupled, enabling each system to evolve independently.

• REST is referred to as stateless because each new API request contains all the information required to complete it, without relying on previous requests. Conforming to these REST principals is referred to as being RESTful.

• REST was developed in parallel with HTTP and makes use of this protocol. It is an elegant way to programmatically access resources over HTTP. It is very flexible because you can use it with any language or tool that supports HTTP.

Other uses of REST

The web is probably the largest known RESTful system. Its behavior is very simple:

1. When you click a link in a web browser, your system requests information by sending a GET request to the specified URL. This URL is a resource.

2. The server that hosts the URL responds, typically with one of two things:

   • If the page exists, the server sends the browser an HTTP 200 response code and the contents of the page.

   • If the page does not exist, the server sends an HTTP 404 response code and an error message indicating that the page cannot be found.

Many software development groups use RESTful APIs. Google, Yahoo, and many public web sites use the RESTful model for information access.

Communicating with the REST API

The REST API allows you to retrieve and update information using HTTP operations. This ability provides some flexibility in how to communicate with the system.

While REST requests and responses can be in a variety of formats, we chose XML. Each resource and XML element is detailed in the API Portal.

Authenticating with the API

To use the REST API, sign in using HTTP BASIC authentication. The method used to authenticate will depend on how you use the API:

• When using a browser to retrieve information from the API, sign in to the browser with a user name and password. When signing in using a browser, the session remains open until the browser is closed.

• When using an HTTP request tool to retrieve, add, update, or remove data using the API, the tool asks for a user name and password each time you submit a request to the system.

• When using a script to communicate with the API, the script must first authenticate with the API. The session remains open for as long as the script is being actively read by the system.

The account you use to sign in to the API must have System Administrator or Facility Administrator privileges.

Understanding URIs used by the API

The API allows self-discovery of an object. When you request information about an object, the system typically returns URIs to its children and, sometimes, its parent. Use one URI to find the next URI in a hierarchy.

When viewing XML in a browser, tools can automatically create links from the URIs returned by the system. Examples of such tools are the Firefox Text Link or Linkificator add-ons. This way, you can select URIs to browse through the API.

Requests are made to the API by sending XML in HTTP calls:

• GET is used to read an item.

• POST to create an item.

• DELETE to delete an item.

• PUT is used to update an item.

In its simplest form, use a browser to enter and read the content of a URI, which allows browsing through the system. When using this method, a GET request is issued to the API for a specified object (referred to as a resource). The request returns XML containing the metadata about that resource. See the following section for details.

If you want to add, update, and delete small amounts of data using the API, use an HTTP request tool, such as the Firefox RESTClient add-on.

Resources and Namespaces

When working with REST, there are references to resources and namespaces.

For references:

A RESTful API groups related information into resources, each of which is referenced with a global identifier (URI).

In the API, for example, every sample in the LIMS has a resource for its information. When scripting, the resource is created or updated with POST and PUT HTTP calls. There are two types of resources: single and list types.

• A list resource is used to access a collection of single resources (such as a listing of all samples).

• The single resource type is used to access details on just one resource (a sample, for example).

It's important to understand how the information in the LIMS has been grouped and structured into resources. To learn more, see Structure of REST Resources.

For namespaces:

An API that uses XML relies on namespaces. In XML, namespaces define the vocabulary of elements and attributes in an XML document. Each REST resource references the XML structure defined by a particular namespace.

When scripting, we use namespaces to look up specific details related to the XML data elements, attributes, and formats that represent a resource. Namespaces also order the subelements of the XML document.

In the current revisions of the API, the PUT and POST methods read the subelements of the XML independent of order, but the namespace still defines the order of the XML provided in GET calls.

Additional Information

For sophisticated write operations and automation of work, you must use a script to communicate with the API. The Cookbook contains examples that demonstrate how to use scripts to perform your work.

Overview

The internal Clarity LIMS API (eg, https://example.claritylims.com/clarity/api) is the API used to deliver the Clarity LIMS web interface. This interface is not typically meant for public consumption. However, some customers use it for troubleshooting and to mitigate system issues.

Preventing CSRF Attacks

As of Clarity LIMS v5.1, access to the internal Clarity LIMS API changed to enhance security and prevent Cross Site Request Forgery (CSRF) attacks. Two new HTTP headers must now be present when issuing PUT, POST, DELETE, and PATCH requests:

• Origin—This header must be set to the scheme and authority of the server being accessed (eg, https:// example.claritylims.com).

• X-Requested-With—This header must be set to XMLHttpRequest.

The attached cURL, Python, and Java examples demonstrate how to authenticate and issue internal API requests. These examples assume a Clarity LIMS server at https://example.claritylims.com.

csrf headers.sh:

csrf headers.py:

csrf headers.java:

When submitting a GET request to certain REST API resources (also known as list resources), the system returns a list of records. For example, submitting a GET request to the samples resource returns a list of all submitted samples stored in the system. Depending on the resource being used, use various query parameters to filter the records based on certain criteria. For more information about the parameters that are available, refer to the reference documentation for the desired resource.

To filter a list, the resource and parameter must be separated with a question mark (?). The parameter and the value you want to base the query on must be separated with an equal sign (=).

When filtering a list of artifacts, combine parameters within the same query statement. You can also repeat certain parameters, specifying a new value with each occurrence of the parameter.

The first parameter must be preceded with a question mark (?). Add additional parameters by separating each parameter with an ampersand (&).

Repeating a parameter with new values:

Combining parameters:

When combining or repeating parameters, each record returned matches one of the parameter values, or all the parameter values, depending on the usage:

• If the query statement contains multiple values for the same parameter, the ampersands are treated as an OR.

• If the query statement contains values for multiple parameters, the ampersands are treated as an AND.

For example, if a project LIMS ID and a process type are provided as parameters, the system returns only the files that match both the project LIMS ID and the process type. To see the files that match the project LIMS ID or the process type, issue two separate GET requests and combine the results.

• /api/v2/processes—This URI returns all processes run in the system.

• /api/v2/processes?type=MALDI—This URI returns all MALDI processes run in the system.

• /api/v2/processes?type=Sample Prep&type=MALDI—This URI returns all MALDI or Sample Prep processes run in the system.

• /api/v2/containers—This URI returns all containers in the system.

• /api/v2/containers?type=Tube—This URI returns all tubes in the system.

• /api/v2/containers?type=Tube&name=27-111&name=27-112—This URI returns the tubes in the system that are named 27-111 OR 27-112.

Using the Last-Modified Parameter

Certain resources include the last-modified query parameter.

When used, the system displays only the results that have been modified because the last-modified date. The last-modified date is represented in ISO 8601 Complete date including hours, minutes, and seconds format: YYYY-MM-DDThh:mm:ssTZD.

Viewing Paginated Results

Lists of records are often large, spanning multiple pages. Many list resources include parameters that are used to work with paginated results.

Using UDF and UDT parameters

Certain resources include parameters that can be used to filter the results displayed based on UDF information that is associated with the results:

• udf.UDFNAME[.OPERATOR]=UDFVALUE—This parameter filters the results based on a specified value for a specified UDF. Any item that contains the value for the UDF is returned, unless parameters include an optional operator filter ( [.OPERATOR] in the expression provided). The lowercase filter operators of min or max are described later.

• udt.name=UDTNAME—This parameter filters the results based on a specified UDT. Any item that has the UDT selected is returned.

• udt.UDTNAME.UDFNAME[.OPERATOR]=UDFVALUE—This parameter filters the results based on a specified value for a specified UDF that resides within a specified UDT. Any item that contains the value for the UDF is returned, unless parameters include an optional operator filter ( [.OPERATOR] in the expression provided). The lowercase filter operators of min or max are described later.

To To filter results using UDF information, use the following query structure:

To filter results using the name of a UDT, use the following query structure:

To filter results using UDF information that is part of a specific UDT, use the following query structure:

When filtering lists and using date or numeric UDF or UDT values, use operators to restrict a query. The following operators are supported:

• .min—This operator displays results that are greater than or equal to the specified value.

• .max—This operator displays results that are less than or equal to the specified value.

Examples:

• /api/v2/processes?type=Sample Prepandudt.name=Plasma—This URI returns all Sample Prep processes run in the system that have the Plasma UDT selected.

• /api/v2/processes?type=Sample Prepandudt.Plasma.Platelet Count.min=50—This URI returns all Sample Prep processes that have a UDF named Platelet Count with a value of 50 or greater, within a UDT named Plasma.

• /api/v2/processes?type=Sample Prepandudf.Sample=Serumandudf.Sample=Tissue—This URI returns all Sample Prep processes that have a UDF named Sample with a value of Serum OR a UDF named Sample with a value of Tissue.More examples of filtering exist in the Cookbook.

When filtering with UDT or UDF parameters, all special characters in the parameter string must be URL encoded. The Pipe ( | ) or the URL-encoded pipe ( ) cannot be used.

When filtering on a UDF that is configured as a Multiline Text UDF, if a value contains a hard return, the value must include the URL-encoded line feed () at the appropriate location. Depending on how API requests are issued (via a browser or a script), spaces in names or values may require URL encoding, and trailing spaces in a name or value always require encoding. For example, for results to be returned, ‘name ‘ requires ‘name’.

The REST API methods attempt to return appropriate HTTP status codes for every request. To use the REST API effectively, a good understanding of HTTP and status codes is required. A complete list of HTTP status codes and definitions can be found at the following website:

The primary status codes used by the REST API are as follows:

• 200 OK: Success.

• 201 Created: A resource was successfully created.

• 400 Bad Request: Invalid data was supplied for the relevant resource type.

• 401 Unauthorized: The requested resource cannot be loaded until valid logon credentials have been entered. If this error is received after logon credentials have been entered, this indicates that the credentials are not valid.

• 403 Forbidden: Access to the requested resource has been denied. (Make sure that the authorized user has administrative privileges.)

• 404 Not Found: The URI requested is invalid or the resource requested does not exist.

• 413 Request Entity Too Large: The request is larger than the server is willing or able to process.

• 500 Internal Server Error: A generic error message, given when there is no suitable specific message.

Error message format

Error messages are returned as exception elements with a message element containing a user-facing error message.

The exception may also include a suggested-actions element with more detail on how to resolve the error.

User-facing XML error messages are not returned for 401 and 403 errors. In these cases, the HTTP error must be resolved.

REST and automation are the key interfaces for scripting. A language-agnostic application programming interface (API) is important to scientists as it allows for broad and diverse integration. Together, REST and automation provide powerful and easy-to-use scripting. However, you first need to understand the conceptual structure and design of these interfaces.

Within the Clarity LIMS Rapid Scripting API, REST technology is used to provide data specifically structured for life science research.

NOTE: If you are new to the REST Web Service, we recommend that you read and .

REST: a web service for data access

The REST Web Service is the fundamental data access interface using XML over HTTP. It is agnostic to programming languages as most languages support HTTP and XML with libraries or built-in methods.

In life science research labs, tracking samples and the data associated with biology, research, and lab work is complex. The REST resources return information that is human-readable and interpretable. Use a web browser to explore the XML returned.

REST represents real laboratory items and activities in self-contained groups of data called resources. It provides access to recorded lab steps and to sample test results, and it provides this access using resources. For example:

• The process and steps resources track the steps in the lab in terms of who did what and when.

• The sample and artifact resources contain information on the submitted sample and test results on sample derivatives (also referred to as derived samples).

The REST resources and their relationships are explained in .

The full details of each resource are described in the .

General concepts

Requests are made to the API by sending XML messages:

• POST is used to create an item.

• GET is used to read an item.

• PUT is used to update an item.

• DELETE is used to delete an item.

Note: HEAD requests are not supported.

The full URL to which requests should be sent will vary depending on the specific installation, but will generally follow this format:

Automation / EPP: GUI trigger for calling scripts

Automation / EPP is used to trigger scripts from within the Clarity LIMS interface.

Script-triggering is often used because the data collected needs to be dispatched for further processing. Automating data processing and returning information, in the appropriate format, to the lab for immediate use increases efficiency and quality.

File handling and file management are fundamental elements in life science scripting. When triggered, scripts can issue a command, transfer files for processing, and collect and transfer files back to the server. To enable triggering of scripts in any programming language, the information and files are provided for batch processing at the operating system command line level.

For Clarity LIMS (v5 and later)

To use automation, administrators complete the following steps:

1. In Clarity LIMS, create and configure master steps.

2. Configure automations that trigger scripts. Enable those automations on the master steps.

3. Use the configured master steps as building blocks to create and configure steps to be run by lab scientists.

Related Resources

The REST API only returns 500 results per request. Because of this feature, with certain resources, you can use the start-index parameter and previous-page and next-page elements to work with large amounts of data.

For example, the following request is submitted to the API:

The response looks like this:

Note the presence of the previous-page and next-page URIs, which allow moving within the pages of results.

Viewing Results from a Specified Point

Use the start-index parameter to view results from a specified point in a list. The first record in a list is index 0, and you can use values that are positive, whole numbers. If the value specified is greater than the number of results available from a resource, the system returns an empty list.

Configuring the Maximum Number of Requests

By default, the REST API only returns 500 results per request. To change the default number, contact the Illumina Support Team.

When multiple users are working on multiple plates in high-throughput labs, programmers may find that the large number of HTTP method calls to the REST API can slow down their scripts.

To improve performance, Illumina has created the following batch resources:

• artifacts.batch.retrieve

• artifacts.batch.update

• containers.batch.create

• containers.batch.retrieve

• containers.batch.update

• files.batch.retrieve

• files.batch.update

• samples.batch.create

• samples.batch.retrieve

• samples.batch.update

Use the batch resources to access a group of artifacts or a group of containers using a single batch method call. Using these resources to iterate a list of items significantly improves script execution times.

Key Concepts

Batch resources are best thought of as unordered collections or lists of items accessed. A POST to batch/create, batch/update or batch/retrieve, therefore, is a request to create, update, or retrieve those items. There is no guaranteed order to batch responses.

Batch resources are nonbreaking additions to the existing REST API. Updated scripts can still use their existing nonbatch methods.

For example, the resources may have URIs (Universal Resource Identifiers) such as:

Batch operations do not require sophisticated HTTP client or server methods. The only HTTP method for batch resources is POST.

What is sent in a batch resource POST?

To update a group of artifacts, use a POST operation to the /artifacts/batch/update resource. The XML input payload consists of a series of elements, as follows.

What is returned from a batch resource POST?

As large data transfers can affect performance, it is important to return concise XML in response to a batch resource request. Therefore, except for retrieve resources, the XML output payload consists of a list of created or updated URI links, such as the following:

Return codes

The batch resources use common HTTP return codes:

• An HTTP 200 (OK) code is returned when batch resources have been successfully created or updated.

• An HTTP 400 error code is returned if the input payload details included incorrect, mixed, or duplicate URI links. For example, if the details of an artifacts.batch.update (list) request included a container resource.

Removing Fields and Types

To remove a UDF or UDT value, submit a PUT request with the desired UDF or UDT omitted from the XML.

Updating Resources

When submitting a PUT, it is critical to update all information. The submitted XML must include all the current UDFs and UDTs for the resource. If the field and type elements for a UDF and UDT are not included, the system removes those fields and types.

To update the UDF information for an item, the PUT request can add new UDF values and update or remove current UDF values. When working with UDTs, replace the current UDT with another UDT, or add or remove fields within the current UDT.

• Update all UDFs and UDTs in a PUT

• Even if the current user-defined values are not changing, include the current UDF and UDT values in the XML representation for a PUT request.

Filtering with UDF and UDT Values

Data formatting of the UDF and UDT values is important when filtering a resource list with a query parameter. When using UDF or UDT values as a query parameter, all nonalphanumeric characters must be URL encoded.

Data Type Formats

UDFs and UDTs are presented as fields and types in the XML. The following example shows a representation of fields and types returned by a GET request for a sample:

Data Type Values in XML

XML resource representations do not render UDF values in the same format as views in the client user interface. The table later in this section compares images taken from the client user interface and the XML from an http GET.

The differences are intentional, to remove ambiguity and aid script writers when handling the data values.

The following table compares the values displayed by the user interface and the API for UDF data types.

Numerics and Significant Digits

Trailing zeros are removed to support both integer and real numerics. Determine the significant digits by looking up the display-precision element of the /configuration/udfs/{udfid} resource.

EPP UDF Date format

When Date UDFs are expanded on the EPP command line, their format differs from the one used in the Clarity LIMS GUI and the REST API.

When submitting a request to the REST API, specify the version of the API being used. The version number is a path parameter in each resource URI. The desired API version is substituted into the request URI as follows:

Changes to the API are tracked with a version number (version major) and a revision number (version minor).

• The version number indicates forwards and backwards compatibility.

• The revision number within the version describes features added to the API that will not negatively affect current functionality.

Only the version number is referenced as part of the request. The revision number simply tracks incremental enhancements to the API.

When a new version of the API is released, update the scripts and code as soon as possible.

Check API Version

To find out what version of the API is available from a given server, submit a GET request to the base API URI. For example, in a web browser, browse to:

The system will return the version:

Exception for Legacy Systems

The API was originally intended for internal use or for just a few customers. In those early days, API versioning was different. If working with legacy scripts, this older functionality can be maintained. For example, if scripts were written before v2 and have nondefault system configuration properties for api.prefix and api.rewrite on the server, the …/api/ URI lists the resources and does not provide version information.

When generating XML, explicitly set the document to UTF-8 character encoding.

If using other encoding methods (eg, MacRoman for OS X), special characters such as μg/ml are stored incorrectly. This could cause data integrity issues.

In Groovy, set the encoding attribute on the StreamingMarkupBuilder object as shown in the following example:

Traversing Up a Genealogy

The artifacts resource includes a parent-process element that provides a URI to the process that created an artifact.

To facilitate walking up the genealogy, the processes resource exposes the parent process for an input artifact in the input-output-map of a process:

The parent-process element does not display if the parent process is not supported by the API.

Traversing Down a Genealogy

The processes resource supports an inputartifactlimsid query parameter. This parameter limits the list of processes to those processes with one of the specified artifacts as an input.

1. Start with the initial sample.
2. The processes for the sample can be queried, as follows.
3. The process contains an input-output-map for the input artifact.
4. The steps can then be repeated using the LIMS ID of each output artifact that is associated with the input artifact.