Workflows, chemistry, hardware, and software are continually changing in the lab. As a result, you can determine which samples were processed after a specific change happened.
Using the processes (list) resource you can construct a query that filters the list using both process type and date modified.
Before you follow the example, make sure you have the following items:
Samples that have been added to the system.
Multiple processes of the Cookbook Example type that have been run on different dates.
A compatible version of API (v2 r21 or later).
In Clarity LIMS, when you search for a specific step type, the search results list shows all steps of that type that have been run, along with detailed information about each one. This information includes the protocol that includes the step, the number of samples in the step, the step LIMS ID, and the date the step was run.
The following screenshot shows the search results for the step type Denature and Anneal RNA (TruSight Tumor 170 v1.0).
The list shows the date run for each step, but not the last modified date. This is because a step can be modified after it was run, without changing the date on which it was run.
To find the steps that meet the two criteria (step type and date modified), you must to do the following steps:
Request a list of all steps (processes), filtered on process type and date modified.
Once you have the list of processes, you can use a script to print the LIMS ID for each process.
To request a list of all processes of a specific type that were modified after a specified date, use a GET method that uses both the ?type and ?last-modified filter on the processes resource:
The GET call returns a list of the first 500 processes that match the filter specified. If more than 500 processes match the filter, only the first 500 are available from the first page.
In the XML returned, each process is an element in the list. Each element contains the URI for the individual process resource, which includes the LIMS ID for the process.
The URI for the list of all processes is http://yourIPaddress/api/processes. In the example code, the list was filtered by appending the following:
This filters the list to show only processes that are of the Cookbook Example type and were modified after the specified date.
The date must be specified in ISO 8601, including the time. In the example, this is accomplished using an instance of a Calendar object and a SimpleDateFormat object, and encoding the date using UTF-8. The date specified is one week prior to the time the code is executed.
All of the REST list resources are paged. Only the first 500 items are returned when you query for a list of items, such as http://youripaddress/api/v2/artifacts.
If you cannot filter the list, you must iterate through the pages of a list resource to find the items that you are looking for. The URI for the next page of resources is always the last element on the page of a list resource.
After requesting an individual process XML resource, you have access to a large collection of data that lets you modify or view each process. Within the process XML, you can also access the artifacts that were inputs or outputs of the process.
After running the script on the command line, output is be generated showing the LIMS ID for each process in the list.
Information about a step is stored in the process resource. In general, automation scripts access information about a step using the processURI, which links to the individual process resource. The input-output-map in the XML returned by the individual process resource gives the script access to the artifacts that were inputs and outputs to the process.
Processing a sample in the lab can be complex and is not always linear. This may be because more than one step (referred to as process in the API and in the Operations Interface in Clarity LIMS v4.x and earlier) is run on the same sample, or because a sample has to be modified or restarted because of quality problems.
The following illustration provides a conceptual representation of a Clarity LIMS workflow and its sample/process hierarchy. In this illustration, the terminal processes are circled.
The following illustration provides a conceptual representation of a LIMS workflow and its sample / process hierarchy. In this illustration, the terminal processes are circled.
This example finds all terminal artifact (sample)-process pairs. The main steps are as follows:
All the processes run on a sample are listed with a process (list) GET method using the ?inputartifactlimsid filter.
All the process outputs for an input sample are found with a process (single) GET.
Iteration through the input-output maps finds all outputs for the input of interest.
Before you follow the example, make sure you have the following items:
A sample to the system.
Several steps that have been run, with several steps run on a single output at least one time.
A compatible version of API (v2 r21 or later).
To walk down the hierarchy from a particular sample, you must do the following steps:
List all the processes that used the sample as an input.
For each process on that list, find all the output artifacts that used that particular input. These output artifacts represent the next level down the hierarchy.
To find the artifacts for the next level down, repeat steps 1 and 2, starting with each output artifact from the previous round.
To find all artifacts in the hierarchy, repeat this process until there are no more output artifacts. The last processes found are the terminal processes.
This example starts from the original submitted sample.
The first step is to retrieve the sample resource via a GET call and find its analyte artifact (derived sample) URI. The analyte artifact of the sample is the input to the first process in the sample hierarchy.
The following GET method provides the full XML structure for the sample including the analyte artifact URI:
The sample.artifact.@limsid contains the original analyte LIMS ID of the sample. For each level of the hierarchy, the artifacts are stored in a Groovy Map called artifactMap. The artifactMap uses the process that generated the artifact as the value, and the artifact LIMS ID as the key. At the top sample level, the list is only comprised of the analyte of the original sample. In the map, the process is set to null for this sample analyte.
To find all the processes run on the artifacts, use a GET method on the process (list) resource with the ? inputartifactlimsid filter.
In the last line of the example code, the processURI string sets up the first part of the URI. The artifact LIMSID is added (concatenated) for each GET call in the following while loop:
In the last line of the example code provided above, the processURI string sets up the first part of the URI.
The artifact LIMSID will be added (concatenated) for each GET call in the while loop below.
The while loop evaluates one level of the hierarchy for every iteration. Each artifact at that level is evaluated. If that artifact was not used as an input to a process, an artifact/process key value pair is stored in the lastProcMap. All the Groovy maps in the previous code use this artifact/process pair structure.
The loop continues until there are no artifacts that had outputs generated. For each artifact evaluated, the processes that used the artifact as an input are found and collected in the processes variable. Because a process can be run without producing outputs, a GET call is done for each of the processes to determine if the artifact generated any outputs.
Any outputs found will form the next level of the hierarchy. The outputs are temporarily collected in the outputArtifactMap. If no processes were found for that artifact, then it is an end leaf node of a hierarchy branch. Those artifact/process pairs are collected in the lastProcMap .
You can iterate through each pair of artifact and process LIMS IDs in outputArtifactMap and print the results to standard output.
Running the script in a console produces the following output:
When working with multiplexing, you can do the following:
When samples are processed in the lab, they are sometimes re-arrayed in complex ways that are pre-defined.
You can use the REST API and automation functionality that will allow a user to initiate a step that:
Uses a file to define a re-array pattern
Executes the step using that re-array pattern. Since the pattern is pre-defined, this will decrease the likelihood of an error in recording the re-array.
To accomplish this automation, you must be able to execute a step using the REST API. This example shows a simple step execution that you can apply to any automated step execution needed in your lab.
For a high-level overview of REST resource structure in Clarity LIMS, including how processes are the key to tracking work, see REST General Concepts.
Before you follow the example, make sure that you have the following items:
Samples that have been added to the system.
A configured step/process that generates analytes (derived samples) and a shared result file.
Samples that have been run through the configured process/step.
A compatible version of API (v2 r21 or later).
Information about a step is stored in the process resource in the API.
Information about a derived sample is stored in the analyte resource in the API. This resource is used as the input and output of a step, and also used to record specific details from lab processing.
To run a step/process on a set of samples, you must first identify the set of samples to be used as inputs.
The samples that are inputs to a step/process can often be identified because they are all in the same container, or because they are all outputs of a previous step / process.
For more information, refer to Find the Contents of a Well Location in a Container and View the Inputs and Outputs of a Process/Step.
In this example, you run the step/process on the samples listed in the following table.
After you have identified the samples, use their LIMS IDs to construct the URIs for the respective analyte (derived sample) artifacts. The artifact URIs are used as the inputs in constructing the XML to POST and execute a process.
You can use StreamingMarkupBuilder to construct the XML needed for the POST, as shown in the following example code:
Executing a process uses the processexecution (prx) namespace (shown in bold in the code example above).
The required elements for a successful POST are:
type – the name of the process being run
technician uri – the URI for the technician that will be listed as running the process
input-output-map – one input output map element for each pair of inputs and outputs
input uri – the URI for the input artifact
output type – the type of artifact of the output
In addition, if the outputs of the process are analytes, then the following are also needed:
container uri – the URI for the container the output will be placed in
value – the well placement for the output
The process type, technician, input artifact, and container must all exist in the system before the process can be executed. So, for example, if there is no container with an empty well, you must create a container before running the process.
The XML constructed must match the configuration of the process type. For example, if the process is configured to have both samples and a shared result file as outputs, you must have both of the following:
An input-output-map for each pair of sample inputs and outputs
An additional input-output-map for the shared result file
If the POST is successful, the process XML is returned:
If the POST is not successful, the XML returned will contain the error that occurred when the POST completed:
After the step / process has successfully executed, you can open the Record Details screen and see the step outputs.
RunningAProcess.groovy:
Steps can have user-defined fields (UDFs)/custom fields that can be used to describe properties of the steps.
For example, while a sample UDF/custom field might describe the gender or species of the sample, a process UDF/custom field might describe the room temperature recorded during the step or the reagent lot identifier. Sometimes, some of the information about a step is not be known until it has completed on the instrument, but after it was run in Clarity LIMS.
In this example, we will record the Actual Equipment Start Time as a process UDF/custom field after the step has been run in Clarity LIMS. The ISO 8601 convention is used for recording the date and time. For more information, see Filter Processes by Date and Type.
NOTE: In the API, information about a step is stored in the process resource.
As of Clarity LIMS v5, the term user-defined field (UDF) has been replaced with custom field in the user interface. However, the API resource is still called UDF.
Master step fields—Configured on master steps. Master step fields only apply to the following:
The master step on which the fields are configured.
The steps derived from those master steps.
Global fields—Configured on entities (eg, submitted sample, derived sample, measurement, etc.). Global fields apply to the entire Clarity LIMS system.
Before you follow the example, make sure you have the following items:
Samples added to the system.
A custom field named Actual Equipment Start Time that has been configured on a master step (master step field).
On the master step , you have configured the field to display on the Record Details milestone, in the Master Step Fields section.
You have run samples through a step based on the master step on which the Actual Equipment Start Time field is configured.
Detailed information for each step run in Clarity LIMS, including its name, LIMS ID, and custom fields can be viewed on the Record Details screen.
In the image below, an Actual Equipment Start Time master step field has been configured to display in the Step Details section of the Record Details screen. However, a value for this field has not yet been specified.
Before you can change the value of a process UDF/custom field, you must first request the individual process resource via a GET HTTP call. The XML returned from a GET on the individual process resource contains the information about that process. The following GET method provides the complete XML structure for the process:
The variable processNode now holds the complete XML structure retrieved from the resource at processURI.
The variable startTimeUDF references the XML element node contained in the structure that relates to the Actual Equipment Start Time UDF/custom field (if one exists).
The variable newStartTime is a string initialized with a value from the method parseInstrumentStartTimes. The details of this method are omitted from this example, but its function is to parse the date and time the instrument started from a log file.
The XML representations of individual REST resources are self-contained entities. Always request the complete XML representation before editing any portion of the XML. If you do not use the complete XML when you update the resource, you can inadvertently change data.
The following code shows the XML structure for the process, as stored in the variable processNode. There are no child UDF/custom field nodes.
After modifying the process stored in the variable processNode, you can use a PUT method to update the process resource.
You can check if the UDF/custom field exists by verifying the value of the startTimeUDF variable. If the value is not null, then the field is defined and you can set a new value in the XML. If the field does not exist, you will must append a new node to the process XML resource using the UDF/custom field name and new value.
Before you can append a node to the XML, you must first specify the namespace for the new node. You can use the Groovy built-in QName class to do this. A QName object defines the qualified name of an XML element and specifies its namespace. The node you are specifying is a UDF element, so the namespace is http://genologics.com/ri/userdefined. The local part is field and the prefix is udf for the QName, which specifies the element as a UDF/custom field.
To append a new node to the process using the appendNode method of the variable processNode (which appends a node with the specified QName, attributes, and value). Specify the following attributes for the UDF/custom field element:
the type
the name
Both of these elements must match a UDF/custom field that has been specified in the Configuration window for the process type.
The variable processNode now holds the complete XML structure for the process with the updated, or added, UDF named Actual Equipment Start Time.
You can save the changes you have made to the process using a PUT method on the process resource:
The PUT updates the sample resource at the specified URI using the complete XML representation, including the new value for Actual Instrument Start Time.
If the PUT was successful, it returns the XML resource, as shown below. The updated information is also available at the http://yourIPaddress/api/v2/processes/A22-BMJ-100927-24-2188\ URI.
If the PUT was unsuccessful, an XML resource is returned with contents that detail why the call was unsuccessful. In the following example error, an incorrect UDF/custom field name was specified. A UDF/custom field named Equipment Start Time was created in the process resource, but no UDF/custom field with that name was configured for the process type/master step.\
The Step Details section of the updated Record Details screen now shows the Actual Equipment Start Time value.
The ability to modify process properties allows you to update automatically and store lab activity information as it becomes available. Information from equipment log files or other data sources can be collected in this way.
Updating per-run or per-process/step information is powerful because the information can be used to optimize lab work (eg, by tracking trends over time). The data can be compared by instrument, length of run, lab conditions, and even against quality of molecular results.
UpdateProcessUDFInfo.groovy:
Pooling steps require that each input analyte artifact (derived sample) in the step be inserted into a pool. You can automate this task by using the API steps pooling endpoint. Automation of pooling allows you to reduce error and user interaction with Clarity LIMS.
In this example, a script pools samples based on the value of the pool id user-defined field (UDF)/custom field of the artifact.
As of Clarity LIMS v5, the term user-defined field (UDF) has been replaced with custom field in the user interface. However, the API resource is still called UDF.
Master step fields—Configured on master steps. Master step fields only apply to the following:
The master step on which the fields are configured.
The steps derived from those master steps.
Global fields—Configured on entities (eg, submitted sample, derived sample, measurement, etc.). Global fields apply to the entire Clarity LIMS system.
To keep this example simple, the script does not handle samples with reagent labels.
In the API, an artifact is an item generated by an earlier step. There are two types of artifacts: analyte (derived sample) and resultfile (measurement). In the Clarity LIMS web interface, the terms artifact, analyte, and resultfile have been replaced with derived sample or measurement.
Before you follow the example, make sure that you have the following items:
A configured analyte UDF/derived-sample custom field named pool id in Clarity LIMS.
Groovy that is installed on the server and accessible at /opt/groovy/bin/groovy
The GLSRestApiUtils.groovy file is stored in /opt/groovy/lib/
The WorkingWithStepsPoolingEndpoint.groovy script that is stored in /opt/gls/clarity/customextensions/
A compatible version of API (v2 r21 or later).
In Clarity LIMS, under Configuration, select the Lab Work tab.
Select an existing Pooling master step or add a new one.
On the master step configuration form, select the Pooling milestone.
On the Pooling Settings form, set the Label Uniqueness toggle switch to Off.
Select Save.
Add a new protocol.
With the protocol selected, add a new Library Pooling step based on the master step you configured.
In Clarity LIMS, under Configuration, select the Automation tab.
Add a new step automation. Associate the automation with the WorkingWithStepsPoolingEndpoint.groovy script. The command line used in this example is as follows.
bash -c "/opt/groovy/bin/groovy -cp /opt/groovy/lib /opt/gls/clarity/customextensions/WorkingWithStepsPoolingEndpoint.groovy -u {username} -p {password} -s {stepURI:v2:http}"
Enable the automation on the configured pooling master step. Select Save.
You can now configure the automation trigger on the step or the master step. If you configure the trigger on the master step, the settings will be locked on all steps derived from the master step.
On the Lab Work tab, select the library pooling step or master step.
On the Step Settings or Master Step Settings form, in the Automation section, configure the automation trigger so that the script is automatically initiated at the beginning of the step:
Trigger Location—Step
Trigger Style—Automatic upon entry
In Clarity LIMS, under Configuration, select the Lab Work tab.
Select the pooling protocol containing the Library Pooling step.
Add the Add Pool ID step that sets the pool id custom field of the samples. Move this step to the top of the Steps list.
Select the Add Pool ID step.
On the Record Details milestone, add the pool id custom field to the Sample Details table.
In Clarity LIMS, under Configuration, select the Lab Work tab.
Create a workflow containing the configured pooling protocol. Activate the workflow.
On the Projects and Samples screen, create a project and add samples to it. Assign the samples to your pooling workflow.
Begin working on the samples. In the first step, enter values into the pool id custom field.
Continue to the Library Pooling step and add samples to the Ice Bucket. Select Begin Work to execute the script.
The script is passed the URI of the pooling step. Then, using the URI, the pool node of the step is retrieved. This node contains an available-inputs node that lists the URIs of the available input artifacts.
The script retrieves all available input artifacts, and then iterates through the list of retrieved artifacts. For each artifact, the script looks for the pool id custom field. If the field is not found, the script moves on to the next artifact. If the field is found, its value is stored in the poolID variable.
When the script encounters a new pool ID, it creates a new pool with a name equal to that ID. Input artifacts are sorted into pools based on the value of its pool id field, and as they are inserted into pools they are removed from the list of available inputs.
After all of the available inputs are iterated through, the updated pool node is sent back to Clarity LIMS:
Artifacts with the same Pool ID UDF / custom field will be automatically added to the same pool.
WorkingWithStepsPoolingEndpoint.groovy:
GLSRestApiUtils.groovy:
Submitted Sample Name
Derived Sample Name
Derived Sample LIMS ID
Container LIMS ID
Container Type
Well
Soleus-1
Soleus-1
AFF853A53AP11
27-4056
96 well plate
A:1
Soleus-2
Soleus-2
AFF853A54AP11
27-4056
96 well plate
A:2
Soleus-3
Soleus-3
AFF853A55AP11
27-4056
96 well plate
A:3
