Laboratories may want to limit what steps Researchers can start. At the time of writing, BaseSpace Clarity LIMS does not natively support protocol-based permissions. However, with an EPP at the beginning of the step we can check to see if the technician/researcher starting the step has been given approval to start the step, and halt the step from starting if they do not have permission. There are several ways this can be done, but special considerations to how these permissions are administered need to be made.

Solution

In order to allow an administrator to easily maintain permissions we will assign users to groups in a config file and our EPP will consume this information. One parameter of the EPP is the groups that are permitted to run the step. When the script is triggered at the start of the step, it will look for the name of the technician starting the step in the config file and determine if the technician is:

• Included in the config file and,

• Has been assigned to a group that is permitted to run the step.

It is important to remember that by exiting a script with a negative number an EPP will fail and the user will not be able to move forward in the step. We will take advantage of this EPP feature and if the technician/researcher is part of a permitted group the step would start as expected. But, if they are not part of a permitted group, entry into the step will be halted and an error box will appear with whatever the last print message was in the script.

Parameters

The EPP command is configured to pass the following parameters:

An example of the full syntax to invoke the script is as follows:

User Interaction

• The config file can reside in any directory that the EPP script will have access to.

• The config file that is used in this example has tab delimited columns of Last Name, First Name, and Groups. The permitted groups need to be separated by commas (see the attached example config file). The script can be easily modified if a different format is desire for the config file

• The EPP should be "automatically initiated" at "the beginning of the step"

If the user is not allowed to move forward a message box will appear and the step is aborted.

Assumptions and Notes

• You are running a version of Python that is supported by Clarity LIMS, as documented in the Clarity LIMS Technical Requirements.

• Both of the attached files are placed on the Clarity LIMS server, in the /opt/gls/clarity/customextensions folder.

• The example code is provided for illustrative purposes only. It does not contain sufficient exception handling for use 'as is' in a production environment.

Attachments

Group_Permissions.py:

config.txt:

Often, workflows do not have a linear configuration. Even in those cases, samples progressing through workflows may be re-queued, replicated, or submitted into a number of parallel workflows.

Attempts to align downstream results with submitted samples may get hindered when trying to account for sample replicates or the dynamic decisions made in the lab.

A visual representation of a samples complete history presented in a clear hierarchical format allows for a digestible report of the work done on the sample. This format provides at-a-glance understanding of the any of the branching or re-queuing of a sample.

Solution

This example describes a python script which, given an artifact, recursively finds all the processes for which that artifact was an input, then finds all the associated output artifacts of that process. This continues for all processes all the way down to the most downstream artifact.

A clear visual representation of the entire history of work on a sample, similar to what was available in the Ops interface, can allow a user to see all the processes and derivations of a sample. This is especially of use for troublesome samples that have branched into numerous downstream replicates which may end up in the same or different sequencing protocol.

Parameters

The script accepts the following parameters:

An example of the full syntax to invoke the script is as follows:

Script output example:

Sibling artifacts will appear aligned vertically with the same indentation. In the above example, 122-1650 Library Pooling (MiSeq) 5.0 created two replicate Analytes, 2-4110 and 2-4111. Analyte 2-4111 was the input to the subsequent step ( 24-1952 ) and no additional work was performed on 2-4110.

Processes performed on an artifact will appear underneath with a tab indentation. In the above example, the first 4 processes ( 3 QC processes and Fragment DNA ) are all using the Root Analyte (CRA201A1PA1) as an input.

Adding Colours with termcolor

Install package termcolor for colour printing support. Entity colours can be configured within the script. Globally turn off colours by changing the variable use_colours to False ( line 16 ).

Assumptions and Notes

• Your configuration conforms with the script's requirements, as documented in .

• You are running a version of Python that is supported by Clarity LIMS, as documented in the Clarity LIMS Technical Requirements.

• The glsapiutil.py file is placed in the working directory.

Attachments

sample_history_colours.py:

sample_history.py:

• Illumina LIMS Integration

• Generating a Hierarchical Sample History

• Protocol-based Permissions

Sequential numbers are sometimes needed for naming conventions and require self-incrementing counters be created and maintained. We do not recommend using the BaseSpace Clarity LIMSdatabase for this. However the Unix “(n)dbm” library provides an easy way to create and manage counters by creating Dbm objects that behave like mappings (dictionaries).

Solution

They way this would work is the attached script ( and the counters file it creates / manages ) would live on the Clarity server and other scripts would depend upon it, and use code similar to below whenever a sequential number was needed. While the script is written in python and uses the dbm module there is nothing inherently Pythonic about this code that couldn’t be reimplemented in another language. However, more information on the Python dbm module can be found at: https://docs.python.org/2/library/dbm.html

User Interaction

• The counters live in a file, the path to which is defined in the cm.setPath() command. The file will be created if it doesn’t exist.

• The file can contain as many counters as you wish (it’s better to have many counters in one file than many files each with only one counter)

• The name of the counter is passed to the function cm.getNextValue(). If this is the first time the counter has been used, it will be created and added to the file.

Assumptions and Notes

• You are running a version of Python that is supported by Clarity LIMS, as documented in the Clarity LIMS Technical Requirements.

• The example code is provided for illustrative purposes only. It does not contain sufficient exception handling for use 'as is' in a production environment.

Attachments

clarityCounters.py:

When a lab takes in samples, they are named by the lab scientist who supplies them. As such, samples may be named in any way imaginable. Conversely, the research facilities processing the samples often have strict naming conventions.

When these two situations occur, there is a need for the sample to renamed with the strict nomenclature of the processing lab. However, in order for the final data to be meaningful for the scientist who supplied the samples, the original name must also be retained.

Solution

In this example, the attached script may be used to rename samples, while retaining their original names in a separate field.

This combination of configuration and python script can be used to set up an integration between Clarity LIMS and Illumina LIMS. There are 2 main parts to this integration:

1. Generating a sample manifest from Clarity LIMS to import the samples into Illumina LIMS.

2. Once the analysis is completed, automatically parsing in the results from Illumina LIMS into Clarity LIMS.

Clarity LIMS can create Illumina-based MiSeq and HiSeq 'flavoured' sample sheets. However, if you are using algorithms or indexes outside of those suggested by Illumina, you may be required to produce your own 'custom' sample sheet.

This example script provides an algorithm that harvests the contents of a flow cell (or any container) that may contain pooled samples, and uses the resulting information to output a custom sample sheet.

Solution

The attached script uses aggressive caching in order to execute as quickly as possible. When extreme levels of multiplexing are involved, the cache size could consume considerable quantities of memory, which may be counter-productive.

Compatibility: API version 2

Many different types of CSV files are attached to BaseSpace Clarity LIMS. This example provides a template for a script that can parse a wide range of simple CSV files into Clarity LIMS. The user can change parameters to read the format of the file to be parsed.

Solution

Protocol step configuration

CSV files are attached to a protocol step. Artifact UDFs, where data will be written to, need to be configured for artifacts and for the protocol step.

Parameters

The script accepts the following parameters:

An example of the full syntax to invoke the script is as follows:

About the Code

The script contains an area with a number of configurable variables. This allows a FAS or bioinformatician to customize the script to parse their specific txt file. The following variables within the script are configurable:

MAPPING MODES

There are many attributes of samples which can be used to map the data in the text file with the corresponding derived samples in Clarity LIMS. The script should be configured such that one of these modes is set to True.

Three modes are available:

For any of the three modes, a mapping column value must be explicitly given. The value is the index of the column containing the mapping data (either artifact name, well location, or UDF value).

If using the mode MapTo_UDFValue, a UDFName must be given. This is the name of the UDF in clarity which will be used to match the value found in the mapping column.

artifactUDFMap

This Python dictionary maps the name of columns in the txt file to artifact UDFs for the outputs of the step. The data from these columns in the file will be written to these UDFs for the output artifacts. The dictionary can contain an unlimited number of UDFs. The dictionary keys, (left side), are the names of the columns in the txt file, and the dictionary values, (right side), are the names of the UDFs as configured for the artifacts.

Assumptions and Notes

• You are running a version of Python that is supported by Clarity LIMS, as documented in the Clarity LIMS Technical Requirements.

• The attached files are placed on the LIMS server, in the /opt/gls/clarity/customextensions folder.

• The example code is provided for illustrative purposes only. It does not contain sufficient exception handling for use 'as is' in a production environment.

Attachments

genericParser.py:

glsfileutil.py:

Compatibility: API version 2 revision 21

A lab user can attach a PDF file containing multiple images to a result file placeholder, the script extracts the images which are automatically attached to corresponding samples as individual result files.

Images within a PDF may be in a number of formats, and will usually be .ppm or .jpeg. The example script includes additional code to convert .ppm images to .jpeg.

Prerequisites

It may at times be desirable to take key data derived during a workflow and copy it to the submitted sample. There are several reasons why this could be useful:

• All key data is combined with all of the submitted sample's data, and becomes available on a single object.

• Key data determined during a workflow can be made immediately available to external collaborators via the LabLink Collaborations Interface, since these users have access to their submitted samples.

Once a sequencing run has occurred, there is often a requirement to store the locations of the FASTQ / BAM files in Clarity LIMS.

For paired-end sequencing, it is likely that the meta-data file that describes the locations of these files will contain two rows for each sample sequenced: one for the first read, and another for the second read.

Such a file is illustrated below:

Column 2 of the file, Sample ID, contains the LIMS IDs of the artifacts for which we want to store the FASTQ file values listed in column 3 (Fastq File).

This example discusses the strategy for parsing and storing data against process inputs, when that data is represented by multiple lines in a data file.

Solution

The attached script will parse a data file containing multiple lines of FASTQ file values, and will store the locations of those FASTQ files in user-defined fields.

Process configuration

In this example, the process is configured to have a single shared ResultFile output.

Parameters

The EPP command is configured to pass the following parameters:

An example of the full syntax to invoke the script is as follows:

User Interaction

The user-interaction is comprised of the following steps:

Step 1

The user runs the process up to the Record Details screen as shown in the following image. Note that initially:

• The Sequencing meta-data file is still to be uploaded.

• The values for the R1 Filename and R2 Filename fields are empty.

Step 2

The user clicks Upload file and attaches the meta-data file. Once attached, the user's screen will resemble this:

Step 3

Now that the meta-data file is attached, the user clicks Parse Meta-data File. This invokes the parsing script.

If parsing was successful, the user's screen will resemble Figure 4 below.

Note that the values for the R1 Filenames and R2 Filenames have been parsed from the file and will be stored in Clarity LIMS.

About the code

The key methods of interest are main(), parseFile() and fetchFile(). The main() method calls parseFile(), which in turn calls fetchFile().

fetchFile() method

The fetchFile() method relies upon the fact that the script is running on the Clarity LIMS server, and as such has access to the local file system in which the file (uploaded in Step 2) now resides.

Thus, fetchFile() can use the API to:

1. Convert the LIMSID of the file to the location on disk.

2. Copy the file to the local working directory, ready to be parsed by parseFile().

parseFile() method

1. The parseFile() method creates two data structures that are used in the subsequent code within the script:

   • The COLS dictionary has the column names from the first line of the file as its key, and the index of the column as the value.

   • The DATA array contains each subsequent line of the file as a single element. Note that this parsing logic is overly-simplistic, and would need to be supplemented in a production environment. For example, if the CSV file being parsed does not have the column names in the first row, then exceptions would likely occur. Similarly, we assume the file being parsed is CSV, and likewise any data elements which themselves contain commas would likely cause a problem. For the sake of clarity such exception handling has been omitted from the script.

Assumptions and Notes

• Both of the attached files are placed on the Clarity LIMS server, in the /opt/gls/clarity/customextensions folder.

• You will need to update the HOSTNAME global variable such that it points to your Clarity LIMS server.

• The example code is provided for illustrative purposes only. It does not contain sufficient exception handling for use 'as is' in a production environment.

Attachments

parseMetadata.py:

There are some algorithms which work well on massively parallel compute clusters. BCL Conversion is such an example and is the basis for this application example.

The concepts illustrated here are not limited to BCL Conversion; as such, they may also be applied in other scenarios. For instance the example PBS script below uses syntax for illumina's CASAVA tool, but could easily be re-purposed for the bcl2fastq tool.

Also, in this example, Portable Batch System (PBS) is used as the job submission mechanism to the compute cluster, which has read/write access to the storage system holding the data to be converted.

Example PBS file

For illustrative purposes, an example PBS file is shown here. (As there are many ways to configure PBS, it is likely that the content of your PBS file(s) will differ from the example provided.)

Solution

Process configuration

In this example, the BCL Conversion process is configured to:

• Accept a ResultFile input.

• Produce at least two ResultFile outputs.

The process is configured with the following process level UDFs:

The syntax for the external program parameter is as follows:

Parameters

User Interaction and Results

1. The user runs the BCL Conversion process on the output of the Illumina Sequencing process. The sequencing process is aware of the Run ID, as this information is stored as a process level user-defined field (UDF).

2. The user supplies the following information, which is stored as process level UDFs on the BCL Conversion process:

   • The name of the folder in which the converted data should be stored.

Assumptions and Notes

• Portable Batch System (PBS) is used as the job submission mechanism to the compute cluster.

• The compute cluster has read/write access to the storage system holding the data to be converted.

• There is an EPP node running on the Clarity LIMS server.

Attachments

ClusterBCL.py: