Submit to a Compute Cluster via PBS
There are some algorithms which work well on massively parallel compute clusters. BCL Conversion is such an example and is the basis for this application example.
The concepts illustrated here are not limited to BCL Conversion; as such, they may also be applied in other scenarios. For instance the example PBS script below uses syntax for illumina's CASAVA tool, but could easily be re-purposed for the bcl2fastq tool.
Also, in this example, Portable Batch System (PBS) is used as the job submission mechanism to the compute cluster, which has read/write access to the storage system holding the data to be converted.
Example PBS file
For illustrative purposes, an example PBS file is shown here. (As there are many ways to configure PBS, it is likely that the content of your PBS file(s) will differ from the example provided.)
Process configuration
In this example, the BCL Conversion process is configured to:
Accept a ResultFile input.
Produce at least two ResultFile outputs.
The process is configured with the following process level UDFs:
The syntax for the external program parameter is as follows:
User Interaction and Results
The user runs the BCL Conversion process on the output of the Illumina Sequencing process. The sequencing process is aware of the Run ID, as this information is stored as a process level user-defined field (UDF).
The user supplies the following information, which is stored as process level UDFs on the BCL Conversion process:
The name of the folder in which the converted data should be stored.
Assumptions and Notes
Portable Batch System (PBS) is used as the job submission mechanism to the compute cluster.
The compute cluster has read/write access to the storage system holding the data to be converted.
There is an EPP node running on the Clarity LIMS server.
ClusterBCL.py:
