Use a configuration package file to copy a configuration set from one server to another or to back up a particular working configuration at a particular time.
Required steps:
Create a configuration manifest file.
Export to configuration package file.
This process involves copying a configuration set from one server to another, by importing a configuration package file.
For example, to move a configuration set to a different environment for testing or troubleshooting purposes, or copy a new configuration set (created and tested on one system) onto another system.
Required steps:
On the source server, create a configuration manifest file, and then export to configuration package file.
On the destination server, import the configuration package file.
There are two approaches:
Comparing configuration manifest files provides a way to determine if there are processes or UDFs missing from a system. The information in the manifest files only allows comparing process and UDF names, not the specific way in which a process or UDF is configured.
Comparing configuration package files helps check how specific processes are configured. If the systems being compared are meant to be identical, this method is more appropriate to use.
Required steps:
On each system, create a configuration manifest file, or a configuration package file.
Run a diff comparison on the two files.
Edit the broken manifest file, export it, and import the resulting configuration package file into the system to add the missing entities.
There are several tools that available to compare files:
Meld (graphical), for Linux, port to MacOS
Standard Unix diff (Linux, MacOS) (use -q for a quick check).
FileMerge (OSX with XCode installed) - /Developer/Applications/Utilities/FileMerge.app
WinMerge (graphical), for Windows - http://winmerge.org/
Combine configuration sets from multiple systems, merge them into a single configuration package file, and then import the file into a new system.
Required steps:
On each source server, create and edit a configuration manifest file.
Merge the entities from all files into a single manifest file.
Export the resulting file to a configuration package file.
On the destination server, import the merged configuration package file.
Copy a configuration set to restore it on another server and use it for testing/troubleshooting purposes.
Required steps:
On the server containing the 'broken' source system, create a full manifest file, containing all of the LIMS system configuration.
Export the manifest to a configuration package file. Save file to media/disk.
On the target server, import the configuration package file created on the source system.
To upgrade or add to a configuration set already installed on a server, two configuration package files are needed: one to back up the working configuration set and one containing the new updated configuration that have been created on a test server.
Required steps:
On the server you want to upgrade, create a full manifest file and export this to a configuration package file. Save this file as a backup.
On the test server, create a manifest file and edit it so that it only includes the entities you want to import.
On the server you want to upgrade, import the configuration package file.
You may want to take a configuration that has been created and tested on one system/site (referred to as the source system in the steps below), and deploy it on another system/site (destination system).
Take a configuration that has been created and tested on one system/site (referred to as the source system in the following steps) and deploy it on another system/site (destination system).
As a best practice, make sure that the configuration is backed up by creating a full manifest file and exporting to a configuration package file (see Step 2). The process is also described in #backup_restore.
Required steps:
On the source system, create a configuration package file containing the tested configuration to import.
On the destination system, create a full manifest file and export this to a configuration package file. Save this file as a backup.
On the destination system, import the configuration package file that was created in step 1.
If there is an API version mismatch, Config Slicer will log a message at the beginning of an import:
Generally, this message is not a cause for concern.
However, if there are warnings about configuration differences during the import, changes that have been made to the API in between the package version and the server version may be responsible.
In addition, if the package being imported is from v4.x, a log message may appear at the beginning of an import:
Detected package is from a pre-5.0 Clarity system. The package will be upgraded to match new configuration requirements.
In this case, the package is upgraded and the resulting configuration is output to a folder in the same location as the package being imported. This upgraded configuration package is the v5.x representation of the v4.x configuration and can be used directly to troubleshoot error occurring on import. The updated packaged can also be imported directly.
The following changes have been included in the latest release of Config Slicer (v 3.0.51):
The entity summary is now shown after the individual differences have written to the log file, as opposed to before these differences.
Support was dropped for the following process type attributes which are no longer used:
SupportsExternalProgram,
ShowInExplorer,
ShowInButtonBar,
OpenPostProcess,
IconConstant
Support was dropped for the show-in-tables property on Custom Fields (UDFs) because it is no longer used.
There is now support for the new Clarity LIMS 5.0 distinction between Protocol Step names and Master Step (ProcessType) names.
It is now possible to slice in and out all the new Master Step settings added in Clarity 5.0, including:
Default Process Template
Instrument Types
Container Types
Reagent Kits
Reagent Types
Control Types
Sample Fields
Queue Fields
Ice Bucket Fields
Step Fields
Step Properties
When importing/validating a slice from 4.x into 5.x, if the package contains any Master Steps (process types) or Protocols, it is upgraded to be compatible with the new configuration available in 5.x. This process is automatic, and the updated package is written out to the directory containing the package being imported or the directory that log file is being written to. If errors occur while importing, this updated package can be manipulated directly and imported to fix them.
Note: If upgrading the package fails, import/validation will fail. In general, this will reflect a mistake in the configuration package being imported.
The following changes were specifically added to support slicing between Clarity LIMS 4.x and BaseSpace Clarity LIMS 5.0. All of these changes will be saved into a new configuration package that will be written to the same location as the package being sliced:
Backwards compatibility for Protocol Step setup configuration.
Support for updates of parent entities after both the parent and child entities have been imported.
This change is required for updating the defaultProcessTemplate and step-fields step properties on Master Steps. Both require that the Master Step (ProcessType), ProcessTemplate, and Master Step Custom Fields (ProcessType UDFs) exist before they can be set on the Master Step.
Support for setting the qcProtocolStep flag on Master Steps, allowing the correct Master Step type to be displayed in the Lab Work Configuration UI. The setting is propagated up from the Protocol Step to the Master Step.
After slicing, it is recommended to examine the configuration closely for any QC Protocol Steps that previously shared a Master Step with nonQC Protocol Steps. This setting may not transfer as expected and there is potential for misconfigured Protocol Steps. In particular, if the Master Step produces measurements, and even just one child step of a Master Step has qcProtocolStep=true, then the Master Step will get qcProtocolStep=true set on it. Thus, all other steps that use that Master Step have qcProtocolStep=true, whether or not it was set before.
Support for setting the default container on Steps and Master Steps in such a way that all behaviour is maintained from 4.2
If every child step of a Master Step has the default container that was defined as a permitted container on the master (through the 'OutputContainerType' process-type-attribute) then the default will be added as a permitted container on the master step
If any child does not have the default container from the Master Step, then the default is removed from the Master and each child that had the container is updated so that the default is the first permitted container (and hence default)
Extra containers and any step properties that are no longer valid on a step will be migrated to new properties or removed.
Step-setup file configuration has been moved to the Master Step and it not possible to have a different set of step-setup files on a step than are specified on the Master. The master step owns the list of files. Both the search-result-file-index attribute on each file element and the message element for each file are defined by the Master and must be duplicated on every step. When moving a 4.x configuration with step-setup files to a 5.x configuration, the following events will occur:
The set of all the step-setup files found on all the child steps in the slice are added to to the Master Step and each child step.
If more than one step in the slice defines step-setup files for the same search-result-file-index, then all the messages for that search-result-file-index will be concatenated by newlines.
The enabled attribute will be set to true for the step-setup on all child steps.
The locked attribute will be set to false for the step-setup on all child steps.
In the case of importAndOverwrite, the step-setup of any existing child steps for an overwritten master step will be included.
Upon validation after import, the following differences have been reconciled:
UDFs that differ by STYLE only
missing defaultProcessTemplate
missing attemptAutoPlacement
missing autoAttachFiles
missing qcWithPlacement
If running Config Slicer v3.0.x against a configuration package or manifest file that was generated with a pre-3.0 version of Config Slicer, an error message displays.
In this scenario the error message will resemble the following:
SOLUTION:
Upgrade the configuration package file by running upgrade-config-slice.jar against it.
This jar file should be located in the same directory as Config Slicer (/tools/config-slicer).
The upgrade script will save an upgraded copy of the configuration package file (to the same directory), which can be inspected and imported.
Example:
To upgrade a manifest file at the same time, simply add it to the script as a second argument:
In this scenario the error message resembles the following:
SOLUTION:
This upgrade process is a little more involved:
Extract the list of process types from the manifest file .
Format them as parameters to Config Slicer's custom manifest generation for process types.
For example, assuming we extracted the process types Process Type 1 and Process Type 2 in step 1, the following command would be used:
Run the custom manifest generation.
Take the list of analyte UDFs and ResultFile UDTs from the manifest file that was generated and add them to the old manifest file .
Add the following line of text to the top of the old manifest file:
The Config Slicer tool is used to move small incremental configuration changes, contained in a configuration set, between Clarity LIMS systems. For example, it moves changes between a test system and a production system.
This configuration tool provides granular export/import functionality that allows the management of configurations that support experimental workflows.
Use this tool to back up, copy, deploy, and restore configuration sets. By making small incremental changes, make sure that the modifications made to the production system are minimal.
Review the following key concepts:
Configuration set—This item may be created by the Illumina Support team or by the customer. It comprises the items (know as entities) that are added to a Clarity LIMS system to allow for customization for a particular scientific experiment or workflow. The Illumina NGS Extensions Package is a good example. See #supported-entities.
Configuration manifest file—This text file determines the configuration set to be exported from a system. The manifest file does not contain the actual configuration data. It only drives the extraction of configuration from a system.
Configuration package file—This XML file contains the top-level entities selected by the configuration manifest file, plus any related child entities. For example, for process types, it includes process type UDFs, process templates (protocols), and output UDFs.
The Config Slicer tool uses an export/import process to transfer configuration sets from a source to a destination server. This process breaks down into the following tasks:
On the source Clarity LIMS server, use the Config Slicer tool to create a configuration manifest file.
Edit the manifest file so that only the required custom configuration set is preserved.
Use the Config Slicer tool to export the edited manifest file to a configuration package file.
On the destination Clarity LIMS server, use the Config Slicer tool to import the configuration package file into the system.
A configuration package file can be imported into multiple systems. Use this feature to create and import multiple custom configuration sets, such as the Illumina TruSeq integration. This functionality also provides the Illumina Support team with a scalable way to keep up with constantly changing protocols.
A configuration set comprises the entities added to a Clarity LIMS system that allow for customization of the system for a particular scientific experiment or workflow. The following entities are currently supported by the Config Slicer tool:
Sample UDFs and UDTs
Container UDFs and UDTs
Project UDFs and UDTs
Artifact groups (experiments)
Reagent types
Control types
Reagent kits
Process types (any configured processes – for example, Pool Samples and Add Multiple Reagents)
Process UDFs and UDTs
Output UDFs
Process templates - UDFs, UDTs, and parameter strings only (other entities such as instruments and researchers are not supported)
Protocols
Workflows
When performing custom manifest generation by workflow, protocol, or process type, the following entities are not exported: Sample, Container, Project UDTs, and UDFs for Project. Account (Lab) and Client (Researcher) UDFs are never exported by config slicer. These are known issues.
Config Slicer does not export/import nonstep automation, nor does it preserve the order of protocols.
When working with Config Slicer on the Clarity LIMS application server, there are no additional prerequisites. The latest version of the config-slicer.jar file is installed as part of Clarity LIMS on the Clarity LIMS server. In a default installation, find the file in the following location:
To work with Config Slicer on a machine other than the Clarity LIMS server, do the following:
Make sure that Java is installed.
Copy the /opt/gls/clarity/tools/config-slicer directory, and its contents, from the Clarity LIMS server to the machine. The config-slicer directory and the config-slicer package should contain the following:
config-slicer-<version>.jar
libs subdirectory (which includes all the libraries referenced by config-slicer-<version>.jar, including groovy-all-2.4.8.jar)
upgrade-config-slice.jar
Refer to #import-export-mode and #guidelines-and-semantics-for-creating-manifest-files.
| Option | Description |
|---|---|
* The importAndOverwrite option lets Config Slicer update existing configuration, rather than create new configuration.
Created and validate a configuration set on the source server.
Have access to the Config Slicer tool and the libs subdirectory.
Create Simple manifest file or Custom manifest file.
Simple manifest file:
On the source server, copy and paste following code to the command line. Edit the variables (version, server IP address, username, password, and manifest file name) to match those in your system.
A command with the variables filled in might look like this:
This step produces a manifest file containing information about the entire system configuration. For best practice, copy this file and rename the copy in a way that reflects the configuration (we'll use newconfiguration.txt for this example). Use the copied file for the next steps.
A manifest file is used as an intermediary step to produce an XML configuration package file.
The manifest file is only relevant to the data that exists in the system at the time it is created. Discard it after creating the configuration package file or save the manifest file for historical auditing purposes. It can provide a record of a known working configuration set on a particular system.
Custom manifest file:
To create a manifest file for only specific workflows, protocols, or process types, follow the steps outlined previously, using -o custom instead of -o example.
When using this operation provide additional parameters (-w, -pr, and/or -pt) specify the exact entities for which to create a manifest.
For example:
Specifying every option is not required. It is also possible to specify more than one of each kind. For example, create a manifest file for a workflow with the following command:
Or for two protocols like so:
Or for two process types and a protocol with this command:
The next step is to edit the manifest file, removing unnecessary information and preserving only the custom configuration to import into the destination system.
Example 1: In this example, everything is deleted from the manifest file, except for the two new process types to export.
Example 2: In this example, the manifest file contains definitions for some new reagent types:
Copy and paste the following code onto the command line. Edit the variables (version, server IP address, username, password, and manifest and package file names) as required.
An edited command might look like the following example:
This step generates a data file in an XML format (newconfiguration.xml in our example) that is compliant with the Rapid Scripting API.
A configuration package file has been exported. This example uses a file named newconfiguration.xml.
Access to the Config Slicer tool on the destination server has been granted.
There are no in progress steps for any of the protocols that are going to be sliced in, otherwise the import of the protocol fails.
On the destination server, copy and paste the following code to the command line. Edit the variables (version, package file name, server IP address, username, and password) as required.
A command with the variables filled in might look like this:
If any of the configuration entities that are about to import, exist in the destination system, Config Slicer either logs a warning or attempts to update them. It depends on the mode being run (see #import-export-mode).
If Clarity LIMS has maintained an internal record of deleted items, the previous information may also apply to deleted entities. This situation may occur if those entities have created outputs that currently still exist in the system.
When running in import mode, entities that exist and are different from the version in the package have a warning and full diff logged.
When running in importAndOverwrite mode, Config Slicer attempts to update entities that exist and are different from the version in the package.
In this scenario, back up configuration package containing copies of the updated entities before they were changed is saved to the directory where the configuration package is located. If that directory is not writable, the backup package is saved to the same directory as the log file.
If the version in the package is identical to the version on the server, no errors are logged and Config Slicer considers that entity successfully imported.
To avoid changing existing configuration (which could possibly break historical data), another option is manually renaming the old entities. Add an extension or a prefix and continue with importing the new configuration package.
Use the following methods to validate whether an import has completed successfully:
Check the Import Log:
For each specific type of configuration that is being imported (e.g. container types, process types, workflows), Config Slicer will log a set of messages. The messages look similar to the following examples:
Before it begins to process a specific entity, the file logs how many entities were found. Any errors or warnings about this set of entities always appear between Found 4 $Entities and Summary of $Entities.
Every entity that is found in the configuration package always appears in the summary, in one category or another. If a scenario occurs where this isn't true, or where the initial count of entities does not match the number in the summary, something has gone wrong and a bug report should be filed.
Validate with Config Slicer:
Running Config Slicer with the validate operation checks every entity in the package to see if it exists on the destination server. It reports results in a format similar to the log format shown previously.
Run the validate operation before or after importing:
Before importing—checks if there could be any problems when importing a configuration from a package. This feature is its primary use as it makes sure that during import, "configuration exists in package but not on server" is not considered an error case.
After importing—makes sure that the results are what was expected.
Example of validate output:
Check Configuration on the Destination Server:
The ultimate test of whether configuration has imported successfully is to check the configuration on the destination server itself. Make sure it looks and behaves as expected.
Configuration can be checked either via the Configuration screen in the Clarity LIMS user interface, or via the configuration endpoints in the API.
To be included in a configuration package file, the top-level entities of the custom configuration set must be explicitly enumerated.
Some of the top-level entities are discrete 'self-contained' units, and do not include other units (for example, container types, reagent types, artifact groups, and non-artifact/non-process type UDFs).
Some top-level entities (process types, for example) automatically include other units (refer to #non-top-level-entities for more information).
For process types, only configured processes, vanilla Transfer processes, Pool Samples (since 7.5), and Add Multiple Reagents (since 7.6) processes are supported. All process type details are exportable/importable.
Some entities are only included as part of other entities.
For example, process templates, process type UDFs, and artifact UDFs are only included when included in a top-level process type. (The latter is a special case, given that the same artifact UDF can be used by multiple process types.)
Some entities may be required by other entities. In these cases, make sure that these entities are exported/imported in the correct order.
For example, because process types may require the existence of a container type, create the container type first.
Required entities are not automatically included. If they do not exist in the destination system, explicitly include them in the manifest file.
For example, suppose that a process type declares a particular container type as a default output plate. If that container type does not exist in the destination system, include that container type in the manifest file.
Import modes affect the transactability of the tool, allowing it to make incremental changes if errors occur or provide an all-or-none option. For example, use the operation validate mode to determine if errors were encountered.
This is the default import mode.
This mode attempts to import as many units as possible. Any failures are logged, but the import operation is not interrupted.
For example, a failing container type import will not prevent other container types from being processed.
Similarly, if a process type fails import because it already exists, any UDFs and process templates for that process type will still be processed.
There is no need to enter an option for this mode.
If this mode is used, the import operation is aborted if it encounters a failure.
For example, if there is an API version mismatch, the operation will abort and no further imports will be executed. Note that any changes already performed are not reverted.
Use the -Strict option to enable this import mode.
Use the validate operation (instead of import) to enable this mode.
This mode produces a report listing showing the following items:
Entities that would be successfully imported because they do not exist on the target server.
Entities that already exist on the target server and are identical.
Entities that already exist on the target server but are different.
Validate mode can only detect a limited set of errors. For example, it can check if a particular piece of configuration already exists. If so, it checks if it is identical to the one included in the configuration package.
This information can help determine if the importAndOverwrite option is needed instead. For details, see #command-line-options-and-usage.
Example of console output:
Use this mode to generate a manifest file if the configuration you want to export is not tied to a specific set of workflows, protocols, or process types.
Use the example operation (instead of import) to enable this mode.
-a,--apiuri <apiuri>
The BaseSpace Clarity LIMS REST API base URI (ends in "/api") (Either this or --server must be provided)
-k,--package <package file>
File to be imported from or exported to (Required if operation is import, importAndOverwrite*, export, or validate). If file is not local a full path is required.
-f,--force <force>*
Force update without prompt when running in importAndOverwrite mode (Optional)
-m,--manifest <manifest>
Manifest file (Required if operation is export or example). If file is not local a full path is required.
-o,--operation <operation>
The operation mode for the Config Slicer tool.
Options are import, export, validate, example, importAndOverwrite, and custom (Required)
-p,--password <password>
The BaseSpace Clarity LIMS REST API password, if encrypted, use "ENC(<encrypted-password>" (Required)
-pr,--protocols <protocols>
The protocols to include in the custom manifest (Optional)
-pt,--processTypes <processTypes>
The process types to include in the custom manifest (Optional)
-s,--server <server>
The BaseSpace Clarity LIMS REST API server (either this or --apiuri must be provided)
-S,--Strict
Strict mode for import (fail fast - default mode is best-effort) (Optional)
-u,--username <username>
The BaseSpace Clarity LIMS REST API username (Required)
-w,--workflows <workflows>
The workflows to include in the custom manifest (Optional)