CWL

Platform Core supports running pipelines defined using Common Workflow Language (CWL).

Compute Type

To specify a compute type for a CWL CommandLineTool, either define the ram and number of cores or use the resource type and size. The Platform Core compute type will automatically be determined based on CWL ResourceRequirement coresMin/coresMax (CPU) and ramMin/ramMax (Memory) values using a "best fit" strategy to meet the minimum specified requirements (See the Compute Types table to see to what the resources are mapped).

For example, take the following ResourceRequirements:

requirements:
    ResourceRequirement:
      ramMin: 10240
      coresMin: 6

This will result in a best fit of standard-large Platform Core compute type request for the task.

If the specified requirements can not be met by any of the presets, the task will be rejected and failed.

See the example below to use the ResourceRequirement in the cwl workflow with the Predefined Compute Types

requirements:
    ResourceRequirement:
        https://platform.illumina.com/rdf/ica/resources:type: fpga2
        https://platform.illumina.com/rdf/ica/resources:size: medium 
        https://platform.illumina.com/rdf/ica/resources:tier: standard

• FPGA requirements can not be set by means of CWL ResourceRequirements.
• The Machine Profile Resource in the graphical editor will override whatever is set for requirements in the ResourceRequirement.

Standard vs Economy

For each compute type, you can choose between the

• Standard - AWS on-demand (Default) or
• Economy - AWS spot instance tiers.

You can set economy mode with the "tier" parameter

requirements:
    ResourceRequirement:
        https://platform.illumina.com/rdf/ica/resources:type: himem
        https://platform.illumina.com/rdf/ica/resources:size: small 
        https://platform.illumina.com/rdf/ica/resources:tier: economy

Considerations

If no Docker image is specified, Ubuntu will be used as default. Both : and / can be used as separator.

CWL Overrides

Platform Core supports overriding workflow requirements at load time using Command Line Interface (CLI) with JSON input. Please refer to CWL documentation for more details on the CWL overrides feature.

In Platform Core you can provide the "override" recipes as a part of the input JSON. The following example uses CWL overrides to change the environment variable requirement at load time.

icav2 projectpipelines start cwl cli-tutorial --data-id fil.a725a68301ee4e6ad28908da12510c25 --input-json '{
  "ipFQ": {
    "class": "File",
    "path": "test.fastq"
  },
  "cwltool:overrides": {
  "tool-fqTOfa.cwl": {
    "requirements": {
      "EnvVarRequirement": {
        "envDef": {
          "MESSAGE": "override_value"
          }
        }                                       
       }
      }
    }
}' --type-input JSON --user-reference overrides-example