Basic Git-Sourced Pipeline Example

Demo Pipeline

This section describes how to configure the nf-core demo pipeline from https://github.com/nf-core/demo/ to run in Platform Core.

The nf-core framework for community-curated bioinformatics pipelines. Philip Ewels, Alexander Peltzer, Sven Fillinger, Harshil Patel, Johannes Alneberg, Andreas Wilm, Maxime Ulysse Garcia, Paolo Di Tommaso & Sven Nahnsen.

Nat Biotechnol. 2020 Feb 13. doi: 10.1038/s41587-020-0439-x.

Configuring the Pipeline

1. Create a new pipeline at Projects > your_project > flow > Pipelines > Create > Nextflow > from Git.
2. Fill out the following details:

Field	Value
Repository url	https://github.com/nf-core/demo (Tip: If you encounter Invalid GitHub repository url, you may have copied over a trailing space in the url)
Pipeline name	The pipeline name is automatically extracted from the repository. You can manually change this if needed, for example to prevent duplicate names.
Version number	Enter the version number. You can choose to use the version number from GitHub (1.1.0) or you can use your own version (for example 1 for being the first local version)
Storage size	As this demo pipeline uses very little resources, the smallest size we can select (3XSmall) will be sufficient.
Git credential	This is a public repository, so we don't actually need a credential and this can be left blank. However, there is a limit to how many anonymous calls can be made to a GitHub repository, so if that limit is exceeded, you will encounter repo lookup rate limit reached and will not be able to import the pipeline. To prevent running into this limit, select a Git credential. If you don't have one, click the create button next to the Git credential. Enter the value of your personal access token and Platform Core will automatically obtain the Git username for it. If you have no access token, you can create one with the Create personal access token on github.com button. The values there are already prefilled, but you can change the note at the top to easily identify for which purpose the token was generated.
Main file path	The main.nf file containing the pipeline logic and data flow is in the root folder, so we keep this as main.nf.
Config file path	nextflow.config containing the execution parameters is in the root folder, so we need to enter nextflow.config. The inputForm file will be generated based on the parameters in this file.
Schema file path	The schema is nextflow_schema.json, so enter this value. The inputForm file will be generated based on the parameters in this file.
Version	You can enter the commit id of the version you want to use or use the tag to identify the version. In this example, we use the tags to identify the version we want. From the screenshot below, you can see that there is a version 1.1.0 of the pipeline, so enter 1.1.0 in the tag field. When you enter this version in Platform Core, the commit-id for that tag will automatically be filled out. This commit-id is the long version, the short 7-character version is not supported.

Method 1 : finding and copying the full commit id

Method 2 : finding and copying the tag

Creating the Input File

As per instructions on the nf-core demo page, create a samplesheet.csv file on your local machine with as contents:

sample,fastq_1,fastq_2
SAMPLE1_PE,https://raw.githubusercontent.com/nf-core/test-datasets/viralrecon/illumina/amplicon/sample1_R1.fastq.gz,https://raw.githubusercontent.com/nf-core/test-datasets/viralrecon/illumina/amplicon/sample1_R2.fastq.gz
SAMPLE2_PE,https://raw.githubusercontent.com/nf-core/test-datasets/viralrecon/illumina/amplicon/sample2_R1.fastq.gz,https://raw.githubusercontent.com/nf-core/test-datasets/viralrecon/illumina/amplicon/sample2_R2.fastq.gz
SAMPLE3_SE,https://raw.githubusercontent.com/nf-core/test-datasets/viralrecon/illumina/amplicon/sample1_R1.fastq.gz,
SAMPLE3_SE,https://raw.githubusercontent.com/nf-core/test-datasets/viralrecon/illumina/amplicon/sample2_R1.fastq.gz,

Using the GUI

1. In Platform Core, navigate to Projects > your_project > Data and upload the created samplesheet.csv file to your project.

Using the CLI

If you have the CLI installed on your system, you can use the commands below to upload the samplesheet. If you do not have an active CLI, please follow these instructions first.

1. List your projects with the command icav2 projects list
2. From this list of projects, enter your demo project by using icav2 projects enter <your_project_uuid> with your_project_uuid replaced with the uuid of your project.
3. If you have created the samplesheet.csv file and put it into your CLI directory, you upload it to the root folder of your project with icav2 projectdata upload samplesheet.csv . If this name is already in use, you can rename the file during upload by using icav2 projectdata upload samplesheet.csv /samplesheet2.csv

Running the Analysis

Using the GUI

With the pipeline configurad and the inputfile created, you are ready to run your analysis.

1. Go to Projects > your_project > flow > Pipelines.
2. Select the created pipeline. (The default name will be nf-core/demo) and choose Start analysis at the top of the screen.
3. You will be presented with the form below.

   • Enter an identifier (user reference) for your pipeline
   • Select the samplesheet.csv file as input and start the analysis.

You can follow the status of your analysis at Projects > your_project > Flow > Analyses.

If your analysis fails with Module path must start with / or ./ prefix -- Offending module: plugin/nf-schema, then the Nextlow version is not set to the latest one. Edit it on your Projects > your_project > flow > Pipelines > your_pipeline > Details

Using the CLI

If you have the CLI installed on your system, you can also use the commands below to work with pipelines. If you do not have an active CLI, please follow these instructions first.

1. List your projects with the command icav2 projects list to retrieve their project uuid.
2. From this list of projects, enter your demo project by using icav2 projects enter <your_project_uuid> with <your_project_uuid> replaced with the uuid of your project.
3. To list the pipelines in your project and see their pipeline uuid, use icav2 projectpipelines list
4. To retrieve the file uuid, use the command icav2 projectdata list --file-name samplesheet.csv . If you used a different name for your samplesheet file, use the name you gave it instead of samplesheet.csv.

You can search for csv files in your project with the command icav2 projectdata list --file-name csv --match-mode fuzzy.

1. To run the pipeline with the input file,

• Select the pipeline (replace <your_pipeline_uuid> with the uuid of your pipeline).
• Set the storage size (3XSmall). If this storage is not available in your subscription, you can use icav2 analysisstorages list to get a list of available storage sizes.
• Set the user reference to give your analysis a name. In this example we use MyDemoGitPipeline as name.
• Point to the input file (replace <your_file_uuid> with the actual file uuid).

  If you want to see a list of input parameters of your pipeline, use icav2 projectpipelines input <your_pipeline_uuid> this will show you the code of the input parameters. In our example, this is "input"

The resulting command to run the pipeline is then:

icav2 projectpipelines start nextflowjson <your_pipeline_uuid> --storage-size 3XSmall --user-reference MyDemoGitPipeline --field-data "input":"<your_file_uuid>"

Optional : Editing the Input Form

Even though the import function will have created an input form based on the configured files, you can customise this form to make it easier to use by removing or defaulting parameters.

1. Go to Projects > your_project > flow > Pipelines > your_pipeline > Edit
2. Navigate to the Inputform files tab. This will open the inputForm.json file.
3. Replace the contents with this minimalist input file

{
  "fields": [
    {
      "id": "input",
      "type": "data",
      "label": "input",
      "helpText": "Select your input file",
      },
      "maxValues": 1,
      "minValues": 1
    },
    {
      "id": "outdir",
      "type": "textbox",
      "label": "outdir",
      "helpText": "The output directory where the results will be saved.",
      "hidden": true,
      "value": "out",
      "minValues": 1
    }
  ]
}

This will result in a minimal input form which only needs the input file selection.