# Git-Sourced Pipelines ## Introduction **Git** can be used as source-control system for your pipelines. This offers portability, versioning and easy integration of existing public pipelines. To use Git-sourced pipelines: 1. If you want to use a private repository, Create [Git credentials ](#git-credentials)with your [personal access tokens](#personal-access-token). 2. [Configure in ICA](#creating-the-pipeline-in-ica) **where on GitHub the pipeline is located**, which **tag or commit-id** of that pipeline to use and select the **credentials** if it is a private repository or to prevent rate limiting. ## Using Private Repositories ### Git Credentials To access private pipelines stored on Git, you need to enter the Git credentials in ICA. This is done at **System Settings > Credentials > Create > Git Credential**. If you want to use a public repository, you do not need credentials or an access token. However, there is a limit to how many anonymous calls can be made to a GitHub repository, so if that limit is exceeded, you will encounter *repo lookup rate limit reached* and will not be able to import the pipeline. For this reason, **it is best practice to use Git credentials even for public repositories.** Fill out the following fields: * **Name**—Provide a name to easily identify your Git credentials * **Git url**—This is fixed to * **Personal Access Token** —the personal access token. Either an existing one, or you can [generate](#personal-access-token) one on Git. When you save and reopen your credentials, the personal access token is hidden. If you want to make changes to the credentials, you will need to re-enter the personal access token. This is to prevent the access token from being copied. * **Git username** — This field **appears after entering the personal access token** and is extracted from your token.
A link is provided to create your [personal access token](#personal-access-token) on [github.com](https://github.com/settings/tokens/new?description=Illumina%20Connected%20Analytics\&scopes=repo) in case you do not have a token. {% hint style="info" %} The link to the personal access token on GitHub.com will open a pop-up window, so if it does not open, you might have a pop-up blocker active. {% endhint %} You can **share the credentials** you own with other users of your tenant. To do so, select your credentials at **System Settings > Credentials** and choose **Share**. Those users will be able to use your credentials, but they will not be able to see the actual access token. ### Personal Access Token Personal access tokens work in the same way as OAuth access tokens. You can create a new personal access token at . * The personal access token must have **scope:repo** (meaning all elements in the repo category must be selected). * If you set an expiration date, all pipelines which use this personal access token will stop working on that date. For this reason, it is advised **not to set an expiration date unless absolutely necessary**. Select **Generate token** at the bottom of the screen to get your personal access token. **Copy the value over to a secure location** after generating it. ## Creating the pipeline in ICA Regardless of using a public or a private repository, you need to create the connection in ICA to pick up pipeline the from Git. 1. Create a new pipeline at **Projects > your\_project > flow > Pipelines > Create > Nextflow > from Git**. 2. Fill out the fields to access the pipeline. During configuration, a test will be performed to see if the repository can be reached with the provided credentials. *Even if the tests fail, you can still save the entered values, this is to prevent you from being blocked from creating a configuration if the repository is unavailable.*\ If you enter the URL for a private repository before you have entered the credentials, ICA will try to access it as if it were a public GitHub URL until you have entered the credentials.
Repository url (max 255 chars)The url where the pipeline is located.
For example https://github.com/nf-core/demo
Do not use a / as last character of your path as this will result in a failed pipeline with Remote resource not found as error.
Pipeline nameName of your pipeline. This is automatically filled out when the repository URL has been validated, but you can change the name.
Version numberThe version of your pipeline. This is not the same as the tag or commit-id, but used to identify the version of your pipeline to people using it. This field can hold 25 characters.
Git credentialOnly needed for private repositories. This is your previously configured Git credential. If you need Git credentials, but you have not configured them in the previous step, you can use the create button to configure them now.
Main file path
(max 255 chars)		For NextFlow, If the main.nf is not in the root folder, edit the main file path to the main.nf file, otherwise keep main.nf.
Config file path
(max 255 chars)		When configuring a nextflow pipeline, this is usually nextflow.config and located in the root folder. Edit this field to match your equivalent file and folder.
Schema file path
(max 255 chars)		When configuring a nextflow pipeline, this is the file which describes the different parameters used by the workflow. This is usually nextflow_schema.json.
VersionUse the long commit-id or the tag to identify the version. ICA will automatically convert the tag to fill out the commit-id.
3. Select **Create** to start importing the pipeline. During import, the **inputForm.json** file is generated from the **nextflow\_schema.json** file. * While the pipeline is being imported, you can still choose **Abort import** by opening the pipeline details if you decide not to import the pipeline. * If the **import fails**, you can investigate the reason by opening the pipeline and looking at the **import summary on the details tab**. * After import, the pipeline will be in **draft** [status](/connected-analytics/project/p-flow/f-pipelines.md#pipeline-statuses). 4. The details of your pipeline are now visible when opening the pipeline details tab. Here you can change the **name** and **version number**, provide a **description**, update the **Nextflow version** or edit other details as needed. 5. Proceed to the **InputForm files** **tab** to **edit the generated inputForm.json** file to match the inputs as needed by your pipeline. While you are editing the input form, you can use the **simulate** button to preview the input form. 6. Once all details are updated, **save your pipeline**. [Analyses](/connected-analytics/project/p-flow/f-analyses.md) with the pipeline can now be executed in the same way as with regular pipelines. {% hint style="info" %} To minimise the risk of triggering API-rate limiting on GitHub, the validations above are not performed when configuring pipelines via the API. {% endhint %} ## Example {% content-ref url="/pages/E9aascoyvWbjDrN3zyEf" %} [Basic Git-Sourced Pipeline Example](/connected-analytics/project/p-flow/f-pipelines/git-sourced-pipelines-experimental/basic-git-sourced-pipeline-example.md) {% endcontent-ref %} ## Restrictions The following restrictions apply to Git-sourced pipelines. * Only **JSON-based NextFlow** are supported, no CWL pipelines or XML-based pipelines. * Because translation from the generic nextflow pipeline to the ica-specific input forms takes place during import, **you cannot use input forms created in ICA in your GitHub repository** as the system will try to reconvert them. * **Only** [**Github.com**](http://github.com) is supported as repository location. * GitHub enterprise is not supported. * **Git-branches are not supported**, only tag or commit-id for version control. --- # Agent Instructions: Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://help.connected.illumina.com/connected-analytics/project/p-flow/f-pipelines/git-sourced-pipelines-experimental.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.