Analyses

An Analysis is the execution of a pipeline.

Starting Analyses

You can start an analysis from both the dedicated analysis screen or from the actual pipeline.

From Analyses

1. Navigate to Projects > your_project > Flow > Analyses.
2. Select Start.
3. Select a single Pipeline.
4. Configure the analysis settings (see below).
5. Select Start Analysis.
6. Refresh to see the analysis status. See lifecycle for more information on statuses.
7. If for some reason, you want to end the analysis before it can complete, select Projects > Your_Project > Flow > Analyses > Manage > Abort. Refresh to see the status update.
8. To perform a completed analysis again, select Rerun.

From Pipelines or Pipeline details

1. Navigate to Projects > your_project > Flow > Pipelines
2. Select the pipeline you want to run or open the pipeline details of the pipeline which you want to run.
3. Select Start Analysis.
4. Configure analysis settings (see below).
5. Select Start Analysis.
6. View the analysis status on the Analyses page. See lifecycle for more information on statuses.
7. If for some reason, you want to end the analysis before it can complete, select Manage > Abort on the Analyses page.

Analysis Settings

The Start Analysis screen provides the configuration options for the analysis.

Field	Entry
User Reference	The unique analysis name.
Pipeline	This is not editable, but provides a link to the pipeline so you want to look up details of the pipeline.
User tags (optional)	One or more tags used to filter the analysis list. Select from existing tags or type a new tag name in the field.
Notification (optional)	Enter your email address if you want to be notified when the analysis completes.
Output Folder1	Select a folder in which the output folder of the analysis should be located. When no folder is selected, the output folder will be located in the root of the project. When you open the folder selection dialog, you have the option to create a new folder (bottom of the screen). You can create nested folders by using the folder/subfolder syntax. Do not use a / before the first folder or after the last subfolder in the folder creation dialog.
Logs Folder	Select a folder where the analysis logs will be stored. When no logs folder is selected, they will be stored as subfolder in the output folder. When you select a logs folder different from your output folder, the folders will be separated. When you open the folder selection dialog, you can create a new folder (bottom of the screen). Create nested folders by using the folder/subfolder syntax. Do not use a / before the first folder or after the last subfolder in the folder creation dialog. When choosing a folder containing data, files with the same name will be overwritten.
Input	Select the input files to use in the analysis. (max. 50,000)
Settings (optional)	Provide input settings.
Resources	Select the storage size for your analysis. The available storage sizes depend on your selected Pricing subscription. See Storage for more information.

¹ When using the API, you can redirect analysis outputs to be outside of the current project.

Aborting Analyses

You can abort a running analysis from either the analysis overview (Projects > your_project > Flow > Analyses > your_analysis > Manage > Abort) or from the analysis details (Projects > your_project > Flow > Analyses > your_analysis > Details tab > Abort).

Viewing Analyses Results

You can view analysis results on the Analyses page or in the output folder on the Data page.

1. Select the Projects > your_project > Flow > Analyses page.
2. Select the desired analysis.
3. From the output files tab, expand the list if needed and select an output file.

   • If you want to add or remove any user or technical tags, you can do so from the data details view.
   • If you want to download the file, select Download.
   • To see the data in the data view where you can easily navigate between files, select Open in data.
4. To preview the file, select the View tab.

To see more details of your analyses, use Projects > your_project > Flow > Analyses > your_analysis. The following tabs will be visible (depending on pipeline):

• Details - View information on the pipeline configuration.
• Output files - View the output of the Analysis.
• Steps - stderr and stdout information.
• CWL - The CWL pipeline definition.
• Nextflow timeline - Nextflow process execution timeline.
• Nextflow execution - Nextflow analysis report. Showing the run times, commands, resource usage and tasks for Nextflow analyses.
• Report - Shows the reports defined on the pipeline report tab.

Other tabs can be available depending on your chosen pipeline.

Rerunning Analyses

Once an analysis has been executed, you can rerun it with the same settings or choose to modify the parameters when rerunning. Modifying the parameters is possible on a per-analysis basis. When selecting multiple analyses at once, they will be executed with the original parameters. Draft pipelines are subject to updates and thus can result in a different outcome when rerunning. Platform Core will display a warning message to inform you of this when you try to rerun an analysis based on a draft pipeline.

When rerunning multiple analysis at the same time (multiselect), the user reference can not be chosen and will be the original user reference (up to 231 characters), followed by _rerun_yyyy-MM-dd_HHmmss.

When there is an XML configuration change on a a pipeline for which you want to rerun an analysis, Platform Core will display a warning and not fill out the parameters as it cannot guarantee their validity for the new XML. You will need to provide the input data and settings again to rerun the analysis.

Some restrictions apply when trying to rerun an analysis.

Analyses	Rerun	Rerun with modified parameters
Analyses using external data	Allowed	-
Analyses using mount paths on input data	Allowed	-
Analyses using user-provided input json	Allowed	-
Analyses using advanced output mappings	-	-
Analyses with draft pipeline	Warn	Warn
Analyses with XML configuration change	Warn	Warn

To rerun one or more analyses with te same settings:

1. Navigate to Projects > Your_Project > Flow > Analyses.
2. In the overview screen, select one or more analyses.
3. Select Manage > Rerun. The analyses will now be executed with the same parameters as their original run.

To rerun a single analysis with modified parameters:

1. Navigate to Projects > Your_Project > Flow > Analyses.
2. In the overview screen, open the details of the analysis you want to rerun by clicking on the analysis user reference.
3. Select Rerun. (at the top right)
4. Update the parameters you want to change.
5. Select Start Analysis The analysis will now be executed with the updated parameters.

You might see parameters (files and folders) on the analysis details tab which are not defined on the XML. This indicates they are added in the xml-pipeline analysis creation endpoint.

Lifecycle

Status	Description	Final State
Requested	The request to start the Analysis is being processed	No
Queued	Analysis has been queued	No
Initializing	Initializing environment and performing validations for Analysis	No
Preparing Inputs	Downloading inputs for Analysis	No
In Progress	Analysis execution is in progress	No
Generating outputs	Transferring the Analysis results	No
Aborting	Analysis has been requested to be aborted	No
Aborted	Analysis has been aborted	Yes
Failed	Analysis has finished with error	Yes
Succeeded	Analysis has finished with success	Yes

During analysis start, Platform Core runs a verification on the input files to see if they are available. When it encounters files that have not completed their upload or transfer, it will report "Data found for parameter [parameter_name], but status is Partial instead of Available". Wait for the file to be available and restart the analysis.

When an analysis is started, the availability of resources may impact the start time of the pipeline or specific steps after execution has started. Analyses are subject to delay when cloud resources are under high load. When the underlying storage provider runs out of storage resources, the Status field of the Analysis details will indicate this. There is no need to abort or rerun the analysis.

Analysis steps logs

During the execution of an analysis, logs are produced for each process involved in the analysis lifecyle. In the analysis details view, the Steps tab is used to view the steps in near real time as they're produced in the running processes. A grid layout is used for analyses with more than 50 steps, a tiled view for analyses with 50 steps or less, though you can choose to also use the grid layout for those by means of the tile/grid button on the top right of the analysis log tab. The steps tab also shows which resources were used as compute type in the different main analysis steps. (For child steps, these are displayed on the parent step)

There are system processes involved in the lifeycle for all analyses (ie. downloading inputs, uploading outputs, etc.) and there are processes which are pipeline-specific, such as processes which execute the pipeline steps. The below table describes the system processes. You can choose to display or hide these system processes with the Show technical steps

Process	Description
Setup Environment	Validate analysis execution environment is prepared
Run Monitor	Monitor resource usage for billing and reporting
Prepare Input Data	Download and mount input data to the shared file system
Pipeline Runner	Parent process to execute the pipeline definition
Finalize Output Data	Upload Output Data

Additional log entries will show for the processes which execute the steps defined in the pipeline.

Each process shows as a distinct entry in the steps view with a Queue Date, Start Date, and End Date.

Timestamp	Description
Queue Date	The time when the process is submitted to the processes scheduler for execution
Start Date	The time when the process has started exection
End Date	The time when the process has stopped execution

The time between the Start Date and the End Date is used to calculate the duration. The time of the duration is used to calculate the usage-based cost for the analysis. Because this is an active calculation, sorting on this field is not supported.

Each log entry in the Steps view contains a checkbox to view the stdout and stderr log files for the process. Clicking a checkbox adds the log as a tab to the log viewer where the log text is displayed and made available for download.

Analysis Cost

To see the price of an analysis in BioInsight Credits (BIC), look at Projects > your_project > Flow > Analyses > your_analysis > Details tab. The pricing section will show you the entitlement bundle, storage detail and price in BIC once the analysis has succeeded, failed or been aborted.

Log Files

By default, the stdout and stderr files are located in the ica_logs subfolder within the analysis. This location can be changed by selecting a different logs folder in the current project at the start of the analysis. Do not use a folder which already contains log files as these will be overwritten. To set the log file location, you can also use the CreateAnalysisLogs section of the Create Analysis endpoints.

If you delete these files, no log information will be available on the analysis details > Steps tab.

You can access the log files from the analysis details (projects > your_project > flow > analysis > your_analysis > details tab)

Log Streaming

Logs can also be streamed using websocket client tooling. The API to retrieve analysis step details returns websocket URLs for each step to stream the logs from stdout/stderr during the step's execution. Upon completion, the websocket URL is no longer available.

Analysis Output Mappings

Currently, only FOLDER type output mappings are supported

By default, analysis outputs are directed to a new folder within the project where the analysis is launched. Analysis output mappings may be specified to redirect outputs to user-specified locations consisting of project and path. An output mapping consists of:

• the source path on the local disk of the analysis execution environment, relative to the working folder.
• the data type, either FILE or FOLDER
• the target project ID to direct outputs to; analysis launcher must have contributor access to the project.
• the target path relative to the root of the project data to write the outputs.

If the output folder already exists, any existing contents with the same filenames as those output from the pipeline will be overwritten by the new analysis

Example

In this example, 2 analysis output mappings are specified. The analysis writes data during execution in the working directory at paths out/test and out/test2. The data contained in these folders are directed to project with ID 4d350d0f-88d8-4640-886d-5b8a23de7d81 and at paths /output-testing-01/ and /output-testing-02/ respectively, relative to the root of the project data.

The following demonstrates the construction of the request body to start an analysis with the output mappings described above:

```json
{
...
    "analysisOutput":
    [
        {
            "sourcePath": "out/test1",
            "type": "FOLDER",
            "targetProjectId": "4d350d0f-88d8-4640-886d-5b8a23de7d81",
            "targetPath": "/output-testing-01/"
        },
        {
            "sourcePath": "out/test2",
            "type": "FOLDER",
            "targetProjectId": "4d350d0f-88d8-4640-886d-5b8a23de7d81",
            "targetPath": "/output-testing-02/"
        }
    ]
}
```

When the analysis completes the outputs can be seen in the Platform Core UI, within the folders designated in the payload JSON during pipeline launch (output-testing-01 and output-testing-02).

You can jump from the Analysis Details to the individual files and folders by opening the output files tab on the detail view (Projects > your_project > Flow > Analyses > your_analysis > Output files tab > your_output_file) and selecting Open in data.

The Output files section of the analyses will always show the generated outputs, even when they have since been deleted from storage. This is done so you can always see which files were generated during the analysis. In this case it will no longer be possible to navigate to the actual output files.

analysis output	logs output	Notes
Default	Default	Logs are a subfolder of the analysis output.
Mapped	Default	Logs are a subfolder of the analysis output.
Default	Mapped	Outputs and logs may be separated.
Mapped	Mapped	Outputs and logs may be separated.

Tags

You can add and remove tags from your analyses.

1. Navigate to Projects > Your_Project > Flow > Analyses.
2. Select the analyses whose tags you want to change.
3. Select Manage > Manage tags.
4. Edit the user tags, reference data tags (if applicable) and technical tags.
5. Select Save to confirm the changes.

Both system tags and customs tags exist. User tags are custom tags which you set to help identify and process information while technical tags are set by the system for processing. Both run-in and run-out tags are set on data to identify which analyses use the data. Connector tags determine data entry methods and reference data tags identify where data is used as reference data.

Hyperlinking

If you want to share a link to an analysis, you can copy and paste the URL from your browser when you have the analysis open. The syntax of the analysis link will be <hostURL>/ica/link/project/<projectUUID>/analysis/<analysisUUID>. Likewise, workflow sessions will use the syntax <hostURL>/ica/link/project/<projectUUID>/workflowSession/<workflowsessionUUID>. To prevent third parties from accessing data via the link when it is shared or forwarded, Platform Core will verify the access rights of every user when they open the link.

Restrictions

Input for analysis is limited to a total of 50,000 files (including multiple copies of the same file). Concurrency limits on analyses prevent resource hogging which could result in resource starvation for other tenants. Additional analyses will be queued and scheduled when currently running analyses complete and free up positions. The theoretical limit is 20, but this can be less in practice, depending on a number of external factors.

Troubleshooting

When your analysis fails, open the analysis details view (Projects > your_project> Flow > Analyses > your_analysis) and select display failed steps. This will give you the steps view filtered on those steps that had non-0 exit codes. If there is only one failed step which has logfiles, the stderr of that step will be displayed.

For pipeline developers: add automatic retrying to the individual steps that fail with error 55 / 56 (provided these steps are idempotent) See tips and tricks for retries.

• Exit code 55 indicates analysis failure on economy instances due to an external event such as spot termination. You can retry the analysis.
• Exit code 56 indicates analysis failure due to pod disruption and deletion by Kubernetes' Pod Garbage Collector (PodGC) because the node it was running on no longer exists. You can retry the anlaysis.