Analyses
An Analysis is the execution of a pipeline.
Starting Analyses
You can start an analysis from both the dedicated analysis screen or from the actual pipeline.
From Analyses
Navigate to Projects > Your_Project > Flow > Analyses.
Select » New Analysis.
Select a single Pipeline.
Configure the analysis settings.
Select » Start Analysis.
Refresh to see the analysis status. See lifecycle for more information on statuses.
If for some reason, you want to end the analysis before it can complete, select Manage > Abort analyses. Refresh to see the status update.
From Pipelines
Navigate to Projects > <Your_Project> > Flow > Pipelines
Select the pipeline to run.
Select » Start New Analysis.
Configure analysis settings.
Select » Start Analysis.
View the analysis status on the Analyses page. See lifecycle for more information on statuses.
If for some reason, you want to end the analysis before it can complete, select Manage > Abort analyses on the Analyses page.
Rerunning Analyses
Once an analysis has been executed, you can rerun it with the same settings or choose to modify the parameters when rerunning. Modifying the parameters is possible on a per-analysis basis. When selecting multiple analyses at once, they will be executed with the original parameters. Draft pipelines are subject to updates and thus can result in a different outcome when rerunning. ICA will display a warning message to inform you of this when you try to rerun an analysis based on a draft pipeline.
When there is an XML configuration change on a a pipline for which you want to rerun an analysis, ICA will display a warning and not fill out the parameters as it cannot guarantee their validity for the new XML.
Some restrictions apply when trying to rerun an analysis.
Analyses | Rerun | Rerun with modified parameters |
---|---|---|
Analyses using external data | Allowed | - |
Analyses using mount paths on input data | Allowed | - |
Analyses using user-provided input json | Allowed | - |
Analyses using advanced output mappings | - | - |
Analyses with draft pipeline | Warn | Warn |
Analyses with XML configuration change | Warn | Warn |
To rerun one or more analyses with te same settings:
Navigate to Projects > Your_Project > Flow > Analyses.
In the overview screen, select one or more anaylses.
Select Manage > Rerun analyses. The analyses will now be executed with the same parameters as their original run.
To rerun a single analysis with modified parameters:
Navigate to Projects > Your_Project > Flow > Analyses.
In the overview screen, open the details of the analysis you want to rerun by clicking on the analysis user reference.
Select Rerun. (at the top right)
Update the parameters you want to change.
Select Start Analysis The analysis will now be executed with the updated parameters.
Lifecycle
Status | Description | Final State |
---|---|---|
Requested | The request to start the Analysis is being processed | No |
Queued | Analysis has been queued | No |
Initializing | Initializing environment and performing validations for Analysis | No |
Preparing Inputs | Downloading inputs for Analysis | No |
In Progress | Analysis execution is in progress | No |
Generating outputs | Transferring the Analysis results | No |
Aborting | Analysis has been requested to be aborted | No |
Aborted | Analysis has been aborted | Yes |
Failed | Analysis has finished with error | Yes |
Succeeded | Analysis has finished with success | Yes |
When an analysis is started, the availability of resources may impact the start time of the pipeline or specific steps after execution has started. Analyses are subject to delay when the system is under high load and the availability of resources is limited.
Logs
During the execution of an analysis, logs are produced for each process involved in the analysis lifecyle. In the analysis details view, the Logs tab is used to view the logs in near real time as they're produced in the running processes. A grid layout is used for analyses with more than 50 steps, a tiled view for analyses with 50 steps or less, though you can choose to also use the grid layout for those by means of the tile/grid button on the top right of the analysis log tab.
There are system processes involved in the lifeycle for all analyses (ie. downloading inputs, uploading outputs, etc.) and there are processes which are pipeline-specific, such as processes which execute the pipeline steps. The below table describes the system processes.
Process | Description |
---|---|
Setup Environment | Validate analysis execution environment is prepared |
Run Monitor | Monitor resource usage for billing and reporting |
Prepare Input Data | Download and mount input data to the shared file system |
Pipeline Runner | Parent process to execute the pipeline definition |
Finalize Output Data | Upload Output Data |
Additional log entries will show for the processes which execute the steps defined in the pipeline.
Each process shows as a distinct entry in the Logs view with a Queue Date, Start Date, and End Date.
Timestamp | Description |
---|---|
Queue Date | The time when the process is submitted to the processes scheduler for execution |
Start Date | The time when the process has started exection |
End Date | The time when the process has stopped execution |
The time between the Start Date and the End Date is used to calculate the duration. The time of the duration is used to calculate the usage-based cost for the analysis. Because this is an active calculation, sorting on this field is not supported.
Each log entry in the Logs view contains a checkbox to view the stdout and stderr log files for the process. Clicking a checkbox adds the log as a tab to the log viewer where the log text is displayed and made available for download.
Log Files
In the analysis output folder, the ica_logs subfolder will contain the stdout and stderr files. If you delete these files, no log information will be available on the analysis details > logs tab.
Log Streaming
Logs can also be streamed using websocket client tooling. The API to retrieve analysis step details returns websocket URLs for each step to stream the logs from stdout/stderr during the step's execution. Upon completion, the websocket URL is no longer available.
Analysis Output Mappings
Currently, this feature is only availabe when launching analyses via API.
Currently, only FOLDER type output mappings are supported
By default, analysis outputs are directed to a new folder within the project where the analysis is launched. Analysis output mappings may be specified to redirect outputs to user-specified locations consisting of project and path. An output mapping consists of:
the source path on the local disk of the analysis execution environment, relative to the working directory
the data type, either FILE or FOLDER
the target project ID to direct outputs to; analysis launcher must have contributor access to the project
the target path relative to the root of the project data to write the outputs
Use the example analysis output mapping below for guidance.
If the output directory already exists, any existing contents with the same filenames as those output from the pipeline will be overwritten by the new analysis
In this example, 2 analysis output mappings are specified. The analysis writes data during execution in the working directory at paths out/test
and out/test2
. The data contained in these folders are directed to project with ID 4d350d0f-88d8-4640-886d-5b8a23de7d81
and at paths /output-testing-01/
and /output-testing-02/
respectively, relative to the root of the project data.
The following demonstrates the construction of the request body to start an analysis with the output mappings described above:
When the analysis completes the outputs can be seen in the ICA UI, within the folders designated in the payload JSON during pipeline launch (output-testing-01
and output-testing-02
).
Tags
You can add and remove tags from your analyses.
Navigate to Projects > Your_Project > Flow > Analyses.
Select the analyses whose tags you want to change.
Select Manage > Manage tags.
Edit the user tags, reference data tags (if applicable) and technical tags.
Select Save to confirm the changes.
Hyperlinking
If you want to share a link to an analysis, you can copy and paste the URL from your browser when you have the analysis open. The syntax of the analysis link will be <hostURL>/ica/link/project/<projectUUID>/analysis/<analysisUUID>
. Likewise, workflow sessions will use the syntax <hostURL>/ica/link/project/<projectUUID>/workflowSession/<workflowsessionUUID>
. To prevent third parties from accessing data via the link when it is shared or forwarded, ICA will verify the access rights of every users when they open the link.
Restrictions
Input for analysis is limited to a total of 50,000 files (including multiple copies of the same file). You can have up to 50 concurrent analyses running per tenant. Additional analyses will be queued and scheduled when currently running analyses complete and free up positions.
Troubleshooting
Analysis failure due to an external event such as spot termination or node draining, will result in exit code 55. Retry the analysis.
Last updated