Features and Enhancements
Flow
Analysis logs (task stdout/stderr files) are now written to a folder named ‘ica_logs’ within the analysis output folder
Default scratch disk size attached to analysis steps reduced from 2TB to 0B to improve cost and performance of analyses. Pipelines created before ICA v2.21.0 will not be impacted
Notifications
Notifications can now be updated and deleted in externally managed Projects
API
Clarified on the Swagger page which sorting options apply to which paging strategy (cursor-based versus offset-based). Changed the default sorting behavior so that:
When no paging strategy is specified and no sort is requested, then cursor-based paging is default
When no paging strategy is specified and sort is requested, then offset-based paging is default
Cohorts
Procedure Search Box: Users can now access additional UI functionalities for Procedures
Users can now access Procedure codes from OMOP
Improved handling of drug codes across all reports, excluding Survival comparison
Ingestion
Users now have enhanced job warning log and API status improvements
Users now require download permissions to facilitate the data ingestion process
Fetch Molecular Files: Improved import – Users can now input a directory path and select sample files individually
Variant Type Summary: Users can now access a new variants tab that summarizes Variant type statistics per gene
Added sorting and filtering capabilities to report tables, such as variants observed in genes
Users can now view sample barcodes, replacing internal auto-increment sample IDs in the Structural Variants table within the Genes tab
“Search subjects” functionality improved with flexible filtering logic that now supports partial matches against a single input string
Fixed Issues
Data Management
Fixed an issue with data copy via the CLI where the file was being copied to a subfolder of the intended location instead of the specified folder
Resolved an issue where browser upload hangs intermittently when creating data
Fixed an issue where the delete popup does not always disappear when deleting data
Fixed an issue where GetFolder API call returns 404 error if the Create and Get operations are performed 100ms apart
Fixed an issue where file copy would fail if the file was located at the root level of User’s S3 storage bucket
Fixed an issue causing data linked from externally managed projects to be incorrectly excluded from the list project data API response
Fixed an issue where User cannot use data URNs to identify the destination folder when interacting with copy data API endpoints
Bundles: Fixed an issue where clicking the back button before saving a new bundle leads to inconsistencies
Flow
Fixed an issue where pipeline documentation is not scrollable when launching pipeline
Fixed an issue with logfiles of a task not being available for streaming while the task is still running
Fixed an issue where using the 're-run' button from the analysis page reverts the storage size selection to default
Fixed an inconsistency where the following two endpoints would show different analysis statuses:
GET /api/projects/{projectId}/analyses
GET /api/projects/{projectId}/analyses/{analysisId}
Improved performance issues with UI loading data records when selecting inputs for analysis
Fixed a caching issue which resulted in delays when running pipelines
Fixed an issue where back button for analysis or pipeline details does not always direct Users back to analysis or pipelines view, respectively
Fixed an issue where system performance is degraded when large batches (e.g., 1,000) of data are added as input to Analyses via the graphical UI. It is recommended to start Analyses with large numbers of input files via API
Base
Fixed an issue where enabling Base from a Base view other than Base Tables returned a warning message
Fixed an issue where Base access was not enabled when a bundle with tables is added to a project without Base (Base is automatically enabled so users can see the bundle's tables). However, access to the bundle's tables is revoked upon the deletion of Base, and was not granted again once Base was re-enabled
Fixed an issue where a Base job to load data into a table never finished because the file was deleted after the job started and before it finished. Now the job will end up in a Failed state
Cohorts
Fixed an issue where needle plot filtered out data points reappear when zooming in the exon when a filter is in place
Fixed an issue where users from a different tenant who accept a project share may encounter a failure at the final step of the data ingestion process
Fixed an issue where users can encounter intermittent errors when browsing and typing for a gene
Fixed an issue where the UI hangs on large genes and returns a 502 error
Fixed Issues
Data Management
Fixed an issue where multiple folder copy jobs with the same destination may get stuck In Progress
Fixed an intermittent issue where tags on the target folder for a batch data update call are not set, but are set for all child data
Flow
Fixed an issue causing intermittent pipeline failures due to an infrastructure error
Features and Enhancements
General
Navigation: If multiple regions are enabled for the same tenant, the region will be indicated in the waffle menu
Logging: Data transfers of BaseSpace Sequence Hub projects with data stored in ICA will be traced in ICA logs
Cohorts
Disease Search Box: Added support for specifying subjects by age of onset of disease(s)
Drug Search Box: Added a new query builder box for Drugs
Ingestion: Support for Drug, drug route, etc. attached to subjects
Cohorts building: Users can build cohorts by specifying drugs, drug route, etc.
Ingestion
Combine different variant types during ingestion (small variants, cnv, sv)
Cohorts supports Illumina Pisces variant caller for hg19 VCFs
Fixed Issues
General
Fixed an issue where the graphical UI hands with ha spinning wheel when saving or executing a command
Fixed an issue where rich text editor for Documentation tab on Pipelines, Tools, Projects and Bundles does not populate with correct styles in edit mode
Data Management
Fixed an issue where multiple clicks on create data in Project API endpoint resulted in multiple requests
Fixed an issue where the secondary data selection screen could not be resized
A spinning wheel icon with ‘copying’ status is displayed at the folder level in the target Project when a folder is being copied. This applies to the actual folder itself and not for folders higher up in the hierarchy
Fixed an issue where API to retrieve a project data update batch is failing with 500 error when either the Technical or the User tags are updated during the batch update request
Fixed an issue where linking jobs fail to complete if other linking jobs are running
Improved performance for data transfer to support BaseSpace Sequence Hub Run transfers
Fixed an issue causing some folder copy jobs to remain in "Partially Succeeded" status despite being completed successfullyBundles: Fixed an issue where the URL and Region where a Docker image is available is not displayed for a Docker image Tool shared via an entitled Bundle
Fixed an issue where the folder copy job was getting stuck copying large amounts of big files
Fixed an issue where the folder counts were not matching expected counts after Data linking
Fixed an issue where delete data popup would occasionally not disappear after deleting data.
Fixed an issue with data copy where referencing data from another region would not result in immediate failure
Fixed issue where uploading a folder using the CLI was not working
Fixed an issue where a Docker image shared via an entitled Bundle can be added to another region
Workflows
Fixed an issue where workflow does not fail if BCL Convert fails for a BCL Convert-only run
Flow
Improved performance when batches of data up to 1000 are added as input to an Analysis
Nextflow engine will return exit code 55 if the pipeline runner task is preempted
Fixed an issue where log files cannot be opened for any steps in an analysis while the analysis is in progress
Fixed an issue with concurrent updates on analysis
Fixed an issue where unknown data inputs in the XML of an analysis are not being ignored
The warning, close, and machine profile icons for Tools can now be seen in the graphical CWL pipeline editor
Fixed an issue where user cannot expand analysis output folder if user permissions change after starting analysis. Now, if a user has the correct permissions to start an analysis, that analysis should be able to finish correctly no matter the permissions at the time it succeeds
Base
Fixed an issue switching back from template to Empty Table did not clear the fields
Data linked from an externally managed project can be added to Base Tables
Fixed an issue in the graphical UI where schema definition does not scroll correctly when many columns are defined
Features and Enhancements
Data Management/API
Added a new endpoint available to change project owner
POST /api/projects/{projectId}:changeOwner { “newOwnerId”:”}
Added a new endpoint to copy data from one project to another:
/api/projects/{projectId}/projectDataCopyBatch
Data Management/CLI
Added the ability to copy files and folders between projects in the UI and CLI. This includes support for copying data from projects with ICA-managed storage (default) to projects with S3-configured storage.
Flow/API
When starting an analysis via the API, you can specify the input files based on HTTP(s). When your analysis is done, you will see the URL corresponding to the inputs in the UI, but you will not be able to start an analysis from the UI using this URL
Added two new endpoints for workflow sessions:
Get /api/projects/{projectId}/workflowSessions
Get /api/projects/{projectId}/workflowSessions/{workflowSessionId}/inputs
Added a new endpoint to retrieve configurations from a workflow session
Flow/CLI
Duplicate analyses submitted via the CLI will be avoided
Flow
Removed the ability to start analyses from data and sample views in the UI where a single input is selected to start analyses in bulk
Flow/Autolaunch ICA Workflow Session and Orchestrated Analyses (launched by the workflow session) now saves outputs in an organized folder structure: /ilmn-analysis/<name_used_to_create_sequencer_run_output_folder>
Base
The Base module has a new feature called ‘Data Catalogue’. This allows you to add usage data from your tenant/project if that data is available for you.
Data Catalogue views will be available and can be used in Base to query on
You will be able to preview and query Data Catalogue views through Base Tables and Query screens
The Data Catalogue will always be up to date with the available views for your tenant/project
Data Catalogue views cannot be shared through a Bundle
Data Catalogue views will also be available to team members that were added after the view was added
Data Catalogue views can be removed from the Base tables and corresponding project
By removing Base from a project, the Data Catalogue will also be removed from that project
Cohorts: Disease Search box
Cohorts now includes a disease search box to search for disease concepts. This replaces the disease concept tree explorer
Disease search box located under a Disease tab in main Query builder
Search box allows for a copy/paste action of codes to be processed as separate query elements. Currently, the feature is limited to a complete valid list
Each disease entered into the search box is displayed as a separate query item and can be set to include or exclude.
Diseases in search box can be used with boolean logic in cohort creation
Search box allows for an auto-complete of diagnosis concepts and identifiers
The disease filter is included in the cohort query summary on cohort page
Fixed Issues
Data Management
Data copy between ICA-managed projects and S3 storage configured projects is supported
Fixed an issue where storage configurations matching ICA-managed buckets would cause volume records to get associated with the wrong storage configuration in the system
API
The endpoint GET/api/projects/{ProjectID}/samples/{SampleID} correctly returns all the own samples and linked samples
Improved handling of bulk update via API when concurrent deletion of file has occurred
CLI
Fixed an issue where projectdata update tags would not update the tags
Fixed an issue to support adding the server-url as a parameter instead of having the config set
Flow
Fixed an issue resulting in failure to send a notification resulting in a failed workflow
Fixed an issue where one workflow session may override another when both are executed at the same time
Base
Fixed an issue where query download in JSON format returns an error
Added a message in the UI when a query takes longer than 30 seconds to inform the user that the query is ongoing and can be monitored in the Activity view
Added a section describing the Data Catalogue functionality
Bench
Fixed an issue where resizing the workspace to current size would prevent users from resizing for the next 6 hours
Cohorts
Fixed an issue where Gene Expression table does not display with TCGA data or for tenants with a hyphen (e.g., ‘genome-group’)
Fixed an issue where user had no way to delete a cohort comparison from a deleted cohort
Fixed an issue in the UI where multi-cohort needle plot tracks are overlapping
Fixed an issue causing failures during annotation step with ‘CNV’ data type when selection ‘GB=hg19’ and ‘CNV data’ for liftover; also observed with ‘SM data’ and ‘hg38’ without liftover (in APS1 and CAC1 regions) due to a ‘404 Not Found’ error.
Fixed Issue
Fixed an issue uploading folders via the CLI
Fixed Issue
Fixed an issue causing CWL pipelines using Docker images that do not contain bash shell executable to fail.
Fixed Issue
Fixed an issue leading to intermittent system instability.
Fixed Issue
Cohorts
Issue fixed where GTEx plot is not available for tenants with a hyphen (e.g. ilmn-demo).
Features and Enhancements
General
Versioning: The ICA version can now be found under your user when you select "About"
Versioning/API: It is possible to retrieve system information about ICA, such as the current version through GET/api/systeminfo
Logging: When an action is initiated by another application, such as BaseSpace Sequence Hub, it will be traced as well in the ICA logs
Data Management
New API endpoints are available for:
Creation of a data update in bulk: POST/api/projects/{projectId}/dataUpdateBatch
A list of data updates for a certain project: GET/api/projects/{projectId}/dataUpdateBatch/{batchId}
A list of items from the batch update: GET/api/projects/{projectId}/dataUpdateBatch/{batchId}/items
A specific item from the batch update: GET/api/projects/{projectId}/dataUpdateBatch/{batchId}/items/{itemId} Note: Batch updates include tags, format, date to be archived and date to be deleted
Data Management/API
The sequencing run information can be retrieved through its Id by using the API endpoint GET/api/sequencingRuns/{sequencingRunId}
Flow:
Auto launch now supports BCL Convert v3.10.9 pipeline and both TruSight Oncology 500 v2 pipelines (from FASTQs)
Removed "fpga-small" from available compute types. Pipelines using "fpga-small" will use the "fpga-medium"-equivalent compute specifications instead
Analyses launched/tracked by BaseSpace Sequence Hub contain relevant BaseSpace information in analysis details view
Flow/API
getPipelineParameters API returns parameter type in response
Added endpoints to retrieve and update a project pipeline definition
New API endpoint available to request the analyses in which a sample is being used
When leaving activationCodeDetailId empty when starting an analysis, the best match activation code will be used
Flow/API/CLI
Include "mountPaths" field in response for API and CLI command to retrieve analysis inputs
API
Two new API endpoints added to accept Terms and Conditions on a bundle:
GET /api/bundles/{bundleId}/termsOfUse/userAcceptance/currentUser Returns you the time of acceptance when you, the current user, accepted the Terms & Conditions.
POST /api/bundles/{bundleId}/termOfUse:accept
Add temporary credentials duration to API documentation
Notifications
List of events to which you can subscribe contains new ICA notification containing analyses updates
Bench
A new Bench permission is being introduced: Administrator. This permission allows users to manages existing workspaces and create new workspaces
The Bench Administrator role allows you to create new Bench workspaces with any permissions even if you as a Bench administrator do not have these permissions. In that case, you can create and modify the workspace, but you cannot enter that workspace. Modifying is only possible when the workspace is stopped
As a Bench Contributor you are not allowed anymore to delete a Bench Workspace, you need the Bench Administrator role.
Cohorts
Users can now ingest raw DRAGEN bulk RNAseq results for genes and transcripts (TPM), with the option to precompute differential expression during ingestion
Added support for running multiple DEseq2 analyses in the ingestion workflow through bulk processing based on sample size and specific requirements
In multiple needle plot view, individual needle plots can now be collapsed and expanded
Pop-outs for needle plot variants now contain additional links to external resources, such as UCSC
For a given cohort, display a distribution of raw expression values (TPM per gene) for a selected attributes
Use of the Cohorts maintains session between core ICA and Cohorts iFrame to prevent unwanted timeouts
Cohorts displays structural variants that include or overlap with a gene of interest
Fixed Issues
General
Collaboration: Fixed an issue where a user is presented with a blank screen when responding to a project invitation
Data Management/API
Improved error handling for API endpoint: DELETE/api/bundles/{bundleId}samples{sampleId}
Fixed an issue where the API endpoint GET /api/samples erroneously returned a 500
API endpoint GET/api/projects/{projectId}/analyses now returns the correct list when filtering on UserTags whereas it previously returned too many
Improved retry mechanism for API endpoint to create folderuploadsession
Data Management/CLI
When an upload of a folder/file is done through the CLI, it returns the information and ID of the folder/file
Data Management
CreatorId is now present on all data, including subfolders
Improved external linking to data inside ICA using deep linking
Improved error handling when creating folders with invalid characters.
Fixed an inconsistency for URN formats on output files from Analyses. This fix will apply only for analyses that are completed starting from ICAv2.18.0
Improved resilience in situations of concurrent linking and unlinking of files and folders from projects
It is only possible to delete a storage configuration if all projects that are using this storage configuration have been hidden and are not active projects anymore
Improved accuracy of the displayed project data size. Prior cost calculations were accurate, but the project data size visualization included technical background data
Fixed an issue where there is a discrepancy in number of configurations between Storage->Configurations and Configurations-> Genomics.Byob.Storage Configuration view
Flow/API
Improved error handling when invalid project-id is used in API endpoint GET /api/projects/{projectId}/pipelines
Fixed an issue when an Analysis completed with error "incomplete folder session", the outputs of the Analysis are not always completely listed in the data listing APIs
Updated ICA Swagger Project > createProject to correctly state that the analysis priority must be in uppercase
Flow
When a spot instance is configured, but revoked by AWS, the pipeline will fail and exit code 55 is returned
Fix to return meaningful error message when instrument run ID is missing from Run Completion event during an auto launched analysis
Improved parallel processing of the same analysis multiple times
Base
Improved error handling when creating queries which use two or more fields with the same name. The error message now reads "Query contains duplicate column names. Please use column alias in the query"
Fixed an issue where queries on tables with many entries fail with NullPointerException
Bench
Clarified that changes to Bench workspace size only take effect after a restart
Cohorts
Fixed issue where counts of subjects are hidden behind attribute names
Fixed issue where the state of checked files are not retained when selecting molecular files that are in multiple nested folders
Fixed issue where projects that contain files from linked bundles cause a time out, resulting in users not being able to select files for ingestion
Fixed an issue where the 'Import Jobs' page loaded within the Data Sets frame, depending on where the import was initiated
Fixed an issue in the Correlation plat where x-axis counts were hidden under attribute names
Fixed an issue where users were previously incorrectly signed out of their active sessions
Fixed Issues
Fixed an issue causing analyses requesting FPGA compute resources to experience long wait times (>24h) or not be scheduled
Features and Enhancements
Data Management
Performance improvements for data link and unlink operations – Larger and more complex folders can now be linked in the graphical UI, and progress can be monitored with a new visual indication under Activity > Batch Jobs
Notifications
Notifications are now available for batch job changes
Flow
Increased the allowed Docker image size from 10GB to >20GB
CWL: Added support for javascript expressions “ResourceRequirements” fields (i.e., type, size, tier, etc.) in CWL Pipeline definitions
Flow/API
Added support for using Pipeline APIs to query Pipelines included in Entitled Bundles (i.e., to retrieve input parameters)
Added support for providing S3 URLs as Pipeline data inputs when launching via the API (using storage credentials)
Added support for specifying multi-value input parameters in a Pipeline launch command
Bench
Project and Tenant Administrators are now allowed to stop running Workspaces
Cohorts
Enhanced ingestion workflow to ingest RNAseq raw data from DRAGEN output into backend Snowflake database
Added support for running multiple DEseq2 analyses in the ingestion workflow through bulk processing based on sample size and specific requirements
Multi-Cohort Marker Frequency - Added Multi-Cohort Marker Frequency tab allowing users to compare expression data across up to four Cohorts at the gene level
Multi-Cohort Marker Frequency includes a pairwise p-value heat map
Multi-Cohort Marker Frequency - Includes frequencies for Somatic and Copy Number Variants
Tab added for a multi-cohort marker frequency analysis in cohort comparisons
Multi-Cohort Needle Plot - Added new tab in the Comparison view with vertically aligned needle plots per cohort for a specified gene, allowing collapsible and expandable individual needle plots
Additional filter logic added to multi-cohort needle plot
Improved DRAGEN data type determination during ingestion allowing for multiple variant type ingestion
Enhanced list of observed variants with grouped phenotypes and individual counts, including a column for total sample count; tooltips/pop-outs provide extended information
Updates to needle plot link outs
Improved the Comparison feature by optimizing API calls to handle subjects with multiple attributes, ensuring successful loading of the page and enabling API invocation only when the user selects or expands a section
Removed unused columns (genotype, mrna_feature_id, allele1, allele2, ref_allele, start_pos, stop_pos, snp_id) from annotated_somatic_mutations table in backend database
Refactored shared functionality for picking consequence type to reduce code duplication in PheWAS-plot and GWAS-plot components
Invalid comparisons on the Comparisons page are now grayed out and disabled This improvement prevents the selection of invalid options
Automatic retry of import jobs when there are failures accessing data from ICA API
Fixed Issues
General
Navigation: Removed breadcrumb indication in the graphical UI
Data Management
The content of hidden Projects can now be displayed
Fixed the TimeModified timestamp on files
Bundles: Resolved issues when linking a large number of files within a folder to a Bundle
Flow
Single values are now passed as a list when starting an Analysis
Pipelines will succeed if the input and output formats specified on the pipeline level match at the Tool level
Fixed an issue causing Analysis failures due to intermittent AWS S3 network errors when downloading input data
CWL: Improved performance on output processing after a CWL Pipeline Analysis completes
Flow/UI: Mount path details for Analysis input files are now visible
Flow/UI: Improved usability when starting an Analysis by filtering entitlement options based o inputs selected and available entitlements
Flow/API
List of Analyses can now be retrieved via the API based on filters for UserReference and UserTags
Base
Fixed an issue where the Scheduler continues to retry uploading files which cannot be loaded
Bench
Resolved an issue when attempting to access Workspaces with multi-factor authentication (MFA) enabled at the Tenant-level
API
Improved error messaging for POST /api/projects/{projectId}/data/{dataId}:scheduleDownload
Cohorts
Fixed issue where Correlation bubble plot not showing for any projects intermittently
Fixed issue where importing Germline/hg19 test file did not load variants for a specific gene in the Needle plot due to missing entries in the Snowflake table
Fixed a bug causing an HTTP 400 error while loading the Cohort for the second time due to the UI passing "undefined" as variantGroup, which failed to convert to the VariantGroup Enum type
Fixed issue where scale (y-axis) of needle plot is changed even if value of sample count gnomAD frequency is not accepted
Fixed an issue where no data was generated in the Base Tables after a successful import job in Canada - Central Region (CAC1)
Fixed issue where long chart axis labels overlap with tick marks on graph
Features and Enhancements
General
Navigation: Updated URLs for Correlation Engine and Emedgene in the waffle menu
Authentication: Using POST /api/tokens:refresh for refreshing the JWT is not possible if it has been created using an API-key.
Authentication: Improved error handling when there is an issue reaching the authentication server
Authentication: Improved usability of "Create OAuth access token" screen
Data Management
You can now select 'CYTOBAND' as format after file upload
Added support for selecting the root folder (of the S3 bucket) for Projects with user-managed
Added support for creating an AWS Storage Configuration with an S3 bucket with Versioning enabled
Auto-launch
Added technical tags for upstream BaseSpace Run information to auto-launched analyses
Added support for multiple versions of BCL Convert for auto-launched analyses
Flow
Added support for '/' as separator in CWL ResourceRequirements when specifying Compute Type
Flow/API
The API to retrieve analysis steps now includes exit code for completed steps
Bench
Restricted Workspaces (Open or Restricted) always allow for access to Project Data within the Workspace
Restricted Bench workspaces have limited access through whitelisted URLs that are checked before entry
Restricted Bench Workspaces allow for Open or Restricted workspaces. Restricted workspaces do not have access to the internet except for user-entered whitelist URLs
Fixed Issues
Data Management
Upload for files names including spaces is now consistent for connector and browser upload. We do still advise not to use spaces in file names in general
Fixed search functionality in Activity > Data Transfers screen
Improved performance on opening samples
Fixed an issue where reference data in download tab initiates an unexpected download
Fixed intermittent issue where the Storage configuration within a Project can go into Error status and can block users from creating records such as folders and files
Service Connector: Improved error message for DELETE/api/connectors/{connectorId}/downloadRules/{downloadRuleId}
Data Management/API
Improved error handling for API endpoints: Delete/api/projects/ {projectId}/bundles/{bundleId} and POST/api/projects/{projectId}/bundles/{bundleId}
Improved error handling for POST/api/projects/{projectId}/base:ConnectionDetails
Bundles
Fixed an issue where the Table view in Bundles is not available when linking to a new Bundle version
Fixed an issue where linking/unlinking a Bundle with Base Tables could result in errors
Bundles/API
Improved error handling for DELETE/api/bundles/{bundleId}/tools/{toolId} and POST/api/bundles/{bundleId}/tools/{toolId}
Improved error message for POST/api/bundles/{bundleId}/samples/{sampleId}
Notifications/API
Custom subscriptions with empty filter expressions will not fail when retrieving them via the API
Improved error handling for POST/api/projects/{projectId}/notificationSubscriptions
Improved notification for Pipeline success events
Flow
When the input for a pipeline is too large, ICA will fail the Analysis and will not retry
Fixed issue where analysis list does not search-filter by ID correctly
Improved error handling when issues occur with provisioning resources
When retry succeeds in a Nextflow pipeline, exit code is now '0' instead of '143'
Flow/API
Fixed an issue causing API error when attempting to launch an Analysis with 50,000 input files
Improved pipeline error code for GET/api/projects/{projectId}/pipelines/{pipelineId} when already unlinked pipeline Id is used for API call
Fixed an issue where Analyses could not be retrieved via API when the Pipeline contained reference data and originated from a different tenant
Fixed filtering analyses on analysisId. Filtering happens via exact match, so part of the Id won't work
Bench/CLI
Fixed issue where the latest CLI version was not available in Bench workspace images
Cohorts
Fixed an issue where CNV data converted from hg19 to hg38 do not show up in Base table views
Fixed an issue accounting for multiple methods of referring to the alternate allele in a deletion from Nirvana data
Fixed intermittent issue where GWAS ingestions not working after Base enabled in a project.
Fixed Issue
Fixed an issue causing incorrect empty storage configuration dropdown during Project creation when using the “I want to manage my own storage” option for users with access to a single region
Features and Enhancements
General
General availability of sequencer integration for Illumina sequencing systems and analysis auto launch
General usability improvements in the graphical interface, including improved navigation structure and ability to switch between applications via the waffle menu in the header
Storage Bundle field will be auto-filled based on the Project location that is being chosen if multiple regions are available
Event Log entries will be paged in the UI and will contain a maximum of 1,000 entries. Exports are limited to the maximum number of entries displayed on the page.
Read-only temporary credentials will be returned when you are not allowed to modify the contents of a file
The ICA UI will only allow selection of storage bundles belonging to ICA during Project creation, and the API will only return storage bundles for ICA
Notifications
Creating Project notifications for BaseSpace externally managed projects is now supported
Flow
Allow attached storage for Pipeline steps to be set to 0 to disable provisioning attached storage and improve performance
Cohorts
GRCh37/hg19-aligned molecular data will get converted to GRCh38/hg38 coordinates to facilitate cross-project analyses and incorporating publicly available data sets.
API
Project list API now contains a parameter to filter on (a) specific workgroup(s)
Two new API endpoints are added to retrieve regular parameters from a pipeline within or without a Project context
Fixed Issues
General
Optimized price calculations resulting in less overhead and logging
Improved error handling:
during Project creation
of own storage Project creation failures.
to indicate connection issue with credential
for graphical CWL draft Pipelines being updated during an Analysis
Improved error messaging in cases where the AWS path contains (a) special character(s)
Fixed an issue causing errors when navigating via deep link to the Analysis Details view
Data Management
Fixed an issue causing data records to remain incorrectly in Unarchiving status when an unarchive operation is requested in the US and Germany regions
API
Fixed returning list of unlinked data in a sample that was linked before in GET/api/projects/{projectId}/data
Fixed error for getSampleCreationBatch when using status filter
CLI
Unarchive of folders is supported when archive or unarchive actions are not in progress for the folder
Improved error message to indicate connection issue with credentials
Flow
Fixed an issue causing incorrect naming of Analysis tasks generated from CWL Expression Tools
Fixed an issue when cloning Pipelines linked from Entitled Bundles to preserve the original Tenant as the Owning Tenant of the cloned Pipeline instead of the cloning user’s Tenant
Fixed an issue causing outputs from CWL Pipelines to not show in the Analysis Details despite being uploaded to the Project Data Analysis output folder when an output folder is empty
When a Contributor starts an Analysis, but is removed afterwards, the Analysis still runs as expected
Fixed an issue where Analyses fail where Nextflow is run a second time
Fixed an issue causing API error when attempting to launch an Analysis with up to 50,000 input files
Fixed an issue causing degraded performance in APIs to retrieve Analysis steps in Pipelines with many steps
Fixed an issue causing Analysis failure during output upload with error “use of closed network connection”
Fixed an issue causing disk capacity alter log to not show when an Analysis fails due to disk capacity and added error message
Fixed an issue preventing cross-tenant users from being able to open a shared CWL pipeline
Base
Improved target Table selection for schedulers to be limited to your own Tables
Bench
Fixed an issue causing Workspaces to hang in the Starting or Stopping statuses
Cohorts
Now handles large VCFs/gVCFs correctly by splitting them into smaller files for subsequent annotation by Nirvana
Features and Enhancements
General
Added a limit to Event Log and Audit UI screens to show 10,000 records
API
Parent output folder can be specified in URN format when launching a Workflow session via the API
Flow
Reduced Analysis delays when system is experiencing heavy load
Improved formatting of Pipeline error text shown in Analysis Details view
Users can now start Analyses from the Analysis Overview screen
Superfluous “Namespace check-0” step was removed to reduce Analysis failures
Number of input files for an Analysis is limited to 50,000
Auto launched Workflow sessions will fail if duplicate sample IDs are detected under Analysis Settings in the Sample Sheet
Base
Activity screen now contains the size of the query
Cohorts
Detect and Lift Genome Build: Cohorts documentation provides set-up instructions to convert hg19/GRCh37 VCFs to GRCh38 before import into Cohorts.
Attribute Queries: Improved the user experience choosing a range of values for numerical attributes when defining a cohort
Export Cohort to ICA Project Data: Improved the user experience exporting list of subjects that match cohort definition criteria to their ICA project for further analysis
Ingest Structural Variants into database
The Cohorts ingestion pipeline supports structural variant VCFs and will deposit all such variants into an ICA Base table if Base is enabled for the given project
Structural variants can be ingested and viewed in base tables
Needle Plot Enhancements
Users can input a numerical value in the Needle Plot legend to display variants with a specific gnomAD frequency percentage or sample count
The needle plot combines variants that are observed among subjects in the current project as well as shared and public projects into a single needle, using an additional shape to indicate these occurrences
Needle Plot legend color changes for Variant severity; pathogenic color coding is the same as the color coding in the visualization; differentiating hue between proteins and variants; and other color coding changes.
Needle plot tool tips that display additional information on variants and mutations are now larger and modal
The needle plot now allows to filter by gnomAD allele frequency and sample count in the selected cohort. Variants include links to view a list of all subjects carrying that variant and export that list.
Remove Samples Individually from Cohorts
Exclude individual subjects from a cohort and save the refined list
The subjects view allows users to exclude individual subjects from subsequent analyses and plots and save these changes Subject exclusions are reset when editing a cohort
Subject Selection in Analysis Visualization: Users can follow the link for subject counts in the needle plot to view a list of subjects carrying the selected variant or mutation.
UI/UX: Start and End time points are available as a date or age with a condition attribute in the subject data summary screen.
Fixed Issues
General
Improved resilience against misconfiguration of the team page when there is an issue with Workgroup availability
Removed ‘IGV (beta)’ button from ‘View’ drop down when selecting Project Data in UI
Data Management
Improved handling of multi-file upload when system is experiencing heavy loads
Fixed an issue to allow upload of zero-byte files via the UI
Fixed issue where other Bundles would not be visible after editing and saving your Bundle
API:
Improved error handling for API endpoint: POST /api/projects/{projectId}/analysisCreationBatch
Improved performance of API endpoint: getbestmatchingfornextflow
Flow
Fixed an issue causing Analysis output mapping to incorrectly use source path as target path
Fixed an issue where the UI may display incorrect or invalid parameters for DRAGEN workflows which do not accurately show the true parameters passed. Settings can be confirmed by looking at the DRAGEN analysis log files.
Base
“Allow jagged rows” setting in the Scheduler has been replaced with “Ignore unknown values” to handle files containing records with more fields than there are Table columns
Improved Base Activity view loading time
Fixed an error message when using the API to load data into a Base Table that has been deleted
Bench
Fixed an issue resulting in incorrect Bench compute pricing calculations
Fixed an issue preventing building Docker images from Workspaces in UK, Australia, and India regions
Fixed an issue where /tmp path is not writeable in a Workspace
Cohorts
Fixed issue where the bubble plot sometimes failed to display results even though the corresponding scatter plot showed data correctly.
The order of messages and warnings for ingestion jobs was not consistent between the UI and an error report sent out via e-mail.
The UI now displays any open cohort view tabs using shortened (“…”) names where appropriate
Issue fixed where ingestions with multiple errors caused halting to the ingestion queue.
The needle plot sometimes showed only one source for a given variant as opposed to all projects in which the variant had been observed in.
Issue fixed with unhandled genotype index format in annotation file to base database table conversion
Status updates via e-mail sometimes contained individual error messages or warnings without a text.
Fixed issue where items show in needle plot with incorrect numbering on the y-axis.
Fixed performance issue with subject count.
Widget bar-chart counts are intermittently cut off over four digits.
Fixed slowness when switching between tabs in query builder
Fixed Issue
Fixed issue with BaseSpace Free Trial and Professional users storing data in ICA
Fixed Issue
Fixed an issue resulting in analysis failures caused by a Kubernetes 404 timeout error
Features and Enhancements
General *
Each tenant supports a maximum of 30,000 Projects
.MAF files are now recognized as .TSV files instead of UNKNOWN
Added VCF.IDX as a recognized file format
General scalability optimizations and performance improvements
API
POST /api/projects/{projectId}/data:createDownloadUrls now supports a list of paths (in addition to a list of IDs)
Fixed Issues
General
Fixed an issue preventing the ‘Owning Project’ column from being used outside of Project
Fixed an issue allowing the region of a Project to be changed. Changing the region of a resource is not supported
Strengthened data separation and improved resilience against cross-Project metadata contamination
Bundles
After creating a new Bundle the user will be taken to the Bundle Overview page
Data Management
Fixed an issue which prevented changing the format of a file back to UNKNOWN
Fixed an issue causing inaccurate upload progress to be displayed for UI uploads. The Service Connector or CLI are recommended for large file uploads.
Fixed an issue showing an incorrect status for data linking batch jobs when data is deleted during the linking job
Service Connector: Fixed an issue allowing download of a Service Connector when no operating system is set
Service Connector: Cleaned up information available on Service Connectors by removing empty address information fields
API
Fixed date formatting for GET /api/eventLog (yyyy-MM-dd’T’HH:mm:ss.SSS’Z’)
Fixed an issue where the GET users API was not case sensitive on email address
Fixed an issue causing the metadata model to be returned twice in PSOT /api/projects/{projectId}/samples:search
Fixed the listProjects API 500 response when using the pageoffset query parameter
The searchProjectSamples API returns Sample metadata for Samples shared via a Bundle
Fixed an issue causing createProjectDataDownloadUrls API 400 and 502 errors when server is under load
Flow
Fixed analysis failures caused by kubernetes 404 timeout error
Fixed an issue where Workflwos would prematurely report completion of an Analysis
Improved Pipeline retry logic to reduce startup delays
Fixed an issue where Nextflow pipelines were created with empty files (Nextflow config is allowed to be empty)
Removed the 1,000 input file limitation when starting an Analysis
Improved the performance of status update messages for pipelines with many parallel steps
Fixed an issue with overlapping fields on the Analysis Details screen
Deactivated the Abort button for Succeeded analyses
Base
Fixed an issue where Pipeline metadata was not captured in the metadata Table generated by the metadata schedule
Error logging and notification enhancements
Bench
Fixed an issue where Workspaces could be started twice
Fixed an issue where the system checkpoint folder was incorrectly created in Project data when opening a file in a Workspace
Features and Enhancements
Analysis system infrastructure updates
Features and Enhancements
Added ability to refresh Batch Jobs updates without needing to leave the Details screen.
Projects will receive a job queuing priority which can be adjusted by an Administrator.
The text "Only showing the first 100 projects. Use the search criteria to find your projects or switch to Table view." when performing queries is now displayed both on the top and bottom of the page for more clarity.
API: Added a new endpoint to retrieve download URLs for data: POST/api/projects/{projectId}/data:createDownloadUrls
API: Added support for paging of the Project Data/getProjectDataChildren endpoint to handle large amounts of data.
API: Added anew endpoint to deprecate a bundle (POST /api/bundles/{bundleId}:deprecate)
API: If the API client provides request header "Accept-Encoding: gzip", then the API applies GZIP compression on the JSON response. This way the size of the response is significantly smaller which improves the download time of the response, resulting in faster end-to-end API calls. In case of compression the API also provides header "Content-Encoding: gzip" in the response, indicating that compression was effectively applied.
Flow: Optimized Analysis storage billing, resulting in reduced pipeline charges.
Flow: Internal details of a (non-graphical) pipeline marked ‘Proprietary’ will not be shared with users from a different tenant.
Flow: A new grid layout is used to display Logs for Analyses with more than 50 steps. The classic view is retained for analyses with 50 steps or less, though you can choose to also use the grid layout by means of a grid button on the top right on the Analysis Log tab.
CLI: Command to launch a CWL and Nextlfow Pipeline now contains the mount path as a parameter.
CLI: Version command now contains the build number.
CLI: Added support for providing the nextflow.config file when creating a Nextflow pipeline via CLI.
API: HTML documentation for aPipeline can now be returned with the following requests:
GET /api/pipelines/{pipelineId}/documentation/HTML
GET /api/projects/{projectId}/pipelines/{pipelineId}/documentation/HTML
API: Added a new endpoint for creating and starting multiple analyses in batch: POST /api/projects/{projectId}/analysisCreationBatch
Flow: Linking to individual Analyses and Workflow sessions is now supported by /ica/link/project//analysis/ and /ica/link/project//workflowSession/
Cohorts: Users can now export subject lists to the ICA Project Data as a file.
Cohorts: Users can query their ingested data through ICA Base. For users who already have ingested private data into ICA Cohorts, another ingestion will need to happen prior to seeing available database shares. Customers can contact support to have previously ingested data sets available in Base.
Cohorts: Correlation bubble plot counts now link to a subject/sample list.
Fixed Issues
Tooltip in the Project Team page provides information about the status of an invite
‘Resend invite’ button in the Project Team page will become available only when the invite is expired instead of from the moment the invite is being send out
Folders, subfolders and files all contain information about which user created the data
Files and folders with UTF-8 character are not supported. Please look at the documentation on how to recover from it in case you already have used them.
Improved performance for creating or hiding a Project in a tenant with many Projects
Service Connector: Updated information in the Service Connector screen to reflect the name change from "Type of Files" to the more accurate "Assign Format"
Service Connector: Folders within a Bundle can be downloaded via the Service Connector
Service Connector: Upload rules can only be modified in the Project where they apply
Service Connector: A message describes when a file is skipped during upload because it already exists in the Project
Service Connector: Fixed an issue where opening the Connectivity tab occasionally results in a null pointer error
Service Connector: Fixed an issue causing excessive logging when downloading files with long file paths
Service Connector: Fixed an issue where the Service Connector log may contain spurious errors which do not impact data transfers
Existing storage configurations are displayed and accessible via API and UI
Newly added storage configurations do no longer remain in ‘Initializing’ state
Fixed error when creating a storage configuration with more than 63 characters
Clicking on a Data folder in flat mode will now open the details of the folder
Only Tools in Released state can be added to a Bundle
Fixed issue preventing new Bundle versions to be created from Restricted Bundles
Deprecated Bundles are displayed upon request in card and table view
Bundles view limited to 100 Bundles
API: Fixed the API spec for ProjectDataTransfer.getDataTransfers
API: Fixed an issue with the projectData getChildren endpoint which returned incorrect object and pagination
API: Fixed an issue where multiple clicks on Create sample batch API endpoint resulted in multiple requests
API: POST /api/projects/{projectId}/data/{dataId}:scheduleDownload can now also perform folder downloads
API: Improved information on the Swagger page for GET /api/pipelines, GET/api/projects/{projectId}/pipelines, and GET/api/projects/{projectId}/pipelines/{pipelineId}
API: Fixed and issue when a user provides the same input multiple times to a multi-value input on an analysis run, that input is only passed to the pipeline once instead of multiple times: POST /api/projects/{projectId}/analysis:nextflow
CLI: Copying files in the CLI from a local directory on MacOS to your Project can result in both the desired file and the metadata file (beginning with ‘./’) being uploaded. The metadata file can safely be deleted from the Project
CLI: Hardened protection against accidental file overwriting
CLI: Improved handling for FUSE when connection to ICA is lost
CLI: icav2 projectdata mount –list shows updated list of mounted Projects
CLI: Paging improvements made for project list, projectanalyses list, and projectsdata list
CLI: When there is no config or session file the user will not be asked to create one for icav2 config reset and icav2 config get
CLI: Fixed an issue where Bundle data could not be seen through FUSE in Bench
CLI: Fixed an error message when missing config file upon entering the Project context
CLI: The unmount is possible without a path and will work via the stored Project ID or with a directory path resulting in an unmount of that path
CLI: Fixed an error when creating a Pipeline using URN for Project identifier
CLI: Attempting to delete a file from an externally-managed project returns an error indicating this not allowed
CLI: Fix to delete session file when config file is not detected
CLI: Paging option added to projectsamples list data
CLI: Fixed “Error finding children for data” error in CLI when downloading a folder
CLI: projectdata list now returns the correct page-size results
Flow: Fixed handling of special characters in CWL pipeline file names
Flow: Fixed an issue where task names exceeding 25 characters cause analysis failure in CWL pipelines
Flow: Fixed an issue which prevented requests for economy tier compute
Flow: Fixed an issue limiting CWL workflow concurrency to two running tasks
Flow: Fixed an issue where analysis file inputs specified in the input.json with ‘location’set to an external URL cause to CWL pipelines to fail
Flow: Fixed an issue resulting in out of sync Pipeline statuses
Flow: Improved Nextflow engine resiliency, including occurrences where Nextflow pipelines fail with ‘pod 404 not found’ error
Flow: Fix issue with intermittent system runtime failures incorrectly causing analysis failures
Flow: Fixed an issue where links to Analysis Details returned errors
Flow: Enabled scrolling for Pipeline documentation
Flow: Improved performance for handling analyses with large numbers of inputs
Flow: Improved handling of hanging Analyses
Flow: Improved error messages for failed Pipelines
Flow: Added documentation on how to use XML configuration files for CWL Pipelines
Flow: Duplicate values for multi-value parameters are no longer automatically removed
Flow: Correct exit code 0 is shown for successful Pipeline steps
Base: Fixed an issue so that only users with correct permissions are allowed to retrieve a list of Base tables
Base: Fixed an issue with metadata scheduler resulting in a null pointer
Base: An empty Table description will not return an error when requesting to list all Tables in a Project
Base: Jobs failed with an error containing 'has locked table' are not shown on the Base Job activity list. They can be displayed by selecting the 'Show transient failures' checkbox at Projects > Activity > Base Jobs.
Base: Users can see Schedulers and their results for the entire tenant if created by a tenant administrator in their project, but not create, edit or run them
Base: Fixed an issue preventing data format change in a schedule
Base: Fixed an issue preventing exporting data to Excel format
Bench: Improved handling to prevent multiple users in a single running Workspace
Bench: Fixed an issue causing Workspaces to be stuck in "Starting" state
Bench: Fixed an issue where usage does not showing up on usage CSV-based report
Bench: Fixed an issue where Bundle data could not be seen via the Fuse driver
Bench: Users can now consistently exit Workspaces with a single click on the ‘Back’ button.
Bench: After leaving a Workspace by clicking on the ‘Back’ button, the Workspace will remain in a ‘Running’ state and become available for a new user to access
Bench: Workspaces in a ‘Stuck’ state can be manually changed to ‘Error’ state, allowing users to restart or delete them
Cohorts: Fixed issue where file system cleanup not occurring after delete.
Cohorts: Fixed sign in and authentication issues in APN1 region.
Cohorts: Fixed issue where gene filter missing when editing a cohort and removing the edited filter and cancelling. The filter was preserved and should not have been.
Cohorts: Fixed issue where users see an application tile in the Illumina application dashboard selection screen called "Cohort Analysis Module".
Cohorts: Correlation: Fixed issue, Data type selections shows half when loading the search result
Cohorts: Fixed issue, Users will see an application tile on the Connected Platform home page screen called “Cohort Analysis Module” if the Cohorts module is added to the domain. Users should not enter the ICA Cohorts through this page. They should enter through ICA."