Features and Enhancements
General
The projectdata upload CLI command will from now on give you the credentials to access the data
System notifications which could already previously be found on https://status.illumina.com/ , both regional and global, are now shown in the ICA UI when an important ICA message needs to be communicated
Data Management
Introduced a limit to the number of data elements that can be put in POST /api/projects/{projectId}/dataUpdateBatch to 100.000 entries
Flow
Users can now access json-based pipeline input forms for both Nextflow and CWL pipelines. API access is not yet available for CWL pipelines
Added GPU compute types (gpu-small, gpu-medium) for use in workflows
Users can now sort analyses by request date instead of start date, which was not always available
The analysis details page has been upgraded with the following features:
The progress bar which could be found on the analyses overview page will now also appear in the details page
A maximum of 5 rows of output are shown for each output parameter, but the output can be displayed in a large popup to have a better overview
Orchestrated analyses are shown in a separate tab
Cohorts
Users can now use the Measurement concept API to create cohorts based on lab measurement data and harmonize their values to perform downstream analysis
Users can now access the Hierarchical concept search API to view the phenotype ontologies
Fixed Issues
General
The mail option is now automatically filtered out for those events that do not support it
Fixed an issue where there was no email sent after rerunning a workflow session
Fixed an issue which caused authentication failures when using a direct link
Made file and folder selection more consistent
Fixed an issue with the CLI where using the “projectsamples get” command to retrieve a sample shared via an entitled bundle in another tenant failed
Fixed filtering so you can only see subscriptions and channels from your own tenant
Improved GUI handling for smaller display sizes
Fixed the workflow session user reference and output folder naming to use BaseSpace Experiment Name when available
Data Management
The unlink action is now greyed out if data not linked is in the selection
Fixed an issue that when deleting folders, the parent folders were deleted first, giving the impression that the parent folder is deleted but not the subfolders and files
Fixed an issue where the connector downloads only downloaded the main folder, not the folder contents
For consistency, it is no longer possible to link to folders or files from within subfolders. Previously, you could link, but the files and folders are always linked to top level instead of the subfolder from where linking was done
Updated error handling for dataUpdateBatch API endpoint
Moving small files (>8Mb) will not trigger a "moving" event, only that the move has completed as out-of-order events caused issues and moving small files happens fast enough to not need the status of being moved, only the completion of the move
Improved error handling when encountering issues during cancellation of data copy/move
Improved error message when trying to unlink data from a project via the API when this data is native to that project and not linked
Fixed issue where analysis can proceed to download input data when any of the inputs are in status other than AVAILABLE, including records within folder data inputs
Flow
Redesigned UI component to prevent issues with Analysis summary display
Fixed an issue where the field content was not set to empty when the field input forms have changed between the original analysis and a rerun
Replaced retry exhaustion message, "External Service failed to process after max retries 503 Unique Reference ID: 1234" with a more useful message to end users that advises them to contact Illumina support: "Attempt to launch ICA workflow session failed after max retries. Please contact Illumina Tech Support for assistance. Unique Reference ID: 1234". This does not replace more specific error messages that provide corrective advice to the user, such as "projectId cannot be null"
For efficiency reasons, pipeline files are limited to a file size of 100 Megabytes
Bench
Fixed an issue which caused .bash_profile to no longer source .bashrc
Fixed the status of deleted docker images which previously were displayed as available
After creating a tool, the Information tab and Create Tool are now no longer accessible to prevent erroneous selection
Cohorts
Fixed layout issue where buttons were moved up when the user selected the option
Fixed issue where user was not able to view PheWas plot when multiple cohorts are open and same gene is searched
Fixed issue where the user was not able to view GWAS plot when multiple cohorts are open and user switched forth and back between cohorts
Fixed issue where users were not able to see the cryogenic map in the gene summary page for gene associated with the chromosome
Fixed Issues
General
Fixed an issue where various Data Transfer Ownership API calls were failing with a 'countryView' constraint violation error
Features and Enhancements
General
Dynamically linked folders and files now have their own icon type, which is a folder/file symbol with a link symbol consisting of three connected circles
Data Management
With the move from static to dynamic data linking, unlinking data is now only possible from the project top level to prevent inconsistencies
The user can now manually, dynamically link a folder
The icav2 project data mount command now supports the “--allow-other” option to allow other users access to the data
The user can now set a time to be archived or deleted for files and folders with the “timeToBeArchived” and “timeToBeDeleted” parameter on the “POST/api/projects/{projectId}/dataUpdateBatch” command
Added 4 new API endpoints which combine the create and retrieve upload information
POST/api/projects/{projectId}/data:createFileWithTemporaryCredentials Creates a file in this project, and retrieves temporary credentials for it
POST/api/projects/{projectId}/data:createFileWithUploadUrl Creates a file in this project, and retrieves an upload url for it
POST/api/projects/{projectId}/data:createFolderWithTemporaryCredentials Creates a folder in this project, and and retrieves temporary credentials for it
POST/api/projects/{projectId}/data:createFolderWithUploadSession Creates a folder in this project, and creates a trackable folder upload session
Flow
The default Nextflow version is now 22.04.03, from 20.10.0
The user can now specify the Nextflow version when deploying a pipeline via the CLI with the “--nextflow-version” flag
Bench
The user now has the option to choose either a tool or bench as a docker image when adding new docker images
It is now possible to open contents of a Bench workspace in a new tab from the Bench details tab > access section
Fixed Issues
General
Improved handling of API calls with an invalid or expired JWT or API token
Data Management
Renamed the "New storage credential" button to "Create storage credential"
Removed the "Edit storage credential" button. The user can now edit the column directly in the open dialog when clicking on the name
Performance improvements to scheduled data download
Fixed an issue where data records were shown more than once when updating the tags\
The data details were erroneously labeled with "size in bytes" while the size was in a variable unit
Fixed an issue where trying to download files could result in the error "Href must not be null" when the file was not available
Fixed an issue where existing data catalog views would return an empty screen caused by a mismatch in role naming
Flow
Fixed an issue that caused opening a pipeline in the read-only view to incorrectly detect there were unsaved changes to the pipeline
Fixed an issue when having different pipeline bundles with the same (name) resource models would result in duplicate listing of these resources
Improved error handling when encountering output folder creation failure, which previously could result in analysis being stuck in REQUESTED status
By default Nextflow will no longer generate the trace report. If you want to enable generating the report, add the section below to your userNextflow.config file:
trace.enabled = true
trace.file = '.ica/user/trace-report.txt'
trace.fields = 'task_id,hash,native_id,process,tag,name,status,exit,module,container,cpus,time,disk,memory,attempt,submit,start,complete,duration,realtime,queue,%cpu,%mem,rss,vmem,peak_rss,peak_vmem,rchar,wchar,syscr,syscw,read_bytes,write_bytes,vol_ctxt,inv_ctxt,env,workdir,script,scratch,error_action'
Fixed the issue where users not allowed to run or rerun workflows could start them from the API or BaseSpace SequenceHub. Now, users that cannot start workflows cannot rerun them.
Cohorts
Fixed issue where the users can select the lower needles despite the overlap of multiple needles at the same location in the needle plot
Fixed issue where the user would not be able to view the cryogenic map in the gene summary page for gene associated with the chromosome
Fixed issue where the user would not be able to view the PheWas plot when multiple cohorts are open and same gene is searched
Features and Enhancements
Data Management
Improved performance of data linking jobs
Fixed Issues
General
Fixed an issue causing slow API responses and 500 errors
Features and Enhancements
General
The CLI readme file will now additionally contain the CLI build number
Data Management
Fixed an issue where there was a discrepancy between the Run Input tags shown to the user and what was stored on the data
Added a 25,000-item limit to the v3 endpoint for batch data linking. Using the v4 endpoint, which does not have this limitation, is recommended
Flow
Analyses and workflow sessions can now be resubmitted, and parameters can be updated upon resubmission
Changed the default image used for CWL pipeline processes with undefined image from docker.io/bash:5 to public.ecr.aws/docker/library/bash:5
Updated the choice of default nextflow docker image which is used when no docker image is defined. It is now public.ecr.aws/lts/ubuntu:22.04_stable
The analysis logs in the analysis details page can be refreshed
The user is now able to write a pipeline which executes files located in the /data/bin directory of the runtime environment
Pipeline files are now shown in a tree structure for easier overview
Cohorts
Updated GWAS UK Biobank data base gives users access to more phenotype information
Users can now incrementally ingest their molecular data for germline, CNV, structural variants, and somatic mutation data
Fixed Issues
General
Added an "All" option to the workgroup selection box in the projects view to reset the filter, which previously required you to delete all characters from the filter
Fixed an issue where updating two base permissions at the same time would sometimes not execute correctly
Fixed an issue where creating grid filters could result in a nullpointer error
Fixed an issue where 'Copy to Clipboard' button did not work anymore
After searching for a folder in the search box and going into that folder, the search box is now cleared
Improved the project permissions API to correctly handle empty values
Previously, when attempting to save and send a message from the Websolutions section without a unique subject, the system would report an error and still send the message. Now the non-unique message subject error is reported and no message is sent
Fixed an issue where linking samples in the sample screen would result in receiving the same "sample(s) linked" message twice
Improved error handling for CLI FUSE driver
Hardened log streaming for ongoing runs to better handle network issues which previously would result in missing log streaming
Add retries for "connected reset by peer" network-related errors during analysis upload tasks
Fixed an issue where inviting a user to collaborate on a project containing base would result in the error "entity not managed" if that user did not have base enabled in any project or if base was not enabled in the project tenant
Data Management
Fixed an issue where data could be moved to a restricted location called /analyses/ and no longer be visible after the move. Please contact Illumina Support with your data move job information to recover your data if you have encountered this issue
Fixed an issue where sorting on data format did not work correctly
Copying empty folders no longer results in a partially copied status
ICA now performs an automatic refresh after unlinking or deleting a sample
Improved handling of file path collisions when handling linked projects during data copy / move
Fixed an issue where, even though uploading a file in a linked folder is not permitted, this would erroneously present a success message without copying the file
Analysis-events which are too large for SQS (256KB) are now truncated at the first 1000 characters when using SQS
Improved error handling when trying to upload files which no longer exist
Fix system degradation under load by introducing rate limit for spawning tasks for a given analysis to 25 per 1 min
The createUploadUrl endpoint can now be used to upload a file directly from the region where it is located. The user can create both new files and overwrite files in status "partial"
Improved the project data list command with wildcard support. For example:
/ or /* will return the contents of the root
/folder/ will return the folder
/folder/* will return the contents of the folder
To optimize performance, a limit has been set to prevent concurrent uploading of more than 100 files
Fixed an issue where folder syncing functionality would sometimes result in “Unhandled exception in scheduling worker”
Flow
Fixed an issue where writing a pipeline which executes files in the /data/bin folder wasn't functioning properly with larger storage classes
Nextflow pipelines no longer require pipeline inputs when starting them via the CLI
Improved error handling when using an unsupported data format in the XML input forms during pipeline creation
Fixed the issue where it was not possible to add links in the detail page for pipelines and bundles
Sorting is no longer supported on duration and average duration columns for analysis grids
In situations where the user would previously get the error "zero choices with weight >= 1" after the first attempt, additional retries will execute to prevent this from occurring
Cohorts
Fixed an issue resulting in a blank error when a cohort with hundreds of diagnostic concepts was created
Features and Enhancements
Flow
Improved analysis queue times during periods of limited compute resource availability
Features and Enhancements
General
New notification to the user when a copy job finishes running
Updated the "GET analysis storage" API endpoint to account for the billing mode of the project. If the billing mode of the project is set to tenant, then the analysis storage of the user's tenant will be returned. If the billing mode of the project is set to project, then the analysis storage of the project's owner tenant will be returned
A ReadMe file containing used libraries and licenses is now available for ICA CLI
Data Management
New DataFormats YAML (.yaml, .cwl), JAVASCRIPT (.js, .javascript), GROOVY (.groovy, .nf), DIFF (.diff), SH (.sh), SQL (.sql) to determine the used syntax highlighting when displaying/editing new pipeline files in the UI
ICAv2 CLI supports moving data both between and within projects
Added an alert to notify users when data sharing or data move is disabled for the project
A new version of the Event Log endpoint has been developed to support paging, retrieval of previous events, and resolution of inconsistencies in date formats. This new endpoint introduces the EventLogListV4 data transfer object
The user is now able to select a single file and download it from the browser directly. This does not apply for folders and multiple files selected at once
User can subscribe to notifications when data is unarchived
The BaseSpace Sequencing Run Experiment name will now be added to the technical tags when a workflow session is launched
Flow
Fastqs with the .ora extension are now supported when staging these for secondary analysis, either as a list of fastqs or as fastq_list_s3.csv files
Before, users had to click on the pipeline on the pipeline overview screen to start a new analysis. Now, you will enter the pipeline in edit mode when you click on the pipeline name. If you want to select a pipeline to start an analysis, you need to check the checkbox
Fixed Issues
General
Removed the refresh button from the workspace detail view as it was superfluous
Fixed an issue where searching for certain characters in the search field of the Projects or Data overviews screen would result in an indecipherable error
Improved security handling around tenant admin-level users in the context of data move
Data Management
Fixed a bug so folders copied from another previously copied folder no longer results in a corrupted file
Fixed an issue where creating a new bundle would result in an error if a project with the same name already exists
Data move between projects from different tenants is now supported
Fixed an issue where not selecting files before using the copy or move commands would result in EmptyDataId errors
For the CLI, Improved notifications when files can not be downloaded correctly
Fixed an issue where scheduled downloads of linked data would fail without warning
Corrected an issue where the tenant billing mode would be erroneously set to Illumina after a data copy
Fixed an issue where BatchCopy on linked data did not work
Flow
Resolved an issue to ensure that when a user creates a pipeline using a docker image shared from an entitled bundle, their analyses utilizing that pipeline can pull the docker image without errors
Removed superfluous options from the analysis status filter
Awaiting input
Pending request
Awaiting previous unit
Fixed an issue where writing a pipeline which executes files in the /data/bin folder wasn't functioning properly with larger storage classes
Fixed an issue where many-step analyses are getting stuck in "In Progress" status
Fixed an issue where the wrapper scripts when running a CommandLineTool in CWL would return a warning
Fixed the issue which caused the "Save as" option not to work when saving pipelines
Base
Fixed an issue where the ICA reference fields in the schema definition had the wrong casing. As a result of this update you might end up with 2 different versions of the reference data (one with keys written with an uppercase letter at the start, another one with keys written entirely in lowercase letters). To fix this:
Update your queries and use the Snowflake function : GET_IGNORE_CASE (ex: select GET_IGNORE_CASE( to_object(ica) , 'data_name' ) from testtableref)
Update the 'old' field names to the new ones (ex: update testtableref_orig set ica = object_delete(object_insert(ica, 'data_name', ica:Data_name), 'Data_name'))
Fixed an issue where using an expression to filter the "Base Job Success" event is not working
Fixed Issues
Flow
Resolved an issue to ensure that when a user creates a pipeline using a docker image shared from an entitled bundle, their analyses utilizing that pipeline can pull the docker image without errors.
Features and Enhancements
General
The left side navigation bar will collapse by default for screen smaller than 800 pixels. The user can expand it by hovering over it
The browser URL may be copied to share analyses, pipelines, samples, tools, workspaces and data in various contexts (project, bundle)
Data Management
Users are now able to move data within and across projects:
The user can:
Move available data
Move up to 1000 files and/or folders in 1 move operation
Retain links to entities (sample, sequencing run, etc.) and other meta-data (tags, app-info) when moving
Move data within a project if the user is a contributor
Move data across projects if (1) in the source project the user has download rights, has at least contributor rights, and data sharing is enabled, and (2) the user has upload rights and at least viewer rights in the target project
Move data across projects with different types of storage configurations (user-defined or default ICA-managed storage)
Select and move data to the folder they are currently in through the graphical UI
Select and move data in a destination project and/or folder through the API
The user cannot:
Move linked data. Only the source data can be moved
Move data to linked data. Can only move data to the source data location
Move data to a folder that is in the process of being moved
Move data which is in the first level of the destination folder
Move data to a destination folder which would create a naming conflict such as a file name duplicate
Move data across regions
New Event Log entries are provided when a user links (ICA_BASE_100) or unlinks (ICA_BASE_101) a Cohorts data set to a bundle
Added support for the following data formats: ora, adat, zarr, tiff and wsi
Flow
New compute types (Transfer Small, Transfer Medium, Transfer Large) are supported and can be used in upload and download tasks to significantly reduce overall analysis runtimes (and overall cost)
API: All the endpoints containing pipeline information now contain the status from the pipeline(s) as well
Bench
External Docker images will no longer display a status as they consistently showed 'Available,' even when the URL is not functional
Cohorts
Performance improvements to needle plot by refactoring its API endpoint to return only sample IDs
Users now click a cancel button that returns them to the landing page
Users can now perform time series analysis for a single patient view
Refresh of PrimateAI data now drives data in variant tables
Users can now access the structural variant tab in the Marker frequency section
Fixed Issues
General
Fixed an issue where, when a user is added to or removed from a workgroup, they could be stuck on an infinite redirect loop when logging in
Fixed syncing discrepancy issues about deleted files in user-managed storage projects with Lifecycle rules & Versioning
Data Access & Management
Sorting API responses for the endpoint GET /api/jobs is possible on the following criteria: timeCreated, timeStarted and timeFinished
Improved the error message when trying to link a bundle which is in a different region than the project
More documentation has been added to the GET /eventLog regarding the order of rows to fetch
Fixed an issue where the API call - POST api/projects/{projectId}/permissions would return an error when DATA_PROVIDER was set for roleProject
Fixed an issue stemming from attempts to copy files from the same source to the same destination, which incorrectly updated file statuses to Partial
CLI: Fixed an issue where the environment variable ICAV2_X_API_KEY did not work
Flow
The analysis is no longer started from the API if error 400 ( 'Content-Type' and 'Accept' do not match) occurs
Base
Fixed an issue where the Base schedule would not run automatically in some cases when files are present in the schedule
Bench
Improved error handling when trying to create a tool with insufficient permissions
Fixed an issue where the user is unable to download docker-image with adhoc-subscription
The "version":"string" field is now included in the API response GET /api/referenceSets. If no version is specified, the field is set to "Not Specified"
Fixed an issue where, under some conditions, fetching a job by id would throw an error if the job was in pending status
Features and Enhancements
Data Management
The GUI now has a limit of 100 characters for the name and 2048 characters for the URL for links in pipelines and bundles
Added a link to create a new connector if needed when scheduling a data download
Improved the data view with additional filtering in the side panel
Flow
New CLI environment variable ICA_NO_RETRY_RATE_LIMITING allows users to disable the retry mechanism. When it is set to "1”, no retries are performed. For any other value, http code 429 will result in 4 retry attempts after 0.5, 2, 10, and 30 seconds * Code-based pipelines will alphabetically order additional files next to the main.nf or workflow.cwl file
When the Compute Type is unspecified, it will be determined automatically based on CPU and Memory values using a "best fit" strategy to meet the minimum specified requirements
Bench
Paths can be whitelisted to allowed URLs on restricted settings
Fixed Issues
General
Fixed an issue where the online help button does not work upon clicking on it
Data Access & Management
Improved automatic resource cleanup when hiding a project
Fixed an issue with the service connector where leading blanks in the path of an upload/download rule would result in errors. It is no longer possible to define rules with leading or trailing blanks
Fixed an issue where a folder copy job fails if the source folder doesn't have metadata set
Linking data to sample has been made consistent between API and GUI
Improved resource handling when uploading large amounts of files via the GUI
Fixed an issue where the API endpoint to retrieve input parameters for a project pipeline linked to a bundle would fail when the user is not entitled on the bundle
Fixed an issue where deleting and adding a bundle to a project in one action does not work
Flow
The event sending protocol was rewritten to limit prematurely exhausting event retries and potentially leaving workflows stuck when experiencing high server loads or outages
Fixed an issue where specifying the minimum number of CPUs using coresMin in a CWL workflow would always result in the allocation of a standard-small instance, regardless of the coresMin value specified
Fixed an issue in the API endpoint to create a Nextflow analysis where tags were incorrectly marked as mandatory inputs in the request body
Fixed an issue with intermittent failures following completion of a workflow session
Base
Improved syntax highlighting in Base queries by making the different colors more distinguishable
Bench
Fixed an issue where the Bench workspace disk size cannot be adjusted when the workspace is stopped. Now, the adjusted size is reflected when the workspace is resumed
Fixed an issue where regions were not populating correctly for Docker images
Fixed an issue where API keys do not get cleaned up after failed workspace starts, leading to unusable workspaces once the API key limit is reached
Features and Enhancements
Cohorts
Users can now query variant lists with a large number of associated phenotypes
Users can now perform multiple concurrent data import jobs
Fixed Issues:
Cohorts
Fixed an issue with displaying shared views when refreshing a Bundle’s shared database in Base
Fixed Issues
Fixed an issue where autolaunch is broken for any users utilizing run and samplesheet inputs stored in BSSH and operating in a personal context, rather than a workgroup.
Features and Enhancements
Data Management
Data (files and folders) may be copied from one folder to another within the same Project
The empty ‘URN’ field in the Project details at Project creation is now removed
The ‘Linked Bundles’ area in the Project details at Project creation is now removed as you are only allowed to link Bundles after Project creation
The card or grid view selected will become the default view when navigating back to the Projects or Bundles views
Added a new API endpoints to retrieve and accept the Terms & Conditions of an entitled bundle:
/api/entitledbundles{entitledBundleId}/termsOfUse
/api/entitledbundles/{entitledBundleId}/termsOfUse/userAcceptance/currentUser
/api/entitledbundles/{entitledBundleId}/termsOfUse:accept
Flow
Added a new API endpoint to retrieve orchestrated analyses of a workflow session
GET /api/projects/{ProjectID}/workflowSessoins/{WorkflowSessionID}/analyses
Code-based pipelines will alphabetically order additional files next to the main.nf or workflow.cwl file
Bench
New JupyterLab - 1.0.19 image published for Bench using the Ubuntu 22.04 base image
Resources have been expanded to include more options for compute families when configuring a workspace. See ICA help documentation for more details
Cohorts
Sample count for an individual cohort may be viewed in the variants table
Filter the variants list table through the filter setting in the needle plot
Execute concurrent jobs from a single tenant
Improved the display of error and warning messages for import jobs
Structural variant tab may be accessed from the Marker frequency section
Fixed Issues
Data Access & Management
Bundles now reflect the correct status when they are released instead of the draft status
Double clicking a file opens the data details popup only once instead of multiple times
Improved performance to prevent timeouts during list generation which resulted in Error 500
The counter is now accurately updated when selecting refresh in the Projects view
Fixed an issue resulting in one job to succeed and one to fail when running two or more file copy jobs at the same time to copy files from same project to same destination folder
Fixed an issue resulting in an error in a sample when linking nested files with the same name
Added a new column to the Source Data tab of the Table view which indicates the upload status of the source data
Removed the unused ‘storage-bundle’ field from the Data details window
Fixed an issue where the Project menu does not update when navigating into a Project in Chrome browsers
(CLI) Fixed an issue where deleting a file/folder via path would result in an error on Windows CLI
Base
Improved schedule handling to prevent an issue where some files were not correctly picked up by the scheduler in exceptional circumstances
Fixed an issue where an incorrect owning tenant is set on a schedule when running it before saving
The number of returned results which is displayed on the scheduler when trying to load files now reflects the total number of files instead of the maximum number of files which could be displayed per page
Fixed an issue where Null Pointer Exception is observed when deleting Base within a Project
Bench
Fixed an issue where users were unable to delete their own Bench image(s) from the docker repository
Cohorts
Fixed an issue where the value in the tumor_or_normal field, in the phenotype table in database, would not set properly for germline and somatic mutation data
Fixed an issue where large genes with subjects containing large sets of diagnostic concepts caused a 503 error
Fixed Issues
Fixed an issue where automated analysis after sequencing run in non-US regions may fail for certain analysis configurations
Features and Enhancements
Data Management
The --exclude-source-path flag has been added to the ‘project data download’ command so that subfolders can be downloaded to the target path without including the parent path
The system automatically re-validates storage credentials updated in the graphical UI
Added a new API endpoint to validate storage configurations after credentials are changed: /api/storageConfigurations/{storageConfigurationId}:validate
Notifications
Added support for multi-version notification event payloads corresponding to versioned API response models
Flow
(API) Improved the analysis-dto by adding a new POST search endpoint as a replacement for the search analysis GET endpoint. The GET endpoint will keep working but we advise using the new POST endpoint.
Improved analysis statuses to reflect the actual status more accurately
Parallelized analysis input data downloads and output data uploads to reduce overall analysis time
No scratch size is allocated if tmpdirMin is not specified
Cohorts
Performance improvements of the ingestion pipeline
Performance improvements to subject list retrieval
Increased the character limit of ingestion log messages to the user
Fixed Issues
Data Access & Management
Fixed an issue where the target user cannot see analysis outputs after a successful transfer of analysis ownership in BaseSpace Sequence Hub
Update the API Swagger documentation to include paging information for: /projects/{projectId}/samples/{sampleId}/data
Fixed an issue resulting in errors when creating a new bundle version
Fixed an issue where the GET API call with the ‘Sort’ parameter returns an error when multiple values are separated by commas followed by a space
Fixed an issue where adding the –eligible-link flag to the ‘projectdata list’ API endpoint caused other flags to not work correctly
Added cursor-based pagination for the ‘projectdata list’ API endpoint
Fixed an issue with the entitled bundles cards view where the region is cut off when the Status is not present
Fixed an issue where bundle filtering on categories did not work as expected
Fixed an issue where file copy across tenants did not work as expected
Added a cross-account permission check so that file copy jobs fail when the cross-account set up is missing instead of being retried indefinitely
Fixed an issue where ‘Get Projects’ API endpoint returns an error when too many projects are in the tenant
Fixed an issue where the UpdateProject API call (PUT /api/projects/{projectId}) returns an error when technical tags are removed from the request
Fixed an issue where users need to confirm they want to cancel an action multiple times when clicking the back button in the graphical UI
Fixed an issue where clicking into a new version of a bundle from the details view does not open the new version, and instead directs to the bundle card view
Flow
Fixed an issue where the analysis logs are returned in the analysis screen “outputs” section and included in the getAnalysisOutputs API response. The log output is no longer considered as part of the analysis outputs
Analysis history screen has been removed
Fixed an issue resulting in inability to retrieve pipeline files via the API when the pipeline is shared cross-tenant
Fixed an issue where the API endpoint to retrieve files for a project pipeline would not return all files for pipelines created via CLI or API
Fixed an issue where the API does not check the proprietary flag of a pipeline before retrieving or downloading the pipeline files
Base
The ‘Download’ button is available to download Base activity data locally (and replaces the non-functional ‘Export’ button for restricted bundles)
Fixed an issue resulting in missing ICA reference fields in table records if the file was loaded into the table with no metadata
Improved consistency of the references included in the scheduler
Bench
Users are now logged out from a terminal window opened in a workspace after a period of inactivity
Fixed an issue where permissions could not be enabled after a workspaces has been created
Fixed an issue where a Contributor could not start/stop a workspace
Cohorts
Fixed an issue where large genes with subjects with large sets of diagnostic concepts cause a 503 error
Fixed an issue where the value in tumor_or_normal field in the phenotype table in the database is not set properly for germline and somatic mutation data
Resolved a discrepancy between the number of samples reported when hovering over the needle plot and the variant list
Features and Enhancements
General
Users can now access the system via https://ica.illumina.com or https://ica.illumina.com/ica
Data Management
Users are now able to revalidate storage configurations in an Error state
Improved existing endpoints and added new endpoints to link and unlink data to a bundle or a project in batch:
POST /api/projects/{projectId}/dataUnlinkingBatch
GET /api/projects/{projectId}/dataUnlinkingBatch/{batchId}
GET /api/projects/{projectId}/dataUnlinkingBatch/{batchId}/items
GET /api/projects/{projectId}/dataUnlinkingBatch/{batchId}/items/{itemId}
Flow
Analyses started via the API can now leverage data stored in BaseSpace Sequence Hub as input
ICA now supports auto-launching analysis pipelines upon sequencing run completion with run data stored in BaseSpace Sequence Hub (instead of ICA)
Updated the API for creating pipelines to include "proprietary" setting, which hides pipeline scripts and details from users who do not belong to the tenant which owns the pipeline and prevents pipeline cloning.
Cohorts
Added support for partial matches against a single input string to the “Search subjects” flexible filtering logic
Users can now view an overview page for a gene when they search for it or click on a gene in the marker frequency charts
ICA Cohorts includes access to both pathogenic and benign variants, which are plotted in the “Pathogenic variants” track underneath the needle plot
Ingestion: UI notifications and/or errors will be displayed in the event of partially completed ingestions
Users can share cohort comparisons with any other users with access to the same project
Fixed Issues
General
Improved the project card view in the UI
Fixed an issue with user administration where changing the permissions of multiple users at the same time would result in users receiving Invalid OAuth access token messages
Data Access & Management
Improved the error message when downloading project data if the storage configuration is not ready for use
Fixed an issue causing Folder Copy jobs to time out and restart, resulting in delays in copy operations
Fixed an issue where only the Docker image of the first restricted bundle that was added could be selected
Improved the performance of folder linking with "api/projects/{ProjectID}/dataLinkingBatch"
The URL for links for "post/api/bundles" endpoint can be up to 2048 characters long
Improved the error response when using offset-based paging on API responses which contain too much data and require cursor-based paging
Fixed an issue resulting in failures downloading data from CLI using a path
The correct error message is displayed if the user does not have a valid subscription when creating a new project
Fixed an issue where changing ownership of a project does not change previous owner access for Base tables
Flow
Input parameters of pipelines are now displayed in the "label (code)" format unless there is no label available or the label equals the code, in which case only the code is shown
Fixed an issue where multiple folders were created upon starting new analyses
Fixed an issue preventing analyses from using inputs with BaseSpace v1pre3 APIs
Fixed an issue causing analyses with a specified output path to incorrectly return an error stating that the data does not exist
The following endpoint "/api/projects/{projectId}/workflowSessions/{workflowSessionId}/inputs" now supports using external data as input
Any value other than "economy" or "standard" for submitted analysis jobs will default to "standard" and use "standard"
The parameter to pass an activationcode is now optional for start-analysis API endpoints
Base
Improved the display of errors in the activity jobs screen if a Meta Data schedule fails
If an error occurs when processing metadata a failed job entry will be added in the Base Activity screen
Fixed an issue where records ingested via schedules from the same file could be duplicated
Fixed an issue where exporting the view shared via bundle would show an error 'Could not find data with ID (fol. ....)'
Resolved a NullPointerException error when clicking on Format and Status filters in the details screen of a Schedule in the Results tab
Fixed an issue where a schedule download would fail when performed by different user than the initial user
Bench
Fixed an issue when trying to query a Base table with a high limit within a workspace
Fixed an apt-get error when building images due to an outdated repository
Fixed an issue where a stopped workspace would display "Workspace paused" instead of "Workspace stopped"
Fixed an issue where large files (e.g., 150GB+) could not be downloaded to a fuse-driver location from a Workspace, and set the new limit to 500GB
Cohorts
Fixed an issue where split Nirvana JSON files are not recognized during ingestion
Fixed an issue causing the UI hangs on large genes and returns a 502 error
Fixed an issue where OMOP files are not correctly converted to CAM data model, preventing OMOP data ingestions
Fixed an issue where OMOP large drug ingestions led to memory issues and preventing further drug data ingestion
Fixed an issue where users from a different tenant accessing a shared project could not ingest data
Features and Enhancements
Flow
Analysis logs (task stdout/stderr files) are now written to a folder named ‘ica_logs’ within the analysis output folder
Default scratch disk size attached to analysis steps reduced from 2TB to 0B to improve cost and performance of analyses. Pipelines created before ICA v2.21.0 will not be impacted
Notifications
Notifications can now be updated and deleted in externally managed Projects
API
Clarified on the Swagger page which sorting options apply to which paging strategy (cursor-based versus offset-based). Changed the default sorting behavior so that:
When no paging strategy is specified and no sort is requested, then cursor-based paging is default
When no paging strategy is specified and sort is requested, then offset-based paging is default
Cohorts
Procedure Search Box: Users can now access additional UI functionalities for Procedures
Users can now access Procedure codes from OMOP
Improved handling of drug codes across all reports, excluding Survival comparison
Ingestion
Users now have enhanced job warning log and API status improvements
Users now require download permissions to facilitate the data ingestion process
Fetch Molecular Files: Improved import – Users can now input a directory path and select sample files individually
Variant Type Summary: Users can now access a new variants tab that summarizes Variant type statistics per gene
Added sorting and filtering capabilities to report tables, such as variants observed in genes
Users can now view sample barcodes, replacing internal auto-increment sample IDs in the Structural Variants table within the Genes tab
“Search subjects” functionality improved with flexible filtering logic that now supports partial matches against a single input string
Fixed Issues
Data Management
Fixed an issue with data copy via the CLI where the file was being copied to a subfolder of the intended location instead of the specified folder
Resolved an issue where browser upload hangs intermittently when creating data
Fixed an issue where the delete popup does not always disappear when deleting data
Fixed an issue where GetFolder API call returns 404 error if the Create and Get operations are performed 100ms apart
Fixed an issue where file copy would fail if the file was located at the root level of User’s S3 storage bucket
Fixed an issue causing data linked from externally managed projects to be incorrectly excluded from the list project data API response
Fixed an issue where User cannot use data URNs to identify the destination folder when interacting with copy data API endpoints
Bundles: Fixed an issue where clicking the back button before saving a new bundle leads to inconsistencies
Flow
Fixed an issue where pipeline documentation is not scrollable when launching pipeline
Fixed an issue with logfiles of a task not being available for streaming while the task is still running
Fixed an issue where using the 're-run' button from the analysis page reverts the storage size selection to default
Fixed an inconsistency where the following two endpoints would show different analysis statuses:
GET /api/projects/{projectId}/analyses
GET /api/projects/{projectId}/analyses/{analysisId}
Improved performance issues with UI loading data records when selecting inputs for analysis
Fixed a caching issue which resulted in delays when running pipelines
Fixed an issue where back button for analysis or pipeline details does not always direct Users back to analysis or pipelines view, respectively
Fixed an issue where system performance is degraded when large batches (e.g., 1,000) of data are added as input to Analyses via the graphical UI. It is recommended to start Analyses with large numbers of input files via API
Base
Fixed an issue where enabling Base from a Base view other than Base Tables returned a warning message
Fixed an issue where Base access was not enabled when a bundle with tables is added to a project without Base (Base is automatically enabled so users can see the bundle's tables). However, access to the bundle's tables is revoked upon the deletion of Base, and was not granted again once Base was re-enabled
Fixed an issue where a Base job to load data into a table never finished because the file was deleted after the job started and before it finished. Now the job will end up in a Failed state
Cohorts
Fixed an issue where needle plot filtered out data points reappear when zooming in the exon when a filter is in place
Fixed an issue where users from a different tenant who accept a project share may encounter a failure at the final step of the data ingestion process
Fixed an issue where users can encounter intermittent errors when browsing and typing for a gene
Fixed an issue where the UI hangs on large genes and returns a 502 error
Fixed Issues
Data Management
Fixed an issue where multiple folder copy jobs with the same destination may get stuck In Progress
Fixed an intermittent issue where tags on the target folder for a batch data update call are not set, but are set for all child data
Flow
Fixed an issue causing intermittent pipeline failures due to an infrastructure error
Features and Enhancements
General
Navigation: If multiple regions are enabled for the same tenant, the region will be indicated in the waffle menu
Logging: Data transfers of BaseSpace Sequence Hub projects with data stored in ICA will be traced in ICA logs
Cohorts
Disease Search Box: Added support for specifying subjects by age of onset of disease(s)
Drug Search Box: Added a new query builder box for Drugs
Ingestion: Support for Drug, drug route, etc. attached to subjects
Cohorts building: Users can build cohorts by specifying drugs, drug route, etc.
Ingestion
Combine different variant types during ingestion (small variants, cnv, sv)
Cohorts supports Illumina Pisces variant caller for hg19 VCFs
Fixed Issues
General
Fixed an issue where the graphical UI hands with ha spinning wheel when saving or executing a command
Fixed an issue where rich text editor for Documentation tab on Pipelines, Tools, Projects and Bundles does not populate with correct styles in edit mode
Data Management
Fixed an issue where multiple clicks on create data in Project API endpoint resulted in multiple requests
Fixed an issue where the secondary data selection screen could not be resized
A spinning wheel icon with ‘copying’ status is displayed at the folder level in the target Project when a folder is being copied. This applies to the actual folder itself and not for folders higher up in the hierarchy
Fixed an issue where API to retrieve a project data update batch is failing with 500 error when either the Technical or the User tags are updated during the batch update request
Fixed an issue where linking jobs fail to complete if other linking jobs are running
Improved performance for data transfer to support BaseSpace Sequence Hub Run transfers
Fixed an issue causing some folder copy jobs to remain in "Partially Succeeded" status despite being completed successfullyBundles: Fixed an issue where the URL and Region where a Docker image is available is not displayed for a Docker image Tool shared via an entitled Bundle
Fixed an issue where the folder copy job was getting stuck copying large amounts of big files
Fixed an issue where the folder counts were not matching expected counts after Data linking
Fixed an issue where delete data popup would occasionally not disappear after deleting data.
Fixed an issue with data copy where referencing data from another region would not result in immediate failure
Fixed issue where uploading a folder using the CLI was not working
Fixed an issue where a Docker image shared via an entitled Bundle can be added to another region
Workflows
Fixed an issue where workflow does not fail if BCL Convert fails for a BCL Convert-only run
Flow
Improved performance when batches of data up to 1000 are added as input to an Analysis
Nextflow engine will return exit code 55 if the pipeline runner task is preempted
Fixed an issue where log files cannot be opened for any steps in an analysis while the analysis is in progress
Fixed an issue with concurrent updates on analysis
Fixed an issue where unknown data inputs in the XML of an analysis are not being ignored
The warning, close, and machine profile icons for Tools can now be seen in the graphical CWL pipeline editor
Fixed an issue where user cannot expand analysis output folder if user permissions change after starting analysis. Now, if a user has the correct permissions to start an analysis, that analysis should be able to finish correctly no matter the permissions at the time it succeeds
Base
Fixed an issue switching back from template to Empty Table did not clear the fields
Data linked from an externally managed project can be added to Base Tables
Fixed an issue in the graphical UI where schema definition does not scroll correctly when many columns are defined
Features and Enhancements
Data Management/API
Added a new endpoint available to change project owner
POST /api/projects/{projectId}:changeOwner { “newOwnerId”:”}
Added a new endpoint to copy data from one project to another:
/api/projects/{projectId}/projectDataCopyBatch
Data Management/CLI
Added the ability to copy files and folders between projects in the UI and CLI. This includes support for copying data from projects with ICA-managed storage (default) to projects with S3-configured storage.
Flow/API
When starting an analysis via the API, you can specify the input files based on HTTP(s). When your analysis is done, you will see the URL corresponding to the inputs in the UI, but you will not be able to start an analysis from the UI using this URL
Added two new endpoints for workflow sessions:
Get /api/projects/{projectId}/workflowSessions
Get /api/projects/{projectId}/workflowSessions/{workflowSessionId}/inputs
Added a new endpoint to retrieve configurations from a workflow session
Flow/CLI
Duplicate analyses submitted via the CLI will be avoided
Flow
Removed the ability to start analyses from data and sample views in the UI where a single input is selected to start analyses in bulk
Flow/Autolaunch ICA Workflow Session and Orchestrated Analyses (launched by the workflow session) now saves outputs in an organized folder structure: /ilmn-analysis/<name_used_to_create_sequencer_run_output_folder>
Base
The Base module has a new feature called ‘Data Catalogue’. This allows you to add usage data from your tenant/project if that data is available for you.
Data Catalogue views will be available and can be used in Base to query on
You will be able to preview and query Data Catalogue views through Base Tables and Query screens
The Data Catalogue will always be up to date with the available views for your tenant/project
Data Catalogue views cannot be shared through a Bundle
Data Catalogue views will also be available to team members that were added after the view was added
Data Catalogue views can be removed from the Base tables and corresponding project
By removing Base from a project, the Data Catalogue will also be removed from that project
Cohorts: Disease Search box
Cohorts now includes a disease search box to search for disease concepts. This replaces the disease concept tree explorer
Disease search box located under a Disease tab in main Query builder
Search box allows for a copy/paste action of codes to be processed as separate query elements. Currently, the feature is limited to a complete valid list
Each disease entered into the search box is displayed as a separate query item and can be set to include or exclude.
Diseases in search box can be used with boolean logic in cohort creation
Search box allows for an auto-complete of diagnosis concepts and identifiers
The disease filter is included in the cohort query summary on cohort page
Fixed Issues
Data Management
Data copy between ICA-managed projects and S3 storage configured projects is supported
Fixed an issue where storage configurations matching ICA-managed buckets would cause volume records to get associated with the wrong storage configuration in the system
API
The endpoint GET/api/projects/{ProjectID}/samples/{SampleID} correctly returns all the own samples and linked samples
Improved handling of bulk update via API when concurrent deletion of file has occurred
CLI
Fixed an issue where projectdata update tags would not update the tags
Fixed an issue to support adding the server-url as a parameter instead of having the config set
Flow
Fixed an issue resulting in failure to send a notification resulting in a failed workflow
Fixed an issue where one workflow session may override another when both are executed at the same time
Base
Fixed an issue where query download in JSON format returns an error
Added a message in the UI when a query takes longer than 30 seconds to inform the user that the query is ongoing and can be monitored in the Activity view
Added a section describing the Data Catalogue functionality
Bench
Fixed an issue where resizing the workspace to current size would prevent users from resizing for the next 6 hours
Cohorts
Fixed an issue where Gene Expression table does not display with TCGA data or for tenants with a hyphen (e.g., ‘genome-group’)
Fixed an issue where user had no way to delete a cohort comparison from a deleted cohort
Fixed an issue in the UI where multi-cohort needle plot tracks are overlapping
Fixed an issue causing failures during annotation step with ‘CNV’ data type when selection ‘GB=hg19’ and ‘CNV data’ for liftover; also observed with ‘SM data’ and ‘hg38’ without liftover (in APS1 and CAC1 regions) due to a ‘404 Not Found’ error.
Fixed Issue
Fixed an issue uploading folders via the CLI
Fixed Issue
Fixed an issue causing CWL pipelines using Docker images that do not contain bash shell executable to fail.
Fixed Issue
Fixed an issue leading to intermittent system instability.
Fixed Issue
Cohorts
Issue fixed where GTEx plot is not available for tenants with a hyphen (e.g. ilmn-demo).
Features and Enhancements
General
Versioning: The ICA version can now be found under your user when you select "About"
Versioning/API: It is possible to retrieve system information about ICA, such as the current version through GET/api/systeminfo
Logging: When an action is initiated by another application, such as BaseSpace Sequence Hub, it will be traced as well in the ICA logs
Data Management
New API endpoints are available for:
Creation of a data update in bulk: POST/api/projects/{projectId}/dataUpdateBatch
A list of data updates for a certain project: GET/api/projects/{projectId}/dataUpdateBatch/{batchId}
A list of items from the batch update: GET/api/projects/{projectId}/dataUpdateBatch/{batchId}/items
A specific item from the batch update: GET/api/projects/{projectId}/dataUpdateBatch/{batchId}/items/{itemId} Note: Batch updates include tags, format, date to be archived and date to be deleted
Data Management/API
The sequencing run information can be retrieved through its Id by using the API endpoint GET/api/sequencingRuns/{sequencingRunId}
Flow:
Auto launch now supports BCL Convert v3.10.9 pipeline and both TruSight Oncology 500 v2 pipelines (from FASTQs)
Removed "fpga-small" from available compute types. Pipelines using "fpga-small" will use the "fpga-medium"-equivalent compute specifications instead
Analyses launched/tracked by BaseSpace Sequence Hub contain relevant BaseSpace information in analysis details view
Flow/API
getPipelineParameters API returns parameter type in response
Added endpoints to retrieve and update a project pipeline definition
New API endpoint available to request the analyses in which a sample is being used
When leaving activationCodeDetailId empty when starting an analysis, the best match activation code will be used
Flow/API/CLI
Include "mountPaths" field in response for API and CLI command to retrieve analysis inputs
API
Two new API endpoints added to accept Terms and Conditions on a bundle:
GET /api/bundles/{bundleId}/termsOfUse/userAcceptance/currentUser Returns you the time of acceptance when you, the current user, accepted the Terms & Conditions.
POST /api/bundles/{bundleId}/termOfUse:accept
Add temporary credentials duration to API documentation
Notifications
List of events to which you can subscribe contains new ICA notification containing analyses updates
Bench
A new Bench permission is being introduced: Administrator. This permission allows users to manages existing workspaces and create new workspaces
The Bench Administrator role allows you to create new Bench workspaces with any permissions even if you as a Bench administrator do not have these permissions. In that case, you can create and modify the workspace, but you cannot enter that workspace. Modifying is only possible when the workspace is stopped
As a Bench Contributor you are not allowed anymore to delete a Bench Workspace, you need the Bench Administrator role.
Cohorts
Users can now ingest raw DRAGEN bulk RNAseq results for genes and transcripts (TPM), with the option to precompute differential expression during ingestion
Added support for running multiple DEseq2 analyses in the ingestion workflow through bulk processing based on sample size and specific requirements
In multiple needle plot view, individual needle plots can now be collapsed and expanded
Pop-outs for needle plot variants now contain additional links to external resources, such as UCSC
For a given cohort, display a distribution of raw expression values (TPM per gene) for a selected attributes
Use of the Cohorts maintains session between core ICA and Cohorts iFrame to prevent unwanted timeouts
Cohorts displays structural variants that include or overlap with a gene of interest
Fixed Issues
General
Collaboration: Fixed an issue where a user is presented with a blank screen when responding to a project invitation
Data Management/API
Improved error handling for API endpoint: DELETE/api/bundles/{bundleId}samples{sampleId}
Fixed an issue where the API endpoint GET /api/samples erroneously returned a 500
API endpoint GET/api/projects/{projectId}/analyses now returns the correct list when filtering on UserTags whereas it previously returned too many
Improved retry mechanism for API endpoint to create folderuploadsession
Data Management/CLI
When an upload of a folder/file is done through the CLI, it returns the information and ID of the folder/file
Data Management
CreatorId is now present on all data, including subfolders
Improved external linking to data inside ICA using deep linking
Improved error handling when creating folders with invalid characters.
Fixed an inconsistency for URN formats on output files from Analyses. This fix will apply only for analyses that are completed starting from ICAv2.18.0
Improved resilience in situations of concurrent linking and unlinking of files and folders from projects
It is only possible to delete a storage configuration if all projects that are using this storage configuration have been hidden and are not active projects anymore
Improved accuracy of the displayed project data size. Prior cost calculations were accurate, but the project data size visualization included technical background data
Fixed an issue where there is a discrepancy in number of configurations between Storage->Configurations and Configurations-> Genomics.Byob.Storage Configuration view
Flow/API
Improved error handling when invalid project-id is used in API endpoint GET /api/projects/{projectId}/pipelines
Fixed an issue when an Analysis completed with error "incomplete folder session", the outputs of the Analysis are not always completely listed in the data listing APIs
Updated ICA Swagger Project > createProject to correctly state that the analysis priority must be in uppercase
Flow
When a spot instance is configured, but revoked by AWS, the pipeline will fail and exit code 55 is returned
Fix to return meaningful error message when instrument run ID is missing from Run Completion event during an auto launched analysis
Improved parallel processing of the same analysis multiple times
Base
Improved error handling when creating queries which use two or more fields with the same name. The error message now reads "Query contains duplicate column names. Please use column alias in the query"
Fixed an issue where queries on tables with many entries fail with NullPointerException
Bench
Clarified that changes to Bench workspace size only take effect after a restart
Cohorts
Fixed issue where counts of subjects are hidden behind attribute names
Fixed issue where the state of checked files are not retained when selecting molecular files that are in multiple nested folders
Fixed issue where projects that contain files from linked bundles cause a time out, resulting in users not being able to select files for ingestion
Fixed an issue where the 'Import Jobs' page loaded within the Data Sets frame, depending on where the import was initiated
Fixed an issue in the Correlation plat where x-axis counts were hidden under attribute names
Fixed an issue where users were previously incorrectly signed out of their active sessions
Fixed Issues
Fixed an issue causing analyses requesting FPGA compute resources to experience long wait times (>24h) or not be scheduled
Features and Enhancements
Data Management
Performance improvements for data link and unlink operations – Larger and more complex folders can now be linked in the graphical UI, and progress can be monitored with a new visual indication under Activity > Batch Jobs
Notifications
Notifications are now available for batch job changes
Flow
Increased the allowed Docker image size from 10GB to >20GB
CWL: Added support for javascript expressions “ResourceRequirements” fields (i.e., type, size, tier, etc.) in CWL Pipeline definitions
Flow/API
Added support for using Pipeline APIs to query Pipelines included in Entitled Bundles (i.e., to retrieve input parameters)
Added support for providing S3 URLs as Pipeline data inputs when launching via the API (using storage credentials)
Added support for specifying multi-value input parameters in a Pipeline launch command
Bench
Project and Tenant Administrators are now allowed to stop running Workspaces
Cohorts
Enhanced ingestion workflow to ingest RNAseq raw data from DRAGEN output into backend Snowflake database
Added support for running multiple DEseq2 analyses in the ingestion workflow through bulk processing based on sample size and specific requirements
Multi-Cohort Marker Frequency - Added Multi-Cohort Marker Frequency tab allowing users to compare expression data across up to four Cohorts at the gene level
Multi-Cohort Marker Frequency includes a pairwise p-value heat map
Multi-Cohort Marker Frequency - Includes frequencies for Somatic and Copy Number Variants
Tab added for a multi-cohort marker frequency analysis in cohort comparisons
Multi-Cohort Needle Plot - Added new tab in the Comparison view with vertically aligned needle plots per cohort for a specified gene, allowing collapsible and expandable individual needle plots
Additional filter logic added to multi-cohort needle plot
Improved DRAGEN data type determination during ingestion allowing for multiple variant type ingestion
Enhanced list of observed variants with grouped phenotypes and individual counts, including a column for total sample count; tooltips/pop-outs provide extended information
Updates to needle plot link outs
Improved the Comparison feature by optimizing API calls to handle subjects with multiple attributes, ensuring successful loading of the page and enabling API invocation only when the user selects or expands a section
Removed unused columns (genotype, mrna_feature_id, allele1, allele2, ref_allele, start_pos, stop_pos, snp_id) from annotated_somatic_mutations table in backend database
Refactored shared functionality for picking consequence type to reduce code duplication in PheWAS-plot and GWAS-plot components
Invalid comparisons on the Comparisons page are now grayed out and disabled This improvement prevents the selection of invalid options
Automatic retry of import jobs when there are failures accessing data from ICA API
Fixed Issues
General
Navigation: Removed breadcrumb indication in the graphical UI
Data Management
The content of hidden Projects can now be displayed
Fixed the TimeModified timestamp on files
Bundles: Resolved issues when linking a large number of files within a folder to a Bundle
Flow
Single values are now passed as a list when starting an Analysis
Pipelines will succeed if the input and output formats specified on the pipeline level match at the Tool level
Fixed an issue causing Analysis failures due to intermittent AWS S3 network errors when downloading input data
CWL: Improved performance on output processing after a CWL Pipeline Analysis completes
Flow/UI: Mount path details for Analysis input files are now visible
Flow/UI: Improved usability when starting an Analysis by filtering entitlement options based o inputs selected and available entitlements
Flow/API
List of Analyses can now be retrieved via the API based on filters for UserReference and UserTags
Base
Fixed an issue where the Scheduler continues to retry uploading files which cannot be loaded
Bench
Resolved an issue when attempting to access Workspaces with multi-factor authentication (MFA) enabled at the Tenant-level
API
Improved error messaging for POST /api/projects/{projectId}/data/{dataId}:scheduleDownload
Cohorts
Fixed issue where Correlation bubble plot not showing for any projects intermittently
Fixed issue where importing Germline/hg19 test file did not load variants for a specific gene in the Needle plot due to missing entries in the Snowflake table
Fixed a bug causing an HTTP 400 error while loading the Cohort for the second time due to the UI passing "undefined" as variantGroup, which failed to convert to the VariantGroup Enum type
Fixed issue where scale (y-axis) of needle plot is changed even if value of sample count gnomAD frequency is not accepted
Fixed an issue where no data was generated in the Base Tables after a successful import job in Canada - Central Region (CAC1)
Fixed issue where long chart axis labels overlap with tick marks on graph
Features and Enhancements
General
Navigation: Updated URLs for Correlation Engine and Emedgene in the waffle menu
Authentication: Using POST /api/tokens:refresh for refreshing the JWT is not possible if it has been created using an API-key.
Authentication: Improved error handling when there is an issue reaching the authentication server
Authentication: Improved usability of "Create OAuth access token" screen
Data Management
You can now select 'CYTOBAND' as format after file upload
Added support for selecting the root folder (of the S3 bucket) for Projects with user-managed
Added support for creating an AWS Storage Configuration with an S3 bucket with Versioning enabled
Auto-launch
Added technical tags for upstream BaseSpace Run information to auto-launched analyses
Added support for multiple versions of BCL Convert for auto-launched analyses
Flow
Added support for '/' as separator in CWL ResourceRequirements when specifying Compute Type
Flow/API
The API to retrieve analysis steps now includes exit code for completed steps
Bench
Restricted Workspaces (Open or Restricted) always allow for access to Project Data within the Workspace
Restricted Bench workspaces have limited access through whitelisted URLs that are checked before entry
Restricted Bench Workspaces allow for Open or Restricted workspaces. Restricted workspaces do not have access to the internet except for user-entered whitelist URLs
Fixed Issues
Data Management
Upload for files names including spaces is now consistent for connector and browser upload. We do still advise not to use spaces in file names in general
Fixed search functionality in Activity > Data Transfers screen
Improved performance on opening samples
Fixed an issue where reference data in download tab initiates an unexpected download
Fixed intermittent issue where the Storage configuration within a Project can go into Error status and can block users from creating records such as folders and files
Service Connector: Improved error message for DELETE/api/connectors/{connectorId}/downloadRules/{downloadRuleId}
Data Management/API
Improved error handling for API endpoints: Delete/api/projects/ {projectId}/bundles/{bundleId} and POST/api/projects/{projectId}/bundles/{bundleId}
Improved error handling for POST/api/projects/{projectId}/base:ConnectionDetails
Bundles
Fixed an issue where the Table view in Bundles is not available when linking to a new Bundle version
Fixed an issue where linking/unlinking a Bundle with Base Tables could result in errors
Bundles/API
Improved error handling for DELETE/api/bundles/{bundleId}/tools/{toolId} and POST/api/bundles/{bundleId}/tools/{toolId}
Improved error message for POST/api/bundles/{bundleId}/samples/{sampleId}
Notifications/API
Custom subscriptions with empty filter expressions will not fail when retrieving them via the API
Improved error handling for POST/api/projects/{projectId}/notificationSubscriptions
Improved notification for Pipeline success events
Flow
When the input for a pipeline is too large, ICA will fail the Analysis and will not retry
Fixed issue where analysis list does not search-filter by ID correctly
Improved error handling when issues occur with provisioning resources
When retry succeeds in a Nextflow pipeline, exit code is now '0' instead of '143'
Flow/API
Fixed an issue causing API error when attempting to launch an Analysis with 50,000 input files
Improved pipeline error code for GET/api/projects/{projectId}/pipelines/{pipelineId} when already unlinked pipeline Id is used for API call
Fixed an issue where Analyses could not be retrieved via API when the Pipeline contained reference data and originated from a different tenant
Fixed filtering analyses on analysisId. Filtering happens via exact match, so part of the Id won't work
Bench/CLI
Fixed issue where the latest CLI version was not available in Bench workspace images
Cohorts
Fixed an issue where CNV data converted from hg19 to hg38 do not show up in Base table views
Fixed an issue accounting for multiple methods of referring to the alternate allele in a deletion from Nirvana data
Fixed intermittent issue where GWAS ingestions not working after Base enabled in a project.
Fixed Issue
Fixed an issue causing incorrect empty storage configuration dropdown during Project creation when using the “I want to manage my own storage” option for users with access to a single region
Features and Enhancements
General
General availability of sequencer integration for Illumina sequencing systems and analysis auto launch
General usability improvements in the graphical interface, including improved navigation structure and ability to switch between applications via the waffle menu in the header
Storage Bundle field will be auto-filled based on the Project location that is being chosen if multiple regions are available
Event Log entries will be paged in the UI and will contain a maximum of 1,000 entries. Exports are limited to the maximum number of entries displayed on the page.
Read-only temporary credentials will be returned when you are not allowed to modify the contents of a file
The ICA UI will only allow selection of storage bundles belonging to ICA during Project creation, and the API will only return storage bundles for ICA
Notifications
Creating Project notifications for BaseSpace externally managed projects is now supported
Flow
Allow attached storage for Pipeline steps to be set to 0 to disable provisioning attached storage and improve performance
Cohorts
GRCh37/hg19-aligned molecular data will get converted to GRCh38/hg38 coordinates to facilitate cross-project analyses and incorporating publicly available data sets.
API
Project list API now contains a parameter to filter on (a) specific workgroup(s)
Two new API endpoints are added to retrieve regular parameters from a pipeline within or without a Project context
Fixed Issues
General
Optimized price calculations resulting in less overhead and logging
Improved error handling:
during Project creation
of own storage Project creation failures.
to indicate connection issue with credential
for graphical CWL draft Pipelines being updated during an Analysis
Improved error messaging in cases where the AWS path contains (a) special character(s)
Fixed an issue causing errors when navigating via deep link to the Analysis Details view
Data Management
Fixed an issue causing data records to remain incorrectly in Unarchiving status when an unarchive operation is requested in the US and Germany regions
API
Fixed returning list of unlinked data in a sample that was linked before in GET/api/projects/{projectId}/data
Fixed error for getSampleCreationBatch when using status filter
CLI
Unarchive of folders is supported when archive or unarchive actions are not in progress for the folder
Improved error message to indicate connection issue with credentials
Flow
Fixed an issue causing incorrect naming of Analysis tasks generated from CWL Expression Tools
Fixed an issue when cloning Pipelines linked from Entitled Bundles to preserve the original Tenant as the Owning Tenant of the cloned Pipeline instead of the cloning user’s Tenant
Fixed an issue causing outputs from CWL Pipelines to not show in the Analysis Details despite being uploaded to the Project Data Analysis output folder when an output folder is empty
When a Contributor starts an Analysis, but is removed afterwards, the Analysis still runs as expected
Fixed an issue where Analyses fail where Nextflow is run a second time
Fixed an issue causing API error when attempting to launch an Analysis with up to 50,000 input files
Fixed an issue causing degraded performance in APIs to retrieve Analysis steps in Pipelines with many steps
Fixed an issue causing Analysis failure during output upload with error “use of closed network connection”
Fixed an issue causing disk capacity alter log to not show when an Analysis fails due to disk capacity and added error message
Fixed an issue preventing cross-tenant users from being able to open a shared CWL pipeline
Base
Improved target Table selection for schedulers to be limited to your own Tables
Bench
Fixed an issue causing Workspaces to hang in the Starting or Stopping statuses
Cohorts
Now handles large VCFs/gVCFs correctly by splitting them into smaller files for subsequent annotation by Nirvana
Features and Enhancements
General
Added a limit to Event Log and Audit UI screens to show 10,000 records
API
Parent output folder can be specified in URN format when launching a Workflow session via the API
Flow
Reduced Analysis delays when system is experiencing heavy load
Improved formatting of Pipeline error text shown in Analysis Details view
Users can now start Analyses from the Analysis Overview screen
Superfluous “Namespace check-0” step was removed to reduce Analysis failures
Number of input files for an Analysis is limited to 50,000
Auto launched Workflow sessions will fail if duplicate sample IDs are detected under Analysis Settings in the Sample Sheet
Base
Activity screen now contains the size of the query
Cohorts
Detect and Lift Genome Build: Cohorts documentation provides set-up instructions to convert hg19/GRCh37 VCFs to GRCh38 before import into Cohorts.
Attribute Queries: Improved the user experience choosing a range of values for numerical attributes when defining a cohort
Export Cohort to ICA Project Data: Improved the user experience exporting list of subjects that match cohort definition criteria to their ICA project for further analysis
Ingest Structural Variants into database
The Cohorts ingestion pipeline supports structural variant VCFs and will deposit all such variants into an ICA Base table if Base is enabled for the given project
Structural variants can be ingested and viewed in base tables
Needle Plot Enhancements
Users can input a numerical value in the Needle Plot legend to display variants with a specific gnomAD frequency percentage or sample count
The needle plot combines variants that are observed among subjects in the current project as well as shared and public projects into a single needle, using an additional shape to indicate these occurrences
Needle Plot legend color changes for Variant severity; pathogenic color coding is the same as the color coding in the visualization; differentiating hue between proteins and variants; and other color coding changes.
Needle plot tool tips that display additional information on variants and mutations are now larger and modal
The needle plot now allows to filter by gnomAD allele frequency and sample count in the selected cohort. Variants include links to view a list of all subjects carrying that variant and export that list.
Remove Samples Individually from Cohorts
Exclude individual subjects from a cohort and save the refined list
The subjects view allows users to exclude individual subjects from subsequent analyses and plots and save these changes Subject exclusions are reset when editing a cohort
Subject Selection in Analysis Visualization: Users can follow the link for subject counts in the needle plot to view a list of subjects carrying the selected variant or mutation.
UI/UX: Start and End time points are available as a date or age with a condition attribute in the subject data summary screen.
Fixed Issues
General
Improved resilience against misconfiguration of the team page when there is an issue with Workgroup availability
Removed ‘IGV (beta)’ button from ‘View’ drop down when selecting Project Data in UI
Data Management
Improved handling of multi-file upload when system is experiencing heavy loads
Fixed an issue to allow upload of zero-byte files via the UI
Fixed issue where other Bundles would not be visible after editing and saving your Bundle
API:
Improved error handling for API endpoint: POST /api/projects/{projectId}/analysisCreationBatch
Improved performance of API endpoint: getbestmatchingfornextflow
Flow
Fixed an issue causing Analysis output mapping to incorrectly use source path as target path
Fixed an issue where the UI may display incorrect or invalid parameters for DRAGEN workflows which do not accurately show the true parameters passed. Settings can be confirmed by looking at the DRAGEN analysis log files.
Base
“Allow jagged rows” setting in the Scheduler has been replaced with “Ignore unknown values” to handle files containing records with more fields than there are Table columns
Improved Base Activity view loading time
Fixed an error message when using the API to load data into a Base Table that has been deleted
Bench
Fixed an issue resulting in incorrect Bench compute pricing calculations
Fixed an issue preventing building Docker images from Workspaces in UK, Australia, and India regions
Fixed an issue where /tmp path is not writeable in a Workspace
Cohorts
Fixed issue where the bubble plot sometimes failed to display results even though the corresponding scatter plot showed data correctly.
The order of messages and warnings for ingestion jobs was not consistent between the UI and an error report sent out via e-mail.
The UI now displays any open cohort view tabs using shortened (“…”) names where appropriate
Issue fixed where ingestions with multiple errors caused halting to the ingestion queue.
The needle plot sometimes showed only one source for a given variant as opposed to all projects in which the variant had been observed in.
Issue fixed with unhandled genotype index format in annotation file to base database table conversion
Status updates via e-mail sometimes contained individual error messages or warnings without a text.
Fixed issue where items show in needle plot with incorrect numbering on the y-axis.
Fixed performance issue with subject count.
Widget bar-chart counts are intermittently cut off over four digits.
Fixed slowness when switching between tabs in query builder
Fixed Issue
Fixed issue with BaseSpace Free Trial and Professional users storing data in ICA
Fixed Issue
Fixed an issue resulting in analysis failures caused by a Kubernetes 404 timeout error
Features and Enhancements
General *
Each tenant supports a maximum of 30,000 Projects
.MAF files are now recognized as .TSV files instead of UNKNOWN
Added VCF.IDX as a recognized file format
General scalability optimizations and performance improvements
API
POST /api/projects/{projectId}/data:createDownloadUrls now supports a list of paths (in addition to a list of IDs)
Fixed Issues
General
Fixed an issue preventing the ‘Owning Project’ column from being used outside of Project
Fixed an issue allowing the region of a Project to be changed. Changing the region of a resource is not supported
Strengthened data separation and improved resilience against cross-Project metadata contamination
Bundles
After creating a new Bundle the user will be taken to the Bundle Overview page
Data Management
Fixed an issue which prevented changing the format of a file back to UNKNOWN
Fixed an issue causing inaccurate upload progress to be displayed for UI uploads. The Service Connector or CLI are recommended for large file uploads.
Fixed an issue showing an incorrect status for data linking batch jobs when data is deleted during the linking job
Service Connector: Fixed an issue allowing download of a Service Connector when no operating system is set
Service Connector: Cleaned up information available on Service Connectors by removing empty address information fields
API
Fixed date formatting for GET /api/eventLog (yyyy-MM-dd’T’HH:mm:ss.SSS’Z’)
Fixed an issue where the GET users API was not case sensitive on email address
Fixed an issue causing the metadata model to be returned twice in PSOT /api/projects/{projectId}/samples:search
Fixed the listProjects API 500 response when using the pageoffset query parameter
The searchProjectSamples API returns Sample metadata for Samples shared via a Bundle
Fixed an issue causing createProjectDataDownloadUrls API 400 and 502 errors when server is under load
Flow
Fixed analysis failures caused by kubernetes 404 timeout error
Fixed an issue where Workflwos would prematurely report completion of an Analysis
Improved Pipeline retry logic to reduce startup delays
Fixed an issue where Nextflow pipelines were created with empty files (Nextflow config is allowed to be empty)
Removed the 1,000 input file limitation when starting an Analysis
Improved the performance of status update messages for pipelines with many parallel steps
Fixed an issue with overlapping fields on the Analysis Details screen
Deactivated the Abort button for Succeeded analyses
Base
Fixed an issue where Pipeline metadata was not captured in the metadata Table generated by the metadata schedule
Error logging and notification enhancements
Bench
Fixed an issue where Workspaces could be started twice
Fixed an issue where the system checkpoint folder was incorrectly created in Project data when opening a file in a Workspace
Features and Enhancements
Analysis system infrastructure updates
Features and Enhancements
Added ability to refresh Batch Jobs updates without needing to leave the Details screen.
Projects will receive a job queuing priority which can be adjusted by an Administrator.
The text "Only showing the first 100 projects. Use the search criteria to find your projects or switch to Table view." when performing queries is now displayed both on the top and bottom of the page for more clarity.
API: Added a new endpoint to retrieve download URLs for data: POST/api/projects/{projectId}/data:createDownloadUrls
API: Added support for paging of the Project Data/getProjectDataChildren endpoint to handle large amounts of data.
API: Added anew endpoint to deprecate a bundle (POST /api/bundles/{bundleId}:deprecate)
API: If the API client provides request header "Accept-Encoding: gzip", then the API applies GZIP compression on the JSON response. This way the size of the response is significantly smaller which improves the download time of the response, resulting in faster end-to-end API calls. In case of compression the API also provides header "Content-Encoding: gzip" in the response, indicating that compression was effectively applied.
Flow: Optimized Analysis storage billing, resulting in reduced pipeline charges.
Flow: Internal details of a (non-graphical) pipeline marked ‘Proprietary’ will not be shared with users from a different tenant.
Flow: A new grid layout is used to display Logs for Analyses with more than 50 steps. The classic view is retained for analyses with 50 steps or less, though you can choose to also use the grid layout by means of a grid button on the top right on the Analysis Log tab.
CLI: Command to launch a CWL and Nextlfow Pipeline now contains the mount path as a parameter.
CLI: Version command now contains the build number.
CLI: Added support for providing the nextflow.config file when creating a Nextflow pipeline via CLI.
API: HTML documentation for aPipeline can now be returned with the following requests:
GET /api/pipelines/{pipelineId}/documentation/HTML
GET /api/projects/{projectId}/pipelines/{pipelineId}/documentation/HTML
API: Added a new endpoint for creating and starting multiple analyses in batch: POST /api/projects/{projectId}/analysisCreationBatch
Flow: Linking to individual Analyses and Workflow sessions is now supported by /ica/link/project//analysis/ and /ica/link/project//workflowSession/
Cohorts: Users can now export subject lists to the ICA Project Data as a file.
Cohorts: Users can query their ingested data through ICA Base. For users who already have ingested private data into ICA Cohorts, another ingestion will need to happen prior to seeing available database shares. Customers can contact support to have previously ingested data sets available in Base.
Cohorts: Correlation bubble plot counts now link to a subject/sample list.
Fixed Issues
Tooltip in the Project Team page provides information about the status of an invite
‘Resend invite’ button in the Project Team page will become available only when the invite is expired instead of from the moment the invite is being send out
Folders, subfolders and files all contain information about which user created the data
Files and folders with UTF-8 character are not supported. Please look at the documentation on how to recover from it in case you already have used them.
Improved performance for creating or hiding a Project in a tenant with many Projects
Service Connector: Updated information in the Service Connector screen to reflect the name change from "Type of Files" to the more accurate "Assign Format"
Service Connector: Folders within a Bundle can be downloaded via the Service Connector
Service Connector: Upload rules can only be modified in the Project where they apply
Service Connector: A message describes when a file is skipped during upload because it already exists in the Project
Service Connector: Fixed an issue where opening the Connectivity tab occasionally results in a null pointer error
Service Connector: Fixed an issue causing excessive logging when downloading files with long file paths
Service Connector: Fixed an issue where the Service Connector log may contain spurious errors which do not impact data transfers
Existing storage configurations are displayed and accessible via API and UI
Newly added storage configurations do no longer remain in ‘Initializing’ state
Fixed error when creating a storage configuration with more than 63 characters
Clicking on a Data folder in flat mode will now open the details of the folder
Only Tools in Released state can be added to a Bundle
Fixed issue preventing new Bundle versions to be created from Restricted Bundles
Deprecated Bundles are displayed upon request in card and table view
Bundles view limited to 100 Bundles
API: Fixed the API spec for ProjectDataTransfer.getDataTransfers
API: Fixed an issue with the projectData getChildren endpoint which returned incorrect object and pagination
API: Fixed an issue where multiple clicks on Create sample batch API endpoint resulted in multiple requests
API: POST /api/projects/{projectId}/data/{dataId}:scheduleDownload can now also perform folder downloads
API: Improved information on the Swagger page for GET /api/pipelines, GET/api/projects/{projectId}/pipelines, and GET/api/projects/{projectId}/pipelines/{pipelineId}
API: Fixed and issue when a user provides the same input multiple times to a multi-value input on an analysis run, that input is only passed to the pipeline once instead of multiple times: POST /api/projects/{projectId}/analysis:nextflow
CLI: Copying files in the CLI from a local directory on MacOS to your Project can result in both the desired file and the metadata file (beginning with ‘./’) being uploaded. The metadata file can safely be deleted from the Project
CLI: Hardened protection against accidental file overwriting
CLI: Improved handling for FUSE when connection to ICA is lost
CLI: icav2 projectdata mount –list shows updated list of mounted Projects
CLI: Paging improvements made for project list, projectanalyses list, and projectsdata list
CLI: When there is no config or session file the user will not be asked to create one for icav2 config reset and icav2 config get
CLI: Fixed an issue where Bundle data could not be seen through FUSE in Bench
CLI: Fixed an error message when missing config file upon entering the Project context
CLI: The unmount is possible without a path and will work via the stored Project ID or with a directory path resulting in an unmount of that path
CLI: Fixed an error when creating a Pipeline using URN for Project identifier
CLI: Attempting to delete a file from an externally-managed project returns an error indicating this not allowed
CLI: Fix to delete session file when config file is not detected
CLI: Paging option added to projectsamples list data
CLI: Fixed “Error finding children for data” error in CLI when downloading a folder
CLI: projectdata list now returns the correct page-size results
Flow: Fixed handling of special characters in CWL pipeline file names
Flow: Fixed an issue where task names exceeding 25 characters cause analysis failure in CWL pipelines
Flow: Fixed an issue which prevented requests for economy tier compute
Flow: Fixed an issue limiting CWL workflow concurrency to two running tasks
Flow: Fixed an issue where analysis file inputs specified in the input.json with ‘location’set to an external URL cause to CWL pipelines to fail
Flow: Fixed an issue resulting in out of sync Pipeline statuses
Flow: Improved Nextflow engine resiliency, including occurrences where Nextflow pipelines fail with ‘pod 404 not found’ error
Flow: Fix issue with intermittent system runtime failures incorrectly causing analysis failures
Flow: Fixed an issue where links to Analysis Details returned errors
Flow: Enabled scrolling for Pipeline documentation
Flow: Improved performance for handling analyses with large numbers of inputs
Flow: Improved handling of hanging Analyses
Flow: Improved error messages for failed Pipelines
Flow: Added documentation on how to use XML configuration files for CWL Pipelines
Flow: Duplicate values for multi-value parameters are no longer automatically removed
Flow: Correct exit code 0 is shown for successful Pipeline steps
Base: Fixed an issue so that only users with correct permissions are allowed to retrieve a list of Base tables
Base: Fixed an issue with metadata scheduler resulting in a null pointer
Base: An empty Table description will not return an error when requesting to list all Tables in a Project
Base: Jobs failed with an error containing 'has locked table' are not shown on the Base Job activity list. They can be displayed by selecting the 'Show transient failures' checkbox at Projects > Activity > Base Jobs.
Base: Users can see Schedulers and their results for the entire tenant if created by a tenant administrator in their project, but not create, edit or run them
Base: Fixed an issue preventing data format change in a schedule
Base: Fixed an issue preventing exporting data to Excel format
Bench: Improved handling to prevent multiple users in a single running Workspace
Bench: Fixed an issue causing Workspaces to be stuck in "Starting" state
Bench: Fixed an issue where usage does not showing up on usage CSV-based report
Bench: Fixed an issue where Bundle data could not be seen via the Fuse driver
Bench: Users can now consistently exit Workspaces with a single click on the ‘Back’ button.
Bench: After leaving a Workspace by clicking on the ‘Back’ button, the Workspace will remain in a ‘Running’ state and become available for a new user to access
Bench: Workspaces in a ‘Stuck’ state can be manually changed to ‘Error’ state, allowing users to restart or delete them
Cohorts: Fixed issue where file system cleanup not occurring after delete.
Cohorts: Fixed sign in and authentication issues in APN1 region.
Cohorts: Fixed issue where gene filter missing when editing a cohort and removing the edited filter and cancelling. The filter was preserved and should not have been.
Cohorts: Fixed issue where users see an application tile in the Illumina application dashboard selection screen called "Cohort Analysis Module".
Cohorts: Correlation: Fixed issue, Data type selections shows half when loading the search result
Cohorts: Fixed issue, Users will see an application tile on the Connected Platform home page screen called “Cohort Analysis Module” if the Cohorts module is added to the domain. Users should not enter the ICA Cohorts through this page. They should enter through ICA."
Features and Enhancements
New notifications added for data upload, data created, and analyses with ‘aborted’ state
Flow: Progress on batch jobs can now be monitored in the Activity screen under the ‘Batch Jobs’ tab
Base: Shared database activity is available in the Base Activity Logging Screen
Cohorts:
Enhanced Attribute Queries: Filter logic now allows users to include or exclude filter criteria.
Ingestion Wizard: Human readable project name added to email notification of ingestion status
Ingestion Wizard: Notification email now includes the project name and job failure details
Ingestion Wizard: File lists are now sortable and filterable by type
Ingestion Wizard: UI enhancements on ingestion wizard selections
Ingestion Wizard: Based on data type chosen, the list of files for ingestion will be filtered for easier navigation of files to ingest
Share Cohorts with Collaborators - Cohort definitions are now sharable in a project
Statistical significance on Cohort Analysis - A p-value calculation is added to the Survival Comparison data points
Fixed Issues
Resolved an error when creating a Bucket Name with more than 63 characters
Fixed an issue causing failure when attempting to link "complex" folders (i.e., contains 100 sub-folders each with sub-folders) via GUI throws a “…transaction is not active…" error
Fixed an issue causing intermittent failure of "complex" folder (i.e., contains 100 sub-folders each with sub-folders) downloads
Fixed an issue so that the API to create data download URL correctly returns a 503 instead of a 404 response when the underlying system is experiencing network-related issues
Fixed an issue where the user is not redirected to the Project overview view upon clicking ‘Hide a Project’
Fixed an issue which prevented linking and unlinking multiple Bundles to and from a Project in a single operation via the GUI
Fixed an issue to show records in Bundle Data Transfers when data from a Bundle is accessed via View Tab and when downloaded
Fixed an issue showing stack trace when attempting to add Secondary Data to Data when not allowed
Fixed issue where the File Upload selection disappears after navigating back to Data after navigating to the Data History Tab
Fixed an issue where create folder session and get upload URL APIs are allowed for data accessible via an entitled Bundle
Fixed an issue where OpenAPI spec for delete project bundle API does not include proper "accept" header when using "Try It" functionality
Fixed an issue where write-access can be acquired for data accessible via an entitled Bundle
Fixed an issue where Project owner of an externally managed Project could not open Analysis Overview
Fixed an issue where multiple OS versions of the Service Connector are incorrectly downloaded at once when a single version is selected
Fixed an issue when selecting the "empty" option in the Operating System drop-down when downloading a connector
Fixed an issue where users with Contributor role and no data upload access can add data to samples without error
Added an event log entry for Service Connector log uploads initiated by Illumina Support
Flow: Fixed an issue to support Aborting an Analysis immediately after starting it
Flow: Fixed an issue when publishDir in Nextflow pipelines contains symlinks to data, those data are not included when uploading analysis outputs to a Project
Flow: Fixed an issue preventing closure of tabs in the Logs section for technical steps in the Analysis Details view
Flow: Cancel button added to the define CWL Pipeline selection dialogue window, and the order of buttons has been changed
Fixed an issue causing pipelines to stall for hours after all steps are completed
Flow: Fixed an issue where Nextflow pipelines fail with ‘pod 404 not found’ error
Base: Linked files from a Bundle can be inserted in a Base table
Base: Fixed an error on previewing Base tables that have been shared from a Bundle
Base: Fixed an issue where Base API to retrieve a list of Base jobs is not paging correctly
Bench: Fixed an issue where Bench Workspaces could not be opened with a double-click
Bench: After the “Save and Start” of a new Bench Workspace, the breadcrumb shows the actual name of the Workspace
Bench: Clarified error messages when creating new Workspaces with file permissions
Cohorts: Issue fixed where cohort is created during filtering process for RNA Expression on Apply
Cohorts: Issue fixed where order of items in the Survival Curve comparison legend are shuffled when reloaded
Cohorts: Issue fixed where long comparison names caused by a string concatenation between cohorts were causing the comparison feature to malfunction
Cohorts: Issue fixed where comparison name updates not reflected in the UI
Cohorts: Issue fixed where option still remains checked after Molecular filter removed from the right panel while on the project tab
Cohorts: Issue fixed where TCGA data not showing Molecular breakdown for selected genes
Cohorts: Issue fixed where data selections in the correlation feature where the second option visually shows half of the result in the UI
Cohorts: Issue fixed where "missense_and_ptv_all" missing from PHWAS/GWAS variant types
Features and Enhancements
Added format auto-detection for VCF (.hard-filtered.vcf.gz, .cnv.vcf.gz) and TAR (.v8.tar, .tar.gz) files
Pipeline documentation will be available if present when starting a new Analysis
Added Workflow Session event topics
Base: Shared databases are visible in Base tables. Shared databases are read-only and queryable tables originating from ICA modules such as Cohorts
Reduced pipeline spin up time delays due to shared files system provisioning
Added support for Illumina applications to add custom attributes to data, samples, workflow sessions, and analyses
API: "creatorId" for Project data records returned in the API
API: Added instructions for paging parameters on list endpoints in API specification
API: The edit and delete API endpoint for Terms of Use are no longer available
Updated ICA CLI to version 2.10.0
Fixed Issues
Fixed an issue allowing unauthorized users to access external-managed Project contents via a direct link
Fixed an issue in the Docker Repository view causing region text and check boxes to be misaligned
Fixed an issue causing storage configuration status to incorrectly report an error
Fixed an error preventing the creation of an AWS S3 storage configuration in a Project
Fixed an issue where the Service Connector status incorrectly showing ‘Currently Connected’ when the associated Connector is not connected
Fixed an intermittent error when clicking to expand a Connector rule details in the Connectivity View
Removed red exclamation mark icon next to "Unlinked Files" tab in the Samples view; improperly indicates an error
Fixed an issue where Bundle Terms of Use acceptance flag unset upon saving bundle version
Fixed an issue preventing users from viewing details of an Entitled Bundle
Fixed an issue resulting in an error message when creating Bundle Terms of Use with an existing version
API: Fixed an issue where the API endpoint to link data to a Bundle to responds with errors when data is part of a Reference Data
API: Fixed an issue causing 500 API error response when retrieving sample details
Fixed an issue causing incomplete list of users when adding users within the same tenant to the Project Team
API: Updated the API spec description for create upload URL and create inline view URL for Project Data
Fix to prevent editing of an existing active terms of use
Improved the Terms of Use UI flow for forcing re-acceptance upon update
API: Added a warning response for API to create custom events when event code exceeds 50 characters
API: Fixed the create data download URL API to correctly return a 503 instead of a 404 response when the underlying system is experiencing network-related issues
API: Added API endpoint to hide a Project
Fixed an issue with Boolean inputs incorrectly displayed as false on Analysis Details view (correct value is actually passed in)
Fixed an issue where CWL parameters containing only a white space are passed incorrectly to underlying pipeline runner
API: Fixed an issue where launch Nextflow analysis API returns a 500 error response when including duplicate parameters
Fixed an issue causing analysis failures due to docker image authorization check timeout
Fixed a shared storage issue causing analyses to remain In Progress until timeout
Fixed an intermittent issue causing Analyses to be stuck in progress with inability to view logs in the UI
Fixed an issue where Analysis output data is not immediately shown in the Analysis Details view upon completion
Fixed an issue causing Analysis failures showing “…Connection timed out (Connection timed out)…” in stdout logs
Fixed an issue causing CWL pipeline runner restarts during an Analysis which may lead to unexpected results
Fixed an issue resulting in stuck analyses when invalid Docker image is specified for a task; timeout reduced to 15 minutes
Fixed the name of Pipeline running logs tab from "workflow_runner" to "pipeline_runner" in Analysis Details view in the UI
Fixed missing start time on Analysis tasks and incorrectly reported costs for such tasks
Fixed an issue resulting in an error message when attempting to create a graphical CWL pipeline with a Tool with a Docker image that is not available in the Project's region
Fixed an issue causing errors when saving a new Bench Workspace
Fixed an issue causing out of sync Bench table and card views when a Workspace is created or deleted
Fixed an issue causing an error when starting Bench Workspace when enabling service account on the Workspace from tenant receiving the Workspace via cross-tenant share
Features and Enhancements
ICA Cohorts is released in production and added to the ICA user documentation
Clicking on a folder in the Data view will open the folder. Once a Folder is opened, the Folder details are available on the top right of the screen.
Added XML Configuration documentation and examples
General ICA robustness enhancements
Fixed Issues
Limit the Projects Overview view to 100 Projects with an indication that filtering must be used to see excluded Projects to improve UI performance
Fixed an issue preventing removal of permissions on externally managed projects
API – Fixed a 500 error response from Search Project Samples API when using a page token parameter
Fixed an issue allowing Bundles to be saved without mandatory Release Version when back-navigating from the Create New Bundle screen in the graphical UI
Fix to prevent data added to Bundle via secondary data upload on Data Details view
API – Fixed an error message when attempting to link data to a Bundle from a Project with Data Sharing disabled
API – Fixed an issue with tag filtering on getBundleSamples API
Fixed an error when filtering on size in Docker Repository where size filter is ignored
Fixed an issue where Analysis stuck in Aborting status when abort request sent during final stages of Analysis (such as Output, Upload)
Fixed an issue where intermittent Analysis failures are caused by system scaling errors under high load
Reduced repetitive Nextflow runner log entries from internal system logs to improve readability
API – Fixed an error in the Launch Nextflow Pipeline API when invalid parameter names are requested
API – Correctly support multi-value input parameters in launch Analysis APIs
Deprecated Tools are only visible when ticking the box "Show deprecated Tools" box in both table and card views
Fixed an issue preventing files linked to a Project from a Bundle as available to load into a Base Table
When exporting a query result from Base the file receives the name specified by the user and contains the header from the table
Removed unsupported AVRO and PARQUET formats from Available Formats to load data into Base tables
Fixed an issue where Bench service account API key is not revoked when a Workspace is stopped
Features and Enhancements
Added support for providing a nextflow.config file with a Nextflow Pipeline to improve pipeline building experience (see Nextflow Configuration docs)
Each tenant is limited to 100 in-progress Analyses at any given time to prevent resource exhaustion; additional analyses will be queued. Note: For assistance with planning batch jobs which require greater than 100 concurrent analyses, please contact Illumina Tech Support.
Added support for deep link URLs for direct navigation to child views in UI
API – Added an API to link Data to Projects in batch
API - Limit number of tags on data to 100; Excess tags in request will be ignored
API - Added "slug" attribute to Pipeline, Project, and Data asset types
API - Added support for the use of Project slug in place of ID in API request bodies
API - Added “Uniform Resource Name” (URN) attribute to Pipelines and Project object types
API - Added Project ID to notification payload for Pipeline Succeeded/Failed events
API - Added owning Project ID for data records in the getAnalysisOutput API response
API - Indicate originating Workflow/Workflow Session on Analyses returned via API, when applicable
Added support to use bundled Pipelines and data to create Analyses via API without first linking to the Project
Added support to automatically mount project data in Bench Workspace with ICA CLI
Fixed Issues
Renamed “Delete” to “Manage” in the Samples page dropdown menu
Removed the IGV button under View when selecting a VCF file
API – Fixed an issue causing the List Files API to incorrectly respond with an error when passing a folder or file path parameter value containing a leading or training space
Fixed an issue where files linked through a Project Connector will appear on root level as folder linking is not supported yet in the Project Connector
Fixed an issue when deleting files from S3 along with records from ICA via either volume delete (with purgeobjectstore=true) or folder delete API calls, if there are more than 1,000 items to delete the system deleted only the first 1,000 items, however, all the data records got deleted from ICA. In this case the data in the user-managed S3 bucket and ICA was out of sync.
Fixed an incorrect count displayed in UI when scheduling a download for a folder containing sub-folders
Fixed an issue where scheduling a download of a folder containing sub-folders creates duplicate data transfer operations
Disabled support for "fileNameExpression" on download rules; if provided when creating a download rule, the value will be ignored
Fixed an issue allowing public domain users to incorrectly access ICA
API – Fixed an issue where API to retrieve folder upload session intermittently responding incorrectly with error
API – Fixed a 500 response from the API to add a pipeline to a bundle when pipeline has already been added
Fixed an issue where user with "Viewer" permission for Flow in a project cannot link Samples from that Project
Return the correct error response when creating a new Nextflow or CWL pipeline through the API or CLI when an existing pipeline with the same name already exists
Improved UI messaging when setting up a Storage Configuration with permission error
Fixed an issue when attempting to edit a saved incomplete Tool (i.e., missing Docker image)
API - Fixed API to create notification channel to properly respond with error when attempting to create a duplicate notification channel
Fixed an issue where boolean Analysis input settings are incorrectly set to false when passed to the Pipeline runner
Fixed an issue when launching a Nextflow pipeline via API with multi-value inputs the order of the listed multi-value input is not preserved when passed to the pipeline runner
Fixed an issue where "Re-run" analysis button is broken for Nextflow pipeline analyses
When different versions of DRAGEN are run on the same host, reset the DRAGEN state so that the next task can correctly configure the FPGA compute instance
Fixed an issue causing intermittent analysis failures due to system error when evaluating permission to pull Docker image
Fixed empty start time attribute for Analysis tasks with a short duration (within a few seconds)
Fixed an issue where analysis start time is set when in fact the analysis is still queued; analysis start time will be set when compute resources (i.e., shared file system) have been requested
Fixed an issue where log file content for the pipeline runner is overwritten if system error occurs during an Analysis causing a pod restart; multiple log files may be created for the pipeline runner task if such an error occurs
Fixed an issue causing UI log stream view for Analysis tasks to incorrectly show errors or blank views
Fixed an issue causing incorrect Analysis task log to be displayed on the UI Logs view
Fixed an issue where analyses get stuck in Running status if input file paths contain leading or trailing spaces in path for analysis stuck in running status if input files contain leading or trailing spaces
Fixed an issue causing Analysis to be incorrectly marked as Success due to incorrect evaluation of exit code of an analysis task
Fixed an issue preventing the same file to be passed to multiple input fields when launching an Analysis
Workflow failures due to unavailable compute resources will fail quickly with a more descriptive error
Fixed an issue where analysis output data is not immediately shown in the Analysis Details view after the analysis completes
Removed the “Print” and “Save” buttons in the Analysis Report
Fixed an issue preventing Pipeline creation and Tool editing when an invalid Tool is saved in the Tool Repository
API – Fixed a 500 response from API to list Projects to properly return a 400 response when providing a search filter value with invalid characters
API - Fixed a 500 response from create download URL API to properly return a 400 response when data is not in "Available" status (ie, Partial/Archived)
Fixed an issue with loading data into a Base Table created directly in Snowflake with fields containing quotes
Fixed an issue where data was added to Base Table twice when multiple Scheduled Jobs run simultaneously
Fixed an issue with previewing Base Tables created directly via Snowflake
Fixed an issue where status is stuck at "In Progress" for Base Copy Table operations
Fixed an issue where Base Tables linked to a project from a bundle are shown as Views (read-only)
Fixed an intermittent issue causing duplicate data in Base Tables
API - Fixed a 500 response when sending a duplicate requests to load data for a Base Table to properly return a 400 response
API - Fixed a 500 response from API to load data into Base Table to correctly return a 400 response when attempting to load data with a type mismatch
API - Corrected Swagger UI responses
Fix error "Unable to save workspace" when attempting to start a Bench Workspace
Fixed Issues
Fixed an intermittent issue where analyses fail due to "Connection timed out" system error (seen in analysis error logs)
Added retry logic to decrease rate of pipeline failures due to internal system error (e.g. errors in pipeline runner log such as connection timeout, 404 pod not found, 500 errors)
Fixed an intermittent issue where "Cannot access folder" error is shown in pipeline error logs (most common in long-running analyses)
Fixed an issue resulting in analysis failures due to system error under high load
Fixed a compute instance configuration issue impacting DRAGEN analyses
Features and Enhancements
API - Added method to retrieve an analysis JSON input data
Added Analysis Details views in the UI
Added Analysis ID to the project analyses
Support deep links for direct navigation to Analysis Details view
Added support for multiple Nextflow versions when creating Nextflow pipeline
API - Added method to create multiple samples in batches
API - Improved response time performance of API that launches analysis with many data inputs
Fixed Issues
Repaired issue where Pipeline-generated Sample Metadata is not picked up by platform
Restored Time to delete function; now executes file deletes at the specified time
Bench example code now lists project data and pagination correctly
Restored ability to link secondary data to primary data
Fixed issue where batch sample creation jobs get stuck in PARTIALLY_SUCCEEDED status
Resolved issue where batch sample creation is not completed if data items could not be linked
API - Fixed issue where mountPaths were ignored when launching analyses via API
Restored function so that bundle contributors can link assets to their bundle
Restored data linking to a sample in an ICA project
Resolved issue where child workflow configurations prevent termination of parent workflow
Removed incorrect error reported in cases where previous failure occurred
Restored pipeline linking for bundle contributors
Improved UI messaging for Storage Configuration permission errors
API - Updated GET /api/projects/{projectId}/data to returns correct file details
Features and Enhancements
Re-labeled "Run" and "Runs" to “Analysis” and “Analyses” in the UI
Allow storage configuration to be disconnected from a hidden Project
Added support for a read-only view of Entitled Bundles and Bundles shared from another Tenant
Changed analysis timeouts for setting up environment, downloading inputs, pipeline runner (inclusive of all child steps), and finalizing output steps to 96 hours
API – Add idempotency key header to createAnalysis API
API – Analysis logs are available via getAnalysisSteps API; includes streamable via websocket endpoint during execution and downloadable S3 URL upon completion
JSON data with array field can be added in a Base table
Cohorts – Additional submenu items now available
CLI – Added support for setting the current Project ID to the value set in the ICA_PROJECT_UUID environment variable upon starting the CLI
CLI – Added support for mounting Project data to local operating system (Read-Only)
CLI – Added support for the outputParentFolderId flag for the ‘projectpripeline start’ command
CLI – Added support for customizing the filename/path of additional Pipeline files with the pipeline create commands
CLI – Added owning Project name/id attributes to the project data list output
CLI – Added owning Project name attribute to output for projectdata list command
CLI – Added commands to create and refresh API tokens (JWT)
CLI – Added support for CLI use without writing to disk by enabling API Key and Server flags for all commands
CLI – Added support for linking data to a Sample with projectdata upload command
Fixed Issues
Fixed an issue with UI stability leading to slow response times
Fixed an issue causing data uploads via browser to hang without error
Fixed an issue with UI automatically refreshing upon load when many Projects are present in the Projects view
Fixed an issue where Accept Share pop-up remains on screen after pressing to Accept the share
Fixed an issue with uploading nested folders using the Windows Connector
Fixed an issue where path separators are misinterpreted in upload rules for Windows Connectors.
Fixed an issue where modifying a local file uploaded (tracked) using the Connector causes a Connector failure
A User with Contributor Role for Flow and a Viewer Role for a Project can start an Analysis using input data from a Sample
API – Fixed an issue where sampleDeepDelete API fails with 500 error response when Sample contains a linked folder in archived status
Fixed an issue where ICA_STARTING_USER_TENANT_NAME is not set to the correct value on initial start
API – Fixed an issue where upload rules are not deleted via the deleteUploadRule API despite a success response
Sensitive fields in metadata models are now shown in italics
Fixed an issue where filtering on the Data view gives incorrect results
All Contributors in a Bundle are allowed to link assets to the Bundle
Content of a Bundle shared via an entitlement in another Tenant is now visible
Fixed an externally managed Project Team to only allow adding members with "Viewer" and "Allow Download" permissions
API – Fixed an issue with project sample APIs returning 500 error response when sample is not linked to the project specified in the request
API – Fixed issues with cursor-based paging on listData and listBundleData APIs returning expected results
Fixed an issue where a User can create empty folders inside an externally managed Project with Viewer permission
Fix auto-detection of AVRO and PARQUET data formats
API – Fixed an issue where the API to link Samples to a Bundle incorrectly returns a 204 response when the Sample is not in Available status and the Sample is not actually linked
API - Fixed an issue where changing the Project owner causes unexpected error responses from the listProjects, listProjectPermissions, and getProjectPermission APIs
API – Fixed an issue where deleteDownloadRule API returns 204 responses but does not delete the specified download rule
Fixed an issue causing delays in operations performed on data
Fixed an intermittent permission error when writing to /scratch from a Pipeline
Fix to make tags optional (not required) with Pipeline launch requests
Fixed an issue causing Analyses to be incorrectly set to SUCCEEDED status when failure occurs
API – Possible to create a Pipeline with a metadata model via API
Fixed an issue where logs are not streamed during Analysis
Fixed an issue when a User tries to start an Analysis with missing required parameters, the analysis is stuck in "awaiting input"
Fixed an issue preventing renaming a pipeline after attempting to save and receiving error that a pipeline exists with the same name
API – Fixed an issue when a Pipeline is launched via API with no input value provided for a nonmandatory input parameter the analysis status remains in "Awaiting Input" indefinitely
API – Fixed an issue causing unpredictable error responses from createAnalysis API when many Analyses are launched in rapid succession
API – Added operationID to API spec for the listAnalysisStorages and getAnalysisRawOutput APIs
Fixed an issue when Analyses are queued and Aborted before they are Running, the status is not correctly updated in the API/UI, and they appear stuck in “Aborting”
API – Fixed an issue with createAnalysis API causing error responses when launching many Analyses in rapid succession
Possible to load JSON in Base with a repeated field (array)
Tables created by the meta and admin data Schedules are updated after a sync
Fixed an issue causing Base Table field names to be changed from lowercase to UPPERCASE during a manual sync
Fixed an intermittent issue with Base Administrative/Meta-data Schedules creating files in project Data when schedules are run
Fixed an intermittent issue with Base Administrative/Meta-data Schedules creating files in project Data when scheduled
Fixed an issue where the User receives an obscure error message when creating a Tool from a Bench Workspace with an image name that already exists
Fixed an error message upon attempting to save Bench Workspace when input values are invalid
Fixed an issue where the Bench Workspace is temporarily inaccessible and stuck in Starting status
CLI – Fixed an issue with list samples command output limited to 20 unique records
Features and Enhancements
The value of the "Show Hidden Projects" filter on the Project Overview page will be saved for each user
Change analysis task log names to be more human-readable
Analyses will queue when system load is high
Add owning project tenant name (ICA_PROJECT_TENANT_NAME) and the tenant name of the user which last started the Workspace (ICA_STARTING_USER_TENANT_NAME) environment variables to Bench Workspaces
Add Cohorts URL environment variable to Bench Workspaces
API - Add support for Cross Origin Resource Sharing (CORS) to the API
Fixed Issues
Fixed an error message in UI when attempting to link a Sample that cannot be linked
API - Fix issue where API to update Sample metadata fields does not update the fields
Fixed error message when creating a Bundle with a name conflict
Fixed error message when a user attempts to view Bundle details without permission
API - Fix Project Bundle APIs to properly link, retrieve, and unlink entitled Bundles
Fixed to block ability to link data with Contributor role and no upload permission on the target project
Fixed an issue where the Owner of an externally managed Project cannot add Team members with Viewer or Download permission
Fixed field for providing an e-mail address when adding a Team member to a Project to be required
Removed temporary directory creation by service connector
Fix for configuring service connector to use proxy settings on Linux clients
Fixed an issue where data downloaded through Service Connector is incomplete
Fixed an issue causing errors in Service Connector when a directory is deleted in the UI but still present on the local machine. Data added to the folder on the local machine will not be uploaded if the corresponding folder is deleted in the UI.
Fix to display an error message when saving a Pipeline with invalid fields
Fix issue where CWL file/directory inputs are not mounted when provided within nested structures (ie, CommandInputRecordSchema, CommandInputArraySchema)
Fix to show error when attempting to run a graphical CWL pipeline with an invalid definition
Fix error when attempting to save a graphical CWL Pipeline with a Tool with an output folder and no specified format
API - Fix to ignore input JSON field when launching graphical CWL pipelines via API
API - Fix ignored default compute type (machine profile) defined in the Pipeline definition when launching a graphical CWL Pipeline via API
Fixed Base Query result export file format set to UNKNOWN
Fixed Base activity log to show Table associated with log event
API - Ignore unknown attributes in request bodies
API - Fixed API error responses formatted as HTML instead of JSON in error scenarios
API - Fix list analyses API not returning the "items[].analysisStorage" field
Features and Enhancements
Samples may only be linked when status is Available
Added Data ID to grid view in UI
Added ability to select and copy tag values to the clipboard
Added an indication of source domain in list of reference data sets when adding new data to Reference Data
Updated the service connector client icon
Added ability to filter projects by workgroup, including projects where user is a member of a workgroup added to the project
Added ability to specify output directory within a project when launching a pipeline
Added ability to Favorite Projects and filter the Projects view by Favorites
Remove obsolete commented text from auto-generated CWL Tool definitions
Scheduler displays the number of files to be loaded into a Base Table
Added an indication that Base Table previews display up to 50 columns and 100 rows
API - New endpoint to manage SecondaryData on Data objects
API - Added support for "mounts" parameter when launching a pipeline to provide input mount paths for file and folder inputs
API - Added fields to API error response schema: HTTP Method, User ID, Tenant Name, Tenant ID
API – Added new endpoints for retrieving Entitled Bundles and to manage Project Bundles
Fixed Issues
Docker images available via entitled bundles are listed in green text in the Docker Repository view
Fixed an issue in OpenAPI spec causing invalid cURL requests in 'Try it Out' functionality for the create upload URL API
Fixed an issue that resulted in empty Workgroup listing via the API and when adding Workgroups to a Project Team
Fixed an intermittent error when navigating to Bundles view
Fixed an issue where Tools available via Bundles are not displayed in the Tool Repository
Fixed an issue where example JSON generation for root Metadata Model fails when the root Metadata Model is modified
Fixed an issue where incorrect reporting of compute start times resulted in incorrect compute billing
Fixed an issue where Nextflow analyses get stuck "In Progress" when output folder is deleted during an ongoing pipeline run
Fixed an issue where incorrect casing on the scatter method 'dotproduct' in CWL auto-generated from graphical pipelines results in failures in pipelines with scatter-gather steps
CLI - Fixed an issue with startAnalysis APIs no longer accepting the 'Accept' header with value "application/problem+json,application/vnd.illumina.v3+json”
API/CLI - Fixed issue where graphical CWL pipelines launched via API/CLI do not use machine profiles (compute types) specified in the pipeline definition
Fixed an issue where pipelines within shared bundles could not execute across regions
Fixed an issue where exported Base Tables with null values populate a CSV with 'N' values instead of empty values
Fixed an error when loading data into Base Tables with an invalid number of columns
Fixed an issue causing View Results in Base Query History to return an expiration error
Fixed an issue that caused JSON import into Base Tables to fail when sub-field (record type) is used in the Table schema
Updated the error message when invalid SQL is used in a Base Query
Fixed an issue with JSON import to Base Tables failing when reference data fields are enabled
Fixed an issue where updated Base Table fields in the UI do not update the underlying Snowflake Table
Fixed an issue resulting in "Unable to create file error" message when creating a Tool in a Bench Workspace
Fixed an issue where Bench Workspaces cannot be deleted
Fixed an intermittent issue where Bench Workspaces unexpectedly fail in an error state
Updated messaging for users without access to Variant Viewer and Cohorts
API - Fixed an issue where the API did not accept multiple 'Accept' header values
API - Fixed an issue where the create folder upload sessions API returns date and time with incorrect formatting
API - Fixed the 500 response on APIs when the user has not yet accessed the UI
API - Fixed missing version header in OpenAPI spec for APIs to link a pipeline to a Project or Bundle
API - Fixed the 500 response from create project API when the 'storageConfigurationSubfolder' value is invalid
API - Fixed a 500 response from the API to link data to a bundle when linking a Folder with child data records
API - Fixed an issue where the API to create File data record creates the data record at the root of the project data when a trailing forward slash ('/') is missing after the folder name
API - Fixed an issue where API to retrieve pipeline reference sets does not display linked reference data
API - Fixed an issue where APIs to initiate analysis perform validation of required properties in request body
Fixed Issues
Fix issue where pipeline launch is blocked when a single file is used as the input to multiple pipeline inputs
Features and Enhancements
A Base query is now possible with null marker 'nan'
A user can search on Secondary Data tab
A user is allowed to run a pipeline from a shared project if the Billing mode is set as 'Tenant'
Add "application" field to Project API model to indicate whether a project is externally managed.
An error is shown when a user tries to build a workspace that is not running.
An error is thrown when the user uses the same name of an existing project connector rule in the same project
Base query can be exported as a view
Bug fix for allowing multiple files with same name but different paths to be passed as input; causes conflict when downloading inputs to pipeline filesystem, one file will be overwritten (random) causing unpredictable results
Data filters and columns now also apply to technical tags
Externally managed projects are now supported via API
Meta data schedule can be created on tenant level
New ICA notification for Data/Sample linked
Preview of the content of a Base table is available with null marker nan
Project connector permissions are aligned with manual linking permissions
Remove obsolete "BlueVantage" option from the Source drop down selections during Base schedule creation
Revert cwltool upgrade
Samples linked via a project connector are visible in the 'receiving' project
The 'Type of Files' field in the configuration of Upload Rules in a Connector is changed to 'Assign Format' so that its purpose is clear
The project connector rules are available across tenants when the user has Contributor rights
The user is able to use the filters 'Type' and 'Status' and sort the files
The yaks-scheduler used to check the pod-to-node fit, but will now defer to the kubernetes default-scheduler as the pod-placement authority. The assumption is that when the pod is not scheduled, there are no nodes available, and a node will be provisioned.
When a user links a sample from another project to the current project, the data will not be automatically linked to the current project as well. The user should explicitly add the data to the current project as well if the user wants to use it.
[Base] When selecting 'Include reference', you will see the record first and when you unfold the record, you will see the fields belonging to the record
When trying to populate the Sample Metadata, we now always use the Project Metadata Model if there is no Pipeline Metadata Model.
[API] Add ability to filter by region for list projects API
[API] Add psToken authorization type to enable creating JWT from psToken via the create token API
[Bench] Add ICA_PROJECT_UUID environment variable to the Jupyter Notebook Bench image.
Fixed Issues
Fix 400 error response when launching an analysis with a pipeline from a bundle
Fix Base Query results file export set to format UNKNOWN
Fix Bench pricing calculation accuracy
Fix for Base table export unexpected failure
Fix for auto-linking pipeline outputs files to a sample when input files provided are linked to the sample
Fix for failed Base table import when imported data contains "nan" (null marker) for a column of type integer
Fix for linking Base Query templates to Bundles
Fix for modifying Bench Workspace image unless Workspace is in Stopped state
Fix for more informative API error response when EULA has not been accepted
Fix for preserving sample data links for samples linked from shared bundles
Fix for retrieving download URL for data in an externally managed project.
Fix for silent error when running Base schedule on Administrative Data or Metadata
Fix intermittent UI issue where checkbox for file selection during docker image add is partially visible
Fix issue where Base schedule for Metadata does not work when source is set to "Tenant"
Fix issue where Base table schema definition changes from record to string after sync
Fix issue where Project Connectors cannot link samples where the source project is owned by a different tenant
Fix issue where an analysis cannot be started when a metadata model is defined for the pipeline
Fix issue where data cannot be removed from a Sample using the "Remove Data" button from the sample details view.
Fix issue where pipelines from linked bundles do not show as available selections in the "Generated by Pipelines" drop down when creating a Base schedule
Fix issue where samples cannot be updated after configuring a Project Connector rule with the sample.
Fix issue where scroll bar is not shown in the metadata models list view
Fix issue where secondary data is not shown in the reference data view
Fix issue where team members added to eternally managed projects cannot be granted Contributor permissions, and cannot link data to other projects
Fix issue where user cannot use sample filtering when samples and files are manually linked to their project and the files do not appear associated with their sample in that project
Fix issue where user is unable to see their input files when using sample filtering if they are linked from another project without selecting the flat view option
Fix issue where workgroups cannot be added to project team of externally managed projects with Contributor permission
Handle K8s api timeouts by retrying query.
Improve pipeline analysis startup times to approximately 10 mins maximum (times may vary)
[API] Fix 500 response for getProjectDataLinkedProjects API
[API] Fix error response when invalid API Key is provided for authorization
[API] Fix get bundles API error response when user has access to a project with a linked bundle that the user does not have access to
[API] Fix getProjectData API to return correct results when using the "id" query param
[API] Fix incorrect error response for Project Data API to create folder upload session
Features and Enhancements
the cwl workflow engine logs now also show debug logs, these includes the javascript expressions evaluated etc. these logs are similar to running cwltool --debug locally
Show "username" of pipeline creator in Pipelines overview view
Removed excessive logging from the "Audit Trail" logging.
Various performance improvements - reuse of connections to Base instances
A particular storage configuration should only be available during project creation in case it is your own storage configuration, or when the owner (or a tenant administrator) gives explicit access to it for other users in their tenant.
Bench - when trying to get an access token through the API, access was denied in case Base was not enabled within the project.
Base - 'You are not allowed to modify data' was shown to the user when clicking the Sync button as a Contributor
Base - importing data with format JSON in a table was not possible.
Files within a sample will only show up in case the data is also explicitly linked to the project/bundle.
Bundles - a user with Contributor access did not have the necessary access rights for managing all resource of the bundle.
Project billing modes can be 'project' or 'tenant'. In case 'tenant', the users from another tenant than the owner of the project was not able to upload files.
Within the Activation Codes view, activation codes for access to the Variants/Cohorts modules were not listed.
It was not possible to add a bundle at project creation. The project had to be saved before the bundle could be added.
The roles explanation in the Team page on a permission record did not show the entire information.
Various stability improvements - In case of a database outage, ICA was not able to automatically reconnect.
When managing docker images, a partial docker image was not allowed to be deleted nor updated resulting in a stuck image.
When exporting the contents of a grid from the UI, the exported file contained tenant ids instead of tenant names.
Base - the export of query results to a json format resulted in a failure in case the query ended with ';'
A non-tenant-admin user was not able to create a metadata model.
Base - exporting query results, specifying a custom delimiter did not work.
When a user clicks the bundle card in the "Bundles" view, and the user has no access to manage the bundle because it is shared with the user by another tenant, the following notification will be displayed: "Details for a bundle from another tenant are not accessible" instead of "Bundle details unavailable"
When a user links a sample from another project to the current project, the data will not be automatically linked to the current project as well. The user should explicitly add the data to the current project as well if the user wants to use it.
Various performance improvements - linking a folder with a lot files to another project or bundle timed out.
Base - loading metadata into a table using a schedule resulted in the same data being added over and over again.
When trying to populate the Sample Metadata, we now always use the Project Metadata Model if there is no Pipeline Metadata Model.
Limit usable character set for the user reference of an analysis run
When an analysis was aborted, the output files which were generated so far were not visible in the details of the analysis, but only in the data view.
Base - schedule metadata population in a table did not work.
Added support for Docker Images to be added to Bundles
Added Project Connectors in the project connectivity section to enable creation of automatic sample and data linking rules between projects.
Added Queue Date to analysis Logs view to indicate when the specified job was submitted to the underlying infrastructure.
Added support for externally-managed projects for Illumina software applications built on top of ICA to manage resources inside ICA projects while limiting tenant users' direct access to the project to read-only capacity.
Improved performance of API calls when using API Key for authentication
Added tenantName to API resource DTOs
Add API to abort analysis run
Added APIs to share a storageConfiguration, share a storageCredential, update a storageCredential secret
Added mime-type field to the DataFormat DTO for the data APIs
Added "objectEtag" field to DataDto for Data APIs to expose e-tag from the cloud provider object store
Added Base APIs to retrieve Jobs, retrieve Tables, and load data into Table
API - POST‚/api/projects/{projectId}/samples:search was missing request parameters to browse through samples in a paged manner.
API - POST /api/projects/{projectId}/analysis:cwl - tags were not persisted when starting a CWL analysis run
The PUT /project/{projectid}/data/{dataid} gives the user the impression that all fields can be updated on the data, but it is a different set for files as for folders. This distinction is not properly documented.
API - GET /api/projects/{projectId}/analyses - when the pipeline did not belong to the tenant, but the user still has access to it (eg: through a bundle), the analysis details could not be retrieved through the API.
API - GET /api/projects/{projectId}/analyses/{analysisId} returned the details of an analysis, even if the projectid did not match the actual project the analysis ran in.
API - GET /projects/{projectId}/samples/{sampleId} did not expose metadata information
API - GET /projects/{projectsId}/data presented the user with an unclear error message in case providing invalid request parameters.
API - An unclear error was presented in the problem response in case an invalid json was provided as request body.
API - An unlcear error was presented in the problem response when creating a project using invalid uuids in the request body.
API - Removed deprecated "sample matching strategy" endpoint
CLI - Added command to update project data
CLI - Added command to retrieve a list of project samples
CLI - Added command to create a project sample
CLI - Added command to list available data formats
CLI - API Key encryption modified to use local value on machine where API Key is configured to prevent hijacking config.
CLI - Added command to get details of a project sample
CLI - Added command to list data linked to a project sample
CLI - Added command to update a project analysis
CLI - Added command to update a project sample
CLI - Added command to unlink data from a sample
CLI - Added commands to delete a project sample
CLI - Added command to link data to a sample
CLI - Added command to complete a sample
CLI - Removed unnecessary flags on the config command.
CLI - For starting CWL and Nextflow pipelines, the flag 'storage-size' was ignored by CLI, and the default storage size of the pipeline was taken.
Fixed Issues
[bugfix] when CWL workflows failed or produced null as one of the outputs, all the outputs were incorrectly processed. in this case, the user may not be able to see the details of the analysis output in UI/API.
[Base] Fix error message when viewing a query from query history that no longer returns results
[bugfix] in some cases after aborting an analysis, it may remain stuck in aborting state indefinitely. Now more resources are explicitly cleaned to avoid this issue from happening.
Fixed bug when showing logfiles of steps in an analysis.
[bugfix] cwl workflow sometimes fails in spite of all steps in the workflows have succeeded. Made the workflow engine robust to detect and retry in this case
When linking a folder containing +2000 files to another project, only 2000 files were actually linked.
When a user did not choose a country when filling out the user account details, login to ICA could fail.
Fixed bug when handling custom output objects in CWL.
Fixes issue with compute type selection when launching CWL graphical pipelines
The Project > Team permissions showed a little pencil in front of the row indicating a modification. With this fix, the pencil will only show when there were actual modifications done to the row in particular.
API - All dates in API responses were formatted in 12 hours notation without AM/PM notation. This way a time in PM is presented as AM. With this fix, all time notations are in 24 hours.
When starting an analysis through the sample page, the start window is not closed after pressing start. Users would click it again, which would start the analysis multiple times. With this fix, the start window is closed after pressing start.
When starting an analysis fails because the backend service returns an error, the status of the analysis is "Awaiting Input". With this fix, the analysis goes to "Failed".
Base - in case the export of query results failed, an empty file was created and never cleaned up. With this fix, there are no leftovers in case of export failure.
When a sample was linked from another project to the current project, the details view of the sample shows all data linked to it, even of the current project does not have access to it. With this fix, only the data which is also linked to the current project will be shown on the details of the sample.
API - PUT /project/{projectid}/data/{dataid} update sometimes had to be done twice for it to persist. With this fix, all editable attributes can be updated at once.
Features and Enhancements
[CLI] Storage size can be entered by name
[CLI] Details of a project analysis can be retrieved by entering the analysis Id
[CLI] A list of metadata models can be retrieved
[CLI] A list of regions can be retrieved
[CLI] A list of storage configurations can be retrieved
[CLI] A pipeline can be linked to a project by using the name/id of the pipeline as an argument
[CLI] A pipeline can be unlinked from a project by defining the name/id as an argument
[CLI] A list of storage bundles can be retrieved
[API] Usability improvements for the list project data API:
Modify filePath query param to behave as prefix instead of exact match by default
Add filePathMatchMode query param
Add parentFolderPath query param
[API] Add API endpoint to refresh a JWT (POST /api/tokens:refresh)
[CLI] Add CLI command to create projects
[API] Support search by tags on samples search API
Renew macOS connector certificate
When a filter is applied and you select a folder, you can see the content from that folder without th* Connector rules for a project can only be changed when you are active in that project
It is possible to link more than 2000 files to a bundle or another project
Data coming from a project in another tenant can be linked when you are invite for collaboration
The error from a failed pipeline analysis is visible on the details page
Species and type are unique within a tenant, not across tenants
Sqs and sns test messages are aligned with the email test message
A collaborating user from another tenant is able to see the details view of a pipeline analysis run.
Bundle data can be used via API
When executing api/projects/{projectId}/data and a file already exist, there is an indication that thresponse.
Connector cancel button is renamed to delete to make its functionality more clearly
[CLI] Modify pipeline start command to use pipeline name or ID as an argument instead of a flag parameter
[API] Removed endpoint to POST on a sample metadata model
Bugs Fixed
[Base] Fix error message when viewing a query from query history that no longer returns results
[CLI] Fix for CLI upload command when uploading a folder with 0 bytes files
[API] Fix create bundle API to return error when bundle with existing name is attempted to be created
[API] Fix for non-functional If-Match header on PUT requests
[CLI] Fix folder upload "panic: close of closed channel" error
[API] Fix API for project creation returning success with invalid storage configuration provided
[Bench] Fix for docker build logs not showing when build is interrupted
Fix population of sample metadata from multiple analyses on the same sample
Fix error message for missing docker image on tool creation
[CLI] Fix pipeline create command to make pipeline code an argument instead of a flag parameter
Fix UI element for tags to remove active button appearance
Fix error when attempting to link pipeline with no pipeline selected
Fix error message when attempting to add field to Base table schema with invalid characters
Fix issue with analysis outputs files and logs not showing after analysis completion
Fix UI issue with pipeline selection in the Generated by Pipelines field of the view to create a Base s
Fix issue with a duplicate workspace being created while editing a workspace
Fix issue for table creation using schedule for metadata
Fix Base viewer role when applied to workgroups
Fixed Issue: optional parameters for Nextflow pipeline cause error when value not provided
Fixed issue when selecting data records to export from the Data overview page on a Bundle
Fix for Base saved queries not visible in shared project
Fix minor price calculation error (rounding) for Base storage
Features and Enhancements
API cursor-based and offset-set paging
Authorize calls using Illumina API key
Multiple reference data elements are possible when starting an analysis
APIs added for:
Account, User, Workgroup
Connector, Download Rule, Upload Rule
Data, Data Format, Data Transfer
Region
Download
Entitlement
Event Log
Project, Project Analysis, Project Data, Project Permission, Project Sample, Project Pipeline
Metadata model
Pipeline
Bundle
Notifications, Event Codes
Base
OpenAPI v3 API Spec published at https://ica.illumina.com/ica/api/swagger/index.html
Added sort parameter to several APIs
Custom event payload is defined as a json
Separate call to retrieve metadata model fields available
Add rclone temporary credentials support in the API to retrieve folder upload session details
Tokens API POST also contains basic authentication
Activation code endpoints are aligned with start analysis endpoints
Base templates cannot be deleted
Information can be written from Bench to Base
Viewer/Contributor permissions are introduced in Snowflake
Saved Template details can be edited
Access via Illumina authentication is available
Snowflake access token can be created
Added new Bundles feature for sharing data with other users
ICA allows the user to connect an S3 bucket as storage instead of an Illumina managed storage
An overview of all storage configurations is available
Secrets can be managed for the storage configuration
A new storage configuration can be added
The health status of a configuration can be monitored
A storage configuration can be deleted when not used in a project
Storage type and health are displayed in the overview
When selecting own storage ICA will return you a list of all BYOBs in the chosen region
A default bucket can be selected for every region
Data is automatically archived/deleted when the grace period (if present) is over
Data is archived/deleted automatically by a housekeeping mechanism
An admin user can change the date to be archived/deleted at any time
Data has a status
Data format is automatically detected at creation
Files can be grouped in a sample
Data can have additional linked projects
The data screen shows project data and data linked via a bundle
3 new columns are present in the data grid: "Owning Project", "Linked Projects" and "Linked Bundles"
The record is displayed in the data grid depending on the data being linked or not
Filter on Status with checkboxes on status values is possible
Total Data Size is available
Data can be linked/unlinked to/from a project
Only root parent files and folders are shown when opening the data grid
Deleted status does not exist anymore
Data from projects with Data Sharing disabled cannot be linked
Project is hidden instead of (soft-)delete
Metadata models can be created for linking to a project
Add support for Nextflow pipeline language
A pipeline input can have multiple formats
Output JSON for CWL analysis is shown
Ongoing logfiles are available for the user
You have two options to define a CWL pipeline: Graphical editor or code
Test connection button is available for SQS
Checkbox is added to toggle view between all rules or only your own rules
Notification channels can be validated
Subscription can be edited
Custom subscriptions are possible
Notifications can be received via SQS, SNS, Mail, and Webhook
All users have an overview of all the notifications used in their tenant
Notification channels are only visible for tenant admins
Notifications can have a filter expression to only receive specific events
A project administrator can grant permission to a single user on a project
A project administrator can invite somebody unknown to ICA to his project
A contributor can manage reference data
A project administrator can define who pays for using my project
A project administrator can invite a workgroup to a project
When multiple project roles are given to a user/workgroup the correct project role is applied
Team-page in projects and bundles are aligned
Reference data can exist in multiple regions
Species and type are being populated per tenant
Data can become reference data
Data tree shows all the regions in which the reference data is available =======
Features and Enhancements
General
The CLI readme file will now additionally contain the CLI build number
Data Management
Fixed an issue where there was a discrepancy between the Run Input tags shown to the user and what was stored on the data
Added a 25,000-item limit to the v3 endpoint for batch data linking. Using the v4 endpoint, which does not have this limitation, is recommended
Flow
Analyses and workflow sessions can now be resubmitted, and parameters can be updated upon resubmission
Changed the default image used for CWL pipeline processes with undefined image from docker.io/bash:5 to public.ecr.aws/docker/library/bash:5
Updated the choice of default nextflow docker image which is used when no docker image is defined. It is now public.ecr.aws/lts/ubuntu:22.04_stable
The analysis logs in the analysis details page can be refreshed
The user is now able to write a pipeline which executes files located in the /data/bin directory of the runtime environment
Pipeline files are now shown in a tree structure for easier overview
Cohorts
Updated GWAS UK Biobank data base gives users access to more phenotype information
Users can now incrementally ingest their molecular data for germline, CNV, structural variants, and somatic mutation data
Fixed Issues
General
Added an "All" option to the workgroup selection box in the projects view to reset the filter, which previously required you to delete all characters from the filter
Fixed an issue where updating two base permissions at the same time would sometimes not execute correctly
Fixed an issue where creating grid filters could result in a nullpointer error
Fixed an issue where 'Copy to Clipboard' button did not work anymore
After searching for a folder in the search box and going into that folder, the search box is now cleared
Improved the project permissions API to correctly handle empty values
Previously, when attempting to save and send a message from the Websolutions section without a unique subject, the system would report an error and still send the message. Now the non-unique message subject error is reported and no message is sent
Fixed an issue where linking samples in the sample screen would result in receiving the same "sample(s) linked" message twice
Improved error handling for CLI FUSE driver
Hardened log streaming for ongoing runs to better handle network issues which previously would result in missing log streaming
Add retries for "connected reset by peer" network-related errors during analysis upload tasks
Fixed an issue where inviting a user to collaborate on a project containing base would result in the error "entity not managed" if that user did not have base enabled in any project or if base was not enabled in the project tenant
Data Management
Fixed an issue where data could be moved to a restricted location called /analyses/ and no longer be visible after the move. Please contact Illumina Support with your data move job information to recover your data if you have encountered this issue
Fixed an issue where sorting on data format did not work correctly
Copying empty folders no longer results in a partially copied status
ICA now performs an automatic refresh after unlinking or deleting a sample
Improved handling of file path collisions when handling linked projects during data copy / move
Fixed an issue where, even though uploading a file in a linked folder is not permitted, this would erroneously present a success message without copying the file
Analysis-events which are too large for SQS (256KB) are now truncated at the first 1000 characters when using SQS
Improved error handling when trying to upload files which no longer exist
Fix system degradation under load by introducing rate limit for spawning tasks for a given analysis to 25 per 1 min
The createUploadUrl endpoint can now be used to upload a file directly from the region where it is located. The user can create both new files and overwrite files in status "partial"
Improved the project data list command with wildcard support. For example:
/ or /* will return the contents of the root
/folder/ will return the folder
/folder/* will return the contents of the folder
To optimize performance, a limit has been set to prevent concurrent uploading of more than 100 files
Fixed an issue where folder syncing functionality would sometimes result in “Unhandled exception in scheduling worker”
Flow
Fixed an issue where writing a pipeline which executes files in the /data/bin folder wasn't functioning properly with larger storage classes
Nextflow pipelines no longer require pipeline inputs when starting them via the CLI
Improved error handling when using an unsupported data format in the XML input forms during pipeline creation
Fixed the issue where it was not possible to add links in the detail page for pipelines and bundles
Sorting is no longer supported on duration and average duration columns for analysis grids
In situations where the user would previously get the error "zero choices with weight >= 1" after the first attempt, additional retries will execute to prevent this from occurring
Cohorts
Fixed an issue resulting in a blank error when a cohort with hundreds of diagnostic concepts was created
Features and Enhancements
Flow
Improved analysis queue times during periods of limited compute resource availability
Features and Enhancements
General
New notification to the user when a copy job finishes running
Updated the "GET analysis storage" API endpoint to account for the billing mode of the project. If the billing mode of the project is set to tenant, then the analysis storage of the user's tenant will be returned. If the billing mode of the project is set to project, then the analysis storage of the project's owner tenant will be returned
A ReadMe file containing used libraries and licenses is now available for ICA CLI
Data Management
New DataFormats YAML (.yaml, .cwl), JAVASCRIPT (.js, .javascript), GROOVY (.groovy, .nf), DIFF (.diff), SH (.sh), SQL (.sql) to determine the used syntax highlighting when displaying/editing new pipeline files in the UI
ICAv2 CLI supports moving data both between and within projects
Added an alert to notify users when data sharing or data move is disabled for the project
A new version of the Event Log endpoint has been developed to support paging, retrieval of previous events, and resolution of inconsistencies in date formats. This new endpoint introduces the EventLogListV4 data transfer object
The user is now able to select a single file and download it from the browser directly. This does not apply for folders and multiple files selected at once
User can subscribe to notifications when data is unarchived
The BaseSpace Sequencing Run Experiment name will now be added to the technical tags when a workflow session is launched
Flow
Fastqs with the .ora extension are now supported when staging these for secondary analysis, either as a list of fastqs or as fastq_list_s3.csv files
Before, users had to click on the pipeline on the pipeline overview screen to start a new analysis. Now, you will enter the pipeline in edit mode when you click on the pipeline name. If you want to select a pipeline to start an analysis, you need to check the checkbox
Fixed Issues
General
Removed the refresh button from the workspace detail view as it was superfluous
Fixed an issue where searching for certain characters in the search field of the Projects or Data overviews screen would result in an indecipherable error
Improved security handling around tenant admin-level users in the context of data move
Data Management
Fixed a bug so folders copied from another previously copied folder no longer results in a corrupted file
Fixed an issue where creating a new bundle would result in an error if a project with the same name already exists
Data move between projects from different tenants is now supported
Fixed an issue where not selecting files before using the copy or move commands would result in EmptyDataId errors
For the CLI, Improved notifications when files can not be downloaded correctly
Fixed an issue where scheduled downloads of linked data would fail without warning
Corrected an issue where the tenant billing mode would be erroneously set to Illumina after a data copy
Fixed an issue where BatchCopy on linked data did not work
Flow
Resolved an issue to ensure that when a user creates a pipeline using a docker image shared from an entitled bundle, their analyses utilizing that pipeline can pull the docker image without errors
Removed superfluous options from the analysis status filter
Awaiting input
Pending request
Awaiting previous unit
Fixed an issue where writing a pipeline which executes files in the /data/bin folder wasn't functioning properly with larger storage classes
Fixed an issue where many-step analyses are getting stuck in "In Progress" status
Fixed an issue where the wrapper scripts when running a CommandLineTool in CWL would return a warning
Fixed the issue which caused the "Save as" option not to work when saving pipelines
Base
Fixed an issue where the ICA reference fields in the schema definition had the wrong casing. As a result of this update you might end up with 2 different versions of the reference data (one with keys written with an uppercase letter at the start, another one with keys written entirely in lowercase letters). To fix this:
Update your queries and use the Snowflake function : GET_IGNORE_CASE (ex: select GET_IGNORE_CASE( to_object(ica) , 'data_name' ) from testtableref)
Update the 'old' field names to the new ones (ex: update testtableref_orig set ica = object_delete(object_insert(ica, 'data_name', ica:Data_name), 'Data_name'))
Fixed an issue where using an expression to filter the "Base Job Success" event is not working
Fixed Issues
Flow
Resolved an issue to ensure that when a user creates a pipeline using a docker image shared from an entitled bundle, their analyses utilizing that pipeline can pull the docker image without errors.
Features and Enhancements
General
The left side navigation bar will collapse by default for screen smaller than 800 pixels. The user can expand it by hovering over it
The browser URL may be copied to share analyses, pipelines, samples, tools, workspaces and data in various contexts (project, bundle)
Data Management
Users are now able to move data within and across projects:
The user can:
Move available data
Move up to 1000 files and/or folders in 1 move operation
Retain links to entities (sample, sequencing run, etc.) and other meta-data (tags, app-info) when moving
Move data within a project if the user is a contributor
Move data across projects if (1) in the source project the user has download rights, has at least contributor rights, and data sharing is enabled, and (2) the user has upload rights and at least viewer rights in the target project
Move data across projects with different types of storage configurations (user-defined or default ICA-managed storage)
Select and move data to the folder they are currently in through the graphical UI
Select and move data in a destination project and/or folder through the API
The user cannot:
Move linked data. Only the source data can be moved
Move data to linked data. Can only move data to the source data location
Move data to a folder that is in the process of being moved
Move data which is in the first level of the destination folder
Move data to a destination folder which would create a naming conflict such as a file name duplicate
Move data across regions
New Event Log entries are provided when a user links (ICA_BASE_100) or unlinks (ICA_BASE_101) a Cohorts data set to a bundle
Added support for the following data formats: ora, adat, zarr, tiff and wsi
Flow
New compute types (Transfer Small, Transfer Medium, Transfer Large) are supported and can be used in upload and download tasks to significantly reduce overall analysis runtimes (and overall cost)
API: All the endpoints containing pipeline information now contain the status from the pipeline(s) as well
Bench
External Docker images will no longer display a status as they consistently showed 'Available,' even when the URL is not functional
Cohorts
Performance improvements to needle plot by refactoring its API endpoint to return only sample IDs
Users now click a cancel button that returns them to the landing page
Users can now perform time series analysis for a single patient view
Refresh of PrimateAI data now drives data in variant tables
Users can now access the structural variant tab in the Marker frequency section
Fixed Issues
General
Fixed an issue where, when a user is added to or removed from a workgroup, they could be stuck on an infinite redirect loop when logging in
Fixed syncing discrepancy issues about deleted files in user-managed storage projects with Lifecycle rules & Versioning
Data Access & Management
Sorting API responses for the endpoint GET /api/jobs is possible on the following criteria: timeCreated, timeStarted and timeFinished
Improved the error message when trying to link a bundle which is in a different region than the project
More documentation has been added to the GET /eventLog regarding the order of rows to fetch
Fixed an issue where the API call - POST api/projects/{projectId}/permissions would return an error when DATA_PROVIDER was set for roleProject
Fixed an issue stemming from attempts to copy files from the same source to the same destination, which incorrectly updated file statuses to Partial
CLI: Fixed an issue where the environment variable ICAV2_X_API_KEY did not work
Flow
The analysis is no longer started from the API if error 400 ( 'Content-Type' and 'Accept' do not match) occurs
Base
Fixed an issue where the Base schedule would not run automatically in some cases when files are present in the schedule
Bench
Improved error handling when trying to create a tool with insufficient permissions
Fixed an issue where the user is unable to download docker-image with adhoc-subscription
The "version":"string" field is now included in the API response GET /api/referenceSets. If no version is specified, the field is set to "Not Specified"
Fixed an issue where, under some conditions, fetching a job by id would throw an error if the job was in pending status
Features and Enhancements
Data Management
The GUI now has a limit of 100 characters for the name and 2048 characters for the URL for links in pipelines and bundles
Added a link to create a new connector if needed when scheduling a data download
Improved the data view with additional filtering in the side panel
Flow
New CLI environment variable ICA_NO_RETRY_RATE_LIMITING allows users to disable the retry mechanism. When it is set to "1”, no retries are performed. For any other value, http code 429 will result in 4 retry attempts after 0.5, 2, 10, and 30 seconds * Code-based pipelines will alphabetically order additional files next to the main.nf or workflow.cwl file
When the Compute Type is unspecified, it will be determined automatically based on CPU and Memory values using a "best fit" strategy to meet the minimum specified requirements
Bench
Paths can be whitelisted to allowed URLs on restricted settings
Fixed Issues
General
Fixed an issue where the online help button does not work upon clicking on it
Data Access & Management
Improved automatic resource cleanup when hiding a project
Fixed an issue with the service connector where leading blanks in the path of an upload/download rule would result in errors. It is no longer possible to define rules with leading or trailing blanks
Fixed an issue where a folder copy job fails if the source folder doesn't have metadata set
Linking data to sample has been made consistent between API and GUI
Improved resource handling when uploading large amounts of files via the GUI
Fixed an issue where the API endpoint to retrieve input parameters for a project pipeline linked to a bundle would fail when the user is not entitled on the bundle
Fixed an issue where deleting and adding a bundle to a project in one action does not work
Flow
The event sending protocol was rewritten to limit prematurely exhausting event retries and potentially leaving workflows stuck when experiencing high server loads or outages
Fixed an issue where specifying the minimum number of CPUs using coresMin in a CWL workflow would always result in the allocation of a standard-small instance, regardless of the coresMin value specified
Fixed an issue in the API endpoint to create a Nextflow analysis where tags were incorrectly marked as mandatory inputs in the request body
Fixed an issue with intermittent failures following completion of a workflow session
Base
Improved syntax highlighting in Base queries by making the different colors more distinguishable
Bench
Fixed an issue where the Bench workspace disk size cannot be adjusted when the workspace is stopped. Now, the adjusted size is reflected when the workspace is resumed
Fixed an issue where regions were not populating correctly for Docker images
Fixed an issue where API keys do not get cleaned up after failed workspace starts, leading to unusable workspaces once the API key limit is reached
Features and Enhancements
Cohorts
Users can now query variant lists with a large number of associated phenotypes
Users can now perform multiple concurrent data import jobs
Fixed Issues:
Cohorts
Fixed an issue with displaying shared views when refreshing a Bundle’s shared database in Base
Fixed Issues
Fixed an issue where autolaunch is broken for any users utilizing run and samplesheet inputs stored in BSSH and operating in a personal context, rather than a workgroup.
Features and Enhancements
Data Management
Data (files and folders) may be copied from one folder to another within the same Project
The empty ‘URN’ field in the Project details at Project creation is now removed
The ‘Linked Bundles’ area in the Project details at Project creation is now removed as you are only allowed to link Bundles after Project creation
The card or grid view selected will become the default view when navigating back to the Projects or Bundles views
Added a new API endpoints to retrieve and accept the Terms & Conditions of an entitled bundle:
/api/entitledbundles{entitledBundleId}/termsOfUse
/api/entitledbundles/{entitledBundleId}/termsOfUse/userAcceptance/currentUser
/api/entitledbundles/{entitledBundleId}/termsOfUse:accept
Flow
Added a new API endpoint to retrieve orchestrated analyses of a workflow session
GET /api/projects/{ProjectID}/workflowSessoins/{WorkflowSessionID}/analyses
Code-based pipelines will alphabetically order additional files next to the main.nf or workflow.cwl file
Bench
New JupyterLab - 1.0.19 image published for Bench using the Ubuntu 22.04 base image
Resources have been expanded to include more options for compute families when configuring a workspace. See ICA help documentation for more details
Cohorts
Sample count for an individual cohort may be viewed in the variants table
Filter the variants list table through the filter setting in the needle plot
Execute concurrent jobs from a single tenant
Improved the display of error and warning messages for import jobs
Structural variant tab may be accessed from the Marker frequency section
Fixed Issues
Data Access & Management
Bundles now reflect the correct status when they are released instead of the draft status
Double clicking a file opens the data details popup only once instead of multiple times
Improved performance to prevent timeouts during list generation which resulted in Error 500
The counter is now accurately updated when selecting refresh in the Projects view
Fixed an issue resulting in one job to succeed and one to fail when running two or more file copy jobs at the same time to copy files from same project to same destination folder
Fixed an issue resulting in an error in a sample when linking nested files with the same name
Added a new column to the Source Data tab of the Table view which indicates the upload status of the source data
Removed the unused ‘storage-bundle’ field from the Data details window
Fixed an issue where the Project menu does not update when navigating into a Project in Chrome browsers
(CLI) Fixed an issue where deleting a file/folder via path would result in an error on Windows CLI
Base
Improved schedule handling to prevent an issue where some files were not correctly picked up by the scheduler in exceptional circumstances
Fixed an issue where an incorrect owning tenant is set on a schedule when running it before saving
The number of returned results which is displayed on the scheduler when trying to load files now reflects the total number of files instead of the maximum number of files which could be displayed per page
Fixed an issue where Null Pointer Exception is observed when deleting Base within a Project
Bench
Fixed an issue where users were unable to delete their own Bench image(s) from the docker repository
Cohorts
Fixed an issue where the value in the tumor_or_normal field, in the phenotype table in database, would not set properly for germline and somatic mutation data
Fixed an issue where large genes with subjects containing large sets of diagnostic concepts caused a 503 error
Fixed Issues
Fixed an issue where automated analysis after sequencing run in non-US regions may fail for certain analysis configurations
Features and Enhancements
Data Management
The --exclude-source-path flag has been added to the ‘project data download’ command so that subfolders can be downloaded to the target path without including the parent path
The system automatically re-validates storage credentials updated in the graphical UI
Added a new API endpoint to validate storage configurations after credentials are changed: /api/storageConfigurations/{storageConfigurationId}:validate
Notifications
Added support for multi-version notification event payloads corresponding to versioned API response models
Flow
(API) Improved the analysis-dto by adding a new POST search endpoint as a replacement for the search analysis GET endpoint. The GET endpoint will keep working but we advise using the new POST endpoint.
Improved analysis statuses to reflect the actual status more accurately
Parallelized analysis input data downloads and output data uploads to reduce overall analysis time
No scratch size is allocated if tmpdirMin is not specified
Cohorts
Performance improvements of the ingestion pipeline
Performance improvements to subject list retrieval
Increased the character limit of ingestion log messages to the user
Fixed Issues
Data Access & Management
Fixed an issue where the target user cannot see analysis outputs after a successful transfer of analysis ownership in BaseSpace Sequence Hub
Update the API Swagger documentation to include paging information for: /projects/{projectId}/samples/{sampleId}/data
Fixed an issue resulting in errors when creating a new bundle version
Fixed an issue where the GET API call with the ‘Sort’ parameter returns an error when multiple values are separated by commas followed by a space
Fixed an issue where adding the –eligible-link flag to the ‘projectdata list’ API endpoint caused other flags to not work correctly
Added cursor-based pagination for the ‘projectdata list’ API endpoint
Fixed an issue with the entitled bundles cards view where the region is cut off when the Status is not present
Fixed an issue where bundle filtering on categories did not work as expected
Fixed an issue where file copy across tenants did not work as expected
Added a cross-account permission check so that file copy jobs fail when the cross-account set up is missing instead of being retried indefinitely
Fixed an issue where ‘Get Projects’ API endpoint returns an error when too many projects are in the tenant
Fixed an issue where the UpdateProject API call (PUT /api/projects/{projectId}) returns an error when technical tags are removed from the request
Fixed an issue where users need to confirm they want to cancel an action multiple times when clicking the back button in the graphical UI
Fixed an issue where clicking into a new version of a bundle from the details view does not open the new version, and instead directs to the bundle card view
Flow
Fixed an issue where the analysis logs are returned in the analysis screen “outputs” section and included in the getAnalysisOutputs API response. The log output is no longer considered as part of the analysis outputs
Analysis history screen has been removed
Fixed an issue resulting in inability to retrieve pipeline files via the API when the pipeline is shared cross-tenant
Fixed an issue where the API endpoint to retrieve files for a project pipeline would not return all files for pipelines created via CLI or API
Fixed an issue where the API does not check the proprietary flag of a pipeline before retrieving or downloading the pipeline files
Base
The ‘Download’ button is available to download Base activity data locally (and replaces the non-functional ‘Export’ button for restricted bundles)
Fixed an issue resulting in missing ICA reference fields in table records if the file was loaded into the table with no metadata
Improved consistency of the references included in the scheduler
Bench
Users are now logged out from a terminal window opened in a workspace after a period of inactivity
Fixed an issue where permissions could not be enabled after a workspaces has been created
Fixed an issue where a Contributor could not start/stop a workspace
Cohorts
Fixed an issue where large genes with subjects with large sets of diagnostic concepts cause a 503 error
Fixed an issue where the value in tumor_or_normal field in the phenotype table in the database is not set properly for germline and somatic mutation data
Resolved a discrepancy between the number of samples reported when hovering over the needle plot and the variant list
Features and Enhancements
Users can now access the system via https://ica.illumina.com or https://ica.illumina.com/ica
Data Management
Users are now able to revalidate storage configurations in an Error state
Improved existing endpoints and added new endpoints to link and unlink data to a bundle or a project in batch:
POST /api/projects/{projectId}/dataUnlinkingBatch
GET /api/projects/{projectId}/dataUnlinkingBatch/{batchId}
GET /api/projects/{projectId}/dataUnlinkingBatch/{batchId}/items
GET /api/projects/{projectId}/dataUnlinkingBatch/{batchId}/items/{itemId}
Flow
Analyses started via the API can now leverage data stored in BaseSpace Sequence Hub as input
ICA now supports auto-launching analysis pipelines upon sequencing run completion with run data stored in BaseSpace Sequence Hub (instead of ICA)
Updated the API for creating pipelines to include "proprietary" setting, which hides pipeline scripts and details from users who do not belong to the tenant which owns the pipeline and prevents pipeline cloning.
Cohorts
Added support for partial matches against a single input string to the “Search subjects” flexible filtering logic
Users can now view an overview page for a gene when they search for it or click on a gene in the marker frequency charts
ICA Cohorts includes access to both pathogenic and benign variants, which are plotted in the “Pathogenic variants” track underneath the needle plot
Ingestion: UI notifications and/or errors will be displayed in the event of partially completed ingestions
Users can share cohort comparisons with any other users with access to the same project
Fixed Issues
General
Improved the project card view in the UI
Fixed an issue with user administration where changing the permissions of multiple users at the same time would result in users receiving Invalid OAuth access token messages
Data Access & Management:
Improved the error message when downloading project data if the storage configuration is not ready for use
Fixed an issue causing Folder Copy jobs to time out and restart, resulting in delays in copy operations
Fixed an issue where only the Docker image of the first restricted bundle that was added could be selected
Improved the performance of folder linking with "api/projects/{ProjectID}/dataLinkingBatch"
The URL for links for "post/api/bundles" endpoint can be up to 2048 characters long
Improved the error response when using offset-based paging on API responses which contain too much data and require cursor-based paging
Fixed an issue resulting in failures downloading data from CLI using a path
The correct error message is displayed if the user does not have a valid subscription when creating a new project
Fixed an issue where changing ownership of a project does not change previous owner access for Base tables
Flow:
Input parameters of pipelines are now displayed in the "label (code)" format unless there is no label available or the label equals the code, in which case only the code is shown
Fixed an issue where multiple folders were created upon starting new analyses
Fixed an issue preventing analyses from using inputs with BaseSpace v1pre3 APIs
Fixed an issue causing analyses with a specified output path to incorrectly return an error stating that the data does not exist
The following endpoint "/api/projects/{projectId}/workflowSessions/{workflowSessionId}/inputs" now supports using external data as input
Any value other than "economy" or "standard" for submitted analysis jobs will default to "standard" and use "standard"
The parameter to pass an activationcode is now optional for start-analysis API endpoints
Base:
Improved the display of errors in the activity jobs screen if a Meta Data schedule fails
If an error occurs when processing metadata a failed job entry will be added in the Base Activity screen
Fixed an issue where records ingested via schedules from the same file could be duplicated
Fixed an issue where exporting the view shared via bundle would show an error 'Could not find data with ID (fol. ....)'
Resolved a NullPointerException error when clicking on Format and Status filters in the details screen of a Schedule in the Results tab
Fixed an issue where a schedule download would fail when performed by different user than the initial user
Bench:
Fixed an issue when trying to query a Base table with a high limit within a workspace
Fixed an apt-get error when building images due to an outdated repository
Fixed an issue where a stopped workspace would display "Workspace paused" instead of "Workspace stopped"
Fixed an issue where large files (e.g., 150GB+) could not be downloaded to a fuse-driver location from a Workspace, and set the new limit to 500GB
Cohorts:
Fixed an issue where split Nirvana JSON files are not recognized during ingestion
Fixed an issue causing the UI hangs on large genes and returns a 502 error
Fixed an issue where OMOP files are not correctly converted to CAM data model, preventing OMOP data ingestions
Fixed an issue where OMOP large drug ingestions led to memory issues and preventing further drug data ingestion
Fixed an issue where users from a different tenant accessing a shared project could not ingest data
Features and Enhancements
Flow:
Analysis logs (task stdout/stderr files) are now written to a folder named ‘ica_logs’ within the analysis output folder
Default scratch disk size attached to analysis steps reduced from 2TB to 0B to improve cost and performance of analyses. Pipelines created before ICA v2.21.0 will not be impacted
Notifications: Notifications can now be updated and deleted in externally managed Projects
API: Clarified on the Swagger page which sorting options apply to which paging strategy (cursor-based versus offset-based). Changed the default sorting behavior so that:
When no paging strategy is specified and no sort is requested, then cursor-based paging is default
When no paging strategy is specified and sort is requested, then offset-based paging is default
Cohorts
Procedure Search Box: Users can now access additional UI functionalities for Procedures
Users can now access Procedure codes from OMOP
Improved handling of drug codes across all reports, excluding Survival comparison
Ingestion
Users now have enhanced job warning log and API status improvements
Users now require download permissions to facilitate the data ingestion process
Fetch Molecular Files: Improved import – Users can now input a directory path and select sample files individually
Variant Type Summary: Users can now access a new variants tab that summarizes Variant type statistics per gene
Added sorting and filtering capabilities to report tables, such as variants observed in genes
Users can now view sample barcodes, replacing internal auto-increment sample IDs in the Structural Variants table within the Genes tab
“Search subjects” functionality improved with flexible filtering logic that now supports partial matches against a single input string
Fixed Issues
Data Management:
Fixed an issue with data copy via the CLI where the file was being copied to a subfolder of the intended location instead of the specified folder
Resolved an issue where browser upload hangs intermittently when creating data
Fixed an issue where the delete popup does not always disappear when deleting data
Fixed an issue where GetFolder API call returns 404 error if the Create and Get operations are performed 100ms apart
Fixed an issue where file copy would fail if the file was located at the root level of User’s S3 storage bucket
Fixed an issue causing data linked from externally managed projects to be incorrectly excluded from the list project data API response
Fixed an issue where User cannot use data URNs to identify the destination folder when interacting with copy data API endpoints
Bundles: Fixed an issue where clicking the back button before saving a new bundle leads to inconsistencies
Flow:
Fixed an issue where pipeline documentation is not scrollable when launching pipeline
Fixed an issue with logfiles of a task not being available for streaming while the task is still running
Fixed an issue where using the 're-run' button from the analysis page reverts the storage size selection to default
Fixed an inconsistency where the following two endpoints would show different analysis statuses:
GET /api/projects/{projectId}/analyses
GET /api/projects/{projectId}/analyses/{analysisId}
Improved performance issues with UI loading data records when selecting inputs for analysis
Fixed a caching issue which resulted in delays when running pipelines
Fixed an issue where back button for analysis or pipeline details does not always direct Users back to analysis or pipelines view, respectively
Fixed an issue where system performance is degraded when large batches (e.g., 1,000) of data are added as input to Analyses via the graphical UI. It is recommended to start Analyses with large numbers of input files via API
Base
Fixed an issue where enabling Base from a Base view other than Base Tables returned a warning message
Fixed an issue where Base access was not enabled when a bundle with tables is added to a project without Base (Base is automatically enabled so users can see the bundle's tables). However, access to the bundle's tables is revoked upon the deletion of Base, and was not granted again once Base was re-enabled
Fixed an issue where a Base job to load data into a table never finished because the file was deleted after the job started and before it finished. Now the job will end up in a Failed state
Cohorts
Fixed an issue where needle plot filtered out data points reappear when zooming in the exon when a filter is in place
Fixed an issue where users from a different tenant who accept a project share may encounter a failure at the final step of the data ingestion process
Fixed an issue where users can encounter intermittent errors when browsing and typing for a gene
Fixed an issue where the UI hangs on large genes and returns a 502 error
Fixed Issues
Data Management:
Fixed an issue where multiple folder copy jobs with the same destination may get stuck In Progress
Fixed an intermittent issue where tags on the target folder for a batch data update call are not set, but are set for all child data
Flow:
Fixed an issue causing intermittent pipeline failures due to an infrastructure error
Features and Enhancements
Navigation: If multiple regions are enabled for the same tenant, the region will be indicated in the waffle menu
Logging: Data transfers of BaseSpace Sequence Hub projects with data stored in ICA will be traced in ICA logs
Cohorts
Disease Search Box: Added support for specifying subjects by age of onset of disease(s)
Drug Search Box: Added a new query builder box for Drugs
Ingestion: Support for Drug, drug route, etc. attached to subjects
Cohorts building: Users can build cohorts by specifying drugs, drug route, etc.
Ingestion:
Combine different variant types during ingestion (small variants, cnv, sv)
Cohorts supports Illumina Pisces variant caller for hg19 VCFs
Fixed Issues
General:
Fixed an issue where the graphical UI hands with ha spinning wheel when saving or executing a command
Fixed an issue where rich text editor for Documentation tab on Pipelines, Tools, Projects and Bundles does not populate with correct styles in edit mode
Data Management:
Fixed an issue where multiple clicks on create data in Project API endpoint resulted in multiple requests
Fixed an issue where the secondary data selection screen could not be resized
A spinning wheel icon with ‘copying’ status is displayed at the folder level in the target Project when a folder is being copied. This applies to the actual folder itself and not for folders higher up in the hierarchy
Fixed an issue where API to retrieve a project data update batch is failing with 500 error when either the Technical or the User tags are updated during the batch update request
Fixed an issue where linking jobs fail to complete if other linking jobs are running
Improved performance for data transfer to support BaseSpace Sequence Hub Run transfers
Fixed an issue causing some folder copy jobs to remain in "Partially Succeeded" status despite being completed successfullyBundles: Fixed an issue where the URL and Region where a Docker image is available is not displayed for a Docker image Tool shared via an entitled Bundle
Fixed an issue where the folder copy job was getting stuck copying large amounts of big files
Fixed an issue where the folder counts were not matching expected counts after Data linking
Fixed an issue where delete data popup would occasionally not disappear after deleting data.
Fixed an issue with data copy where referencing data from another region would not result in immediate failure
Fixed issue where uploading a folder using the CLI was not working
Fixed an issue where a Docker image shared via an entitled Bundle can be added to another region
Workflows: Fixed an issue where workflow does not fail if BCL Convert fails for a BCL Convert-only run
Flow:
Improved performance when batches of data up to 1000 are added as input to an Analysis
Nextflow engine will return exit code 55 if the pipeline runner task is preempted
Fixed an issue where log files cannot be opened for any steps in an analysis while the analysis is in progress
Fixed an issue with concurrent updates on analysis
Fixed an issue where unknown data inputs in the XML of an analysis are not being ignored
The warning, close, and machine profile icons for Tools can now be seen in the graphical CWL pipeline editor
Fixed an issue where user cannot expand analysis output folder if user permissions change after starting analysis. Now, if a user has the correct permissions to start an analysis, that analysis should be able to finish correctly no matter the permissions at the time it succeeds
Base:
Fixed an issue switching back from template to Empty Table did not clear the fields
Data linked from an externally managed project can be added to Base Tables
Fixed an issue in the graphical UI where schema definition does not scroll correctly when many columns are defined
Features and Enhancements
Data Management/API: Added a new endpoint available to change project owner
POST /api/projects/{projectId}:changeOwner { “newOwnerId”:”}
Data Management/API: Added a new endpoint to copy data from one project to another:
/api/projects/{projectId}/projectDataCopyBatch
Data Management/CLI: Added the ability to copy files and folders between projects in the UI and CLI. This includes support for copying data from projects with ICA-managed storage (default) to projects with S3-configured storage.
Flow/API: When starting an analysis via the API, you can specify the input files based on HTTP(s). When your analysis is done, you will see the URL corresponding to the inputs in the UI, but you will not be able to start an analysis from the UI using this URL
Flow: Removed the ability to start analyses from data and sample views in the UI where a single input is selected to start analyses in bulk
Flow/CLI: Duplicate analyses submitted via the CLI will be avoided
Flow/API: Added two new endpoints for workflow sessions:
Get /api/projects/{projectId}/workflowSessions
Get /api/projects/{projectId}/workflowSessions/{workflowSessionId}/inputs
Flow/API: Added a new endpoint to retrieve configurations from a workflow session
Flow/Autolaunch ICA Workflow Session and Orchestrated Analyses (launched by the workflow session) now saves outputs in an organized folder structure: /ilmn-analysis/<name_used_to_create_sequencer_run_output_folder>
Base: The Base module has a new feature called ‘Data Catalogue’. This allows you to add usage data from your tenant/project if that data is available for you.
Data Catalogue views will be available and can be used in Base to query on.
You will be able to preview and query Data Catalogue views through Base Tables and Query screens
The Data Catalogue will always be up to date with the available views for your tenant/project
Data Catalogue views cannot be shared through a Bundle
Data Catalogue views will also be available to team members that were added after the view was added
Data Catalogue views can be removed from the Base tables and corresponding project
By removing Base from a project, the Data Catalogue will also be removed from that project
Cohorts: Disease Search box
Cohorts now includes a disease search box to search for disease concepts. This replaces the disease concept tree explorer
Disease search box located under a Disease tab in main Query builder
Search box allows for a copy/paste action of codes to be processed as separate query elements. Currently, the feature is limited to a complete valid list
Each disease entered into the search box is displayed as a separate query item and can be set to include or exclude.
Diseases in search box can be used with boolean logic in cohort creation
Search box allows for an auto-complete of diagnosis concepts and identifiers
The disease filter is included in the cohort query summary on cohort page
Fixed Issues
Data Management: Data copy between ICA-managed projects and S3 storage configured projects is supported
Data Management: Fixed an issue where storage configurations matching ICA-managed buckets would cause volume records to get associated with the wrong storage configuration in the system
Data Management/API: The endpoint GET/api/projects/{ProjectID}/samples/{SampleID} correctly returns all the own samples and linked samples
Data Management/API: Improved handling of bulk update via API when concurrent deletion of file has occurred
Data Management/CLI: Fixed an issue where projectdata update tags would not update the tags
Flow: Fixed an issue resulting in failure to send a notification resulting in a failed workflow
Flow: Fixed an issue where one workflow session may override another when both are executed at the same time
Base: Fixed an issue where query download in JSON format returns an error
Base: Added a message in the UI when a query takes longer than 30 seconds to inform the user that the query is ongoing and can be monitored in the Activity view
Base: Added a section describing the Data Catalogue functionality
Bench: Fixed an issue where resizing the workspace to current size would prevent users from resizing for the next 6 hours
Cohorts: Fixed an issue where Gene Expression table does not display with TCGA data or for tenants with a hyphen (e.g., ‘genome-group’)
Cohorts: Fixed an issue where user had no way to delete a cohort comparison from a deleted cohort
Cohorts: Fixed an issue in the UI where multi-cohort needle plot tracks are overlapping
Cohorts: Fixed an issue causing failures during annotation step with ‘CNV’ data type when selection ‘GB=hg19’ and ‘CNV data’ for liftover; also observed with ‘SM data’ and ‘hg38’ without liftover (in APS1 and CAC1 regions) due to a ‘404 Not Found’ error.
CLI: Fixed an issue to support adding the server-url as a parameter instead of having the config set
Fixed Issue
Fixed an issue uploading folders via the CLI
Fixed Issue
Fixed an issue causing CWL pipelines using Docker images that do not contain bash shell executable to fail.
Fixed Issue
Fixed an issue leading to intermittent system instability.
Fixed Issue
Cohorts: Issue fixed where GTEx plot is not available for tenants with a hyphen (e.g. ilmn-demo).
Features and Enhancements
Data Management: New API endpoints are available for:
Creation of a data update in bulk: POST/api/projects/{projectId}/dataUpdateBatch
A list of data updates for a certain project: GET/api/projects/{projectId}/dataUpdateBatch/{batchId}
A list of items from the batch update: GET/api/projects/{projectId}/dataUpdateBatch/{batchId}/items
A specific item from the batch update: GET/api/projects/{projectId}/dataUpdateBatch/{batchId}/items/{itemId} Note: Batch updates include tags, format, date to be archived and date to be deleted
Data Management/API: The sequencing run information can be retrieved through its Id by using the API endpoint GET/api/sequencingRuns/{sequencingRunId}
Flow: Auto launch now supports BCL Convert v3.10.9 pipeline and both TruSight Oncology 500 v2 pipelines (from FASTQs)
Flow/API: getPipelineParameters API returns parameter type in response
Flow/API: Added endpoints to retrieve and update a project pipeline definition
Flow/API/CLI: Include "mountPaths" field in response for API and CLI command to retrieve analysis inputs
Flow: Removed "fpga-small" from available compute types. Pipelines using "fpga-small" will use the "fpga-medium"-equivalent compute specifications instead
Flow/API: New API endpoint available to request the analyses in which a sample is being used
Flow/API: When leaving activationCodeDetailId empty when starting an analysis, the best match activation code will be used
Flow: Analyses launched/tracked by BaseSpace Sequence Hub contain relevant BaseSpace information in analysis details view
Bench: A new Bench permission is being introduced: Administrator. This permission allows users to manages existing workspaces and create new workspaces
Bench: The Bench Administrator role allows you to create new Bench workspaces with any permissions even if you as a Bench administrator do not have these permissions. In that case, you can create and modify the workspace, but you cannot enter that workspace. Modifying is only possible when the workspace is stopped
Bench: As a Bench Contributor you are not allowed anymore to delete a Bench Workspace, you need the Bench Administrator role.
Cohorts: Users can now ingest raw DRAGEN bulk RNAseq results for genes and transcripts (TPM), with the option to precompute differential expression during ingestion
Cohorts: Added support for running multiple DEseq2 analyses in the ingestion workflow through bulk processing based on sample size and specific requirements
Cohorts: In multiple needle plot view, individual needle plots can now be collapsed and expanded
Cohorts: Pop-outs for needle plot variants now contain additional links to external resources, such as UCSC
Cohorts: For a given cohort, display a distribution of raw expression values (TPM per gene) for a selected attributes
Cohorts: Use of the Cohorts maintains session between core ICA and Cohorts iFrame to prevent unwanted timeouts
Cohorts: Cohorts displays structural variants that include or overlap with a gene of interest
API: Two new API endpoints added to accept Terms and Conditions on a bundle:
GET /api/bundles/{bundleId}/termsOfUse/userAcceptance/currentUser Returns you the time of acceptance when you, the current user, accepted the Terms & Conditions.
POST /api/bundles/{bundleId}/termOfUse:accept
API: Add temporary credentials duration to API documentation
Notifications: List of events to which you can subscribe contains new ICA notification containing analyses updates
Versioning: The ICA version can now be found under your user when you select "About"
Versioning/API: It is possible to retrieve system information about ICA, such as the current version through GET/api/systeminfo
Logging: When an action is initiated by another application, such as BaseSpace Sequence Hub, it will be traced as well in the ICA logs
Fixed Issues
Data Management/API: Improved error handling for API endpoint: DELETE/api/bundles/{bundleId}samples{sampleId}
Data Management: CreatorId is now present on all data, including subfolders
Data Management: Improved external linking to data inside ICA using deep linking
Data Management: Improved error handling when creating folders with invalid characters.
Data Management: Fixed an inconsistency for URN formats on output files from Analyses. This fix will apply only for analyses that are completed starting from ICAv2.18.0
Data Management: Improved resilience in situations of concurrent linking and unlinking of files and folders from projects
Data Management/API: Fixed an issue where the API endpoint GET /api/samples erroneously returned a 500
Data Management/API: API endpoint GET/api/projects/{projectId}/analyses now returns the correct list when filtering on UserTags whereas it previously returned too many
Data Management: It is only possible to delete a storage configuration if all projects that are using this storage configuration have been hidden and are not active projects anymore
Data Management: Improved accuracy of the displayed project data size. Prior cost calculations were accurate, but the project data size visualization included technical background data
Data Management/API: Improved retry mechanism for API endpoint to create folderuploadsession
Data Management/CLI: When an upload of a folder/file is done through the CLI, it returns the information and ID of the folder/file
Data Management: Fixed an issue where there is a discrepancy in number of configurations between Storage->Configurations and Configurations-> Genomics.Byob.Storage Configuration view
Collaboration: Fixed an issue where a user is presented with a blank screen when responding to a project invitation
Flow/API: Improved error handling when invalid project-id is used in API endpoint GET /api/projects/{projectId}/pipelines
Flow: When a spot instance is configured, but revoked by AWS, the pipeline will fail and exit code 55 is returned
Flow/API: Fixed an issue when an Analysis completed with error "incomplete folder session", the outputs of the Analysis are not always completely listed in the data listing APIs
Flow/API: Updated ICA Swagger Project > createProject to correctly state that the analysis priority must be in uppercase
Flow: Fix to return meaningful error message when instrument run ID is missing from Run Completion event during an auto launched analysis
Flow: Improved parallel processing of the same analysis multiple times
Base: Improved error handling when creating queries which use two or more fields with the same name. The error message now reads "Query contains duplicate column names. Please use column alias in the query"
Base: Fixed an issue where queries on tables with many entries fail with NullPointerException
Bench: Clarified that changes to Bench workspace size only take effect after a restart
Cohorts: Fixed issue where counts of subjects are hidden behind attribute names
Cohorts: Fixed issue where the state of checked files are not retained when selecting molecular files that are in multiple nested folders
Cohorts: Fixed issue where projects that contain files from linked bundles cause a time out, resulting in users not being able to select files for ingestion
Cohorts: Fixed an issue where the 'Import Jobs' page loaded within the Data Sets frame, depending on where the import was initiated
Cohorts: Fixed an issue in the Correlation plat where x-axis counts were hidden under attribute names
Fixed an issue where users were previously incorrectly signed out of their active sessions
Fixed Issues
Fixed an issue causing analyses requesting FPGA compute resources to experience long wait times (>24h) or not be scheduled
Features and Enhancements
Data Management: Performance improvements for data link and unlink operations – Larger and more complex folders can now be linked in the graphical UI, and progress can be monitored with a new visual indication under Activity > Batch Jobs
Flow: Increased the allowed Docker image size from 10GB to >20GB
Flow/CWL: Added support for javascript expressions “ResourceRequirements” fields (i.e., type, size, tier, etc.) in CWL Pipeline definitions
Flow/API: Added support for using Pipeline APIs to query Pipelines included in Entitled Bundles (i.e., to retrieve input parameters)
Flow/API: Added support for providing S3 URLs as Pipeline data inputs when launching via the API (using storage credentials)
Flow/API: Added support for specifying multi-value input parameters in a Pipeline launch command
Bench: Project and Tenant Administrators are now allowed to stop running Workspaces
Notifications: Notifications are now available for batch job changes
Cohorts: Enhanced ingestion workflow to ingest RNAseq raw data from DRAGEN output into backend Snowflake database
Cohorts: Added support for running multiple DEseq2 analyses in the ingestion workflow through bulk processing based on sample size and specific requirements
Cohorts: Multi-Cohort Marker Frequency - Added Multi-Cohort Marker Frequency tab allowing users to compare expression data across up to four Cohorts at the gene level
Cohorts: Multi-Cohort Marker Frequency includes a pairwise p-value heat map
Cohorts: Multi-Cohort Marker Frequency - Includes frequencies for Somatic and Copy Number Variants
Cohorts: Tab added for a multi-cohort marker frequency analysis in cohort comparisons
Cohorts: Multi-Cohort Needle Plot - Added new tab in the Comparison view with vertically aligned needle plots per cohort for a specified gene, allowing collapsible and expandable individual needle plots
Cohorts: Additional filter logic added to multi-cohort needle plot
Cohorts: Improved DRAGEN data type determination during ingestion allowing for multiple variant type ingestion
Cohorts: Enhanced list of observed variants with grouped phenotypes and individual counts, including a column for total sample count; tooltips/pop-outs provide extended information
Cohorts: Updates to needle plot link outs
Cohorts: Improved the Comparison feature by optimizing API calls to handle subjects with multiple attributes, ensuring successful loading of the page and enabling API invocation only when the user selects or expands a section
Cohorts: Removed unused columns (genotype, mrna_feature_id, allele1, allele2, ref_allele, start_pos, stop_pos, snp_id) from annotated_somatic_mutations table in backend database
Cohorts: Refactored shared functionality for picking consequence type to reduce code duplication in PheWAS-plot and GWAS-plot components
Cohorts: Invalid comparisons on the Comparisons page are now grayed out and disabled This improvement prevents the selection of invalid options
Cohorts: Automatic retry of import jobs when there are failures accessing data from ICA API
Fixed Issues
Data Management: The content of hidden Projects can now be displayed
Data Management: Fixed the TimeModified timestamp on files
Bundles: Resolved issues when linking a large number of files within a folder to a Bundle
Flow: Single values are now passed as a list when starting an Analysis
Flow: Pipelines will succeed if the input and output formats specified on the pipeline level match at the Tool level
Flow: Fixed an issue causing Analysis failures due to intermittent AWS S3 network errors when downloading input data
Flow/CWL: Improved performance on output processing after a CWL Pipeline Analysis completes
Flow/UI: Mount path details for Analysis input files are now visible
Flow/UI: Improved usability when starting an Analysis by filtering entitlement options based o inputs selected and available entitlements
Flow/API: List of Analyses can now be retrieved via the API based on filters for UserReference and UserTags
Base: Fixed an issue where the Scheduler continues to retry uploading files which cannot be loaded
Bench: Resolved an issue when attempting to access Workspaces with multi-factor authentication (MFA) enabled at the Tenant-level
API: Improved error messaging for POST /api/projects/{projectId}/data/{dataId}:scheduleDownload
Navigation: Removed breadcrumb indication in the graphical UI
Cohorts: Fixed issue where Correlation bubble plot not showing for any projects intermittently
Cohorts: Fixed issue where importing Germline/hg19 test file did not load variants for a specific gene in the Needle plot due to missing entries in the Snowflake table
Cohorts: Fixed a bug causing an HTTP 400 error while loading the Cohort for the second time due to the UI passing "undefined" as variantGroup, which failed to convert to the VariantGroup Enum type
Cohorts: Fixed issue where scale (y-axis) of needle plot is changed even if value of sample count gnomAD frequency is not accepted
Cohorts: Fixed an issue where no data was generated in the Base Tables after a successful import job in Canada - Central Region (CAC1)
Cohorts: Fixed issue where long chart axis labels overlap with tick marks on graph
Features and Enhancements
Data Management: You can now select 'CYTOBAND' as format after file upload
Data Management: Added support for selecting the root folder (of the S3 bucket) for Projects with user-managed
Data Management: Added support for creating an AWS Storage Configuration with an S3 bucket with Versioning enabled
Auto-launch: Added technical tags for upstream BaseSpace Run information to auto-launched analyses
Auto-launch: Added support for multiple versions of BCL Convert for auto-launched analyses
Flow: Added support for '/' as separator in CWL ResourceRequirements when specifying Compute Type
Flow/API: The API to retrieve analysis steps now includes exit code for completed steps
Bench: Restricted Workspaces (Open or Restricted) always allow for access to Project Data within the Workspace
Bench: Restricted Bench workspaces have limited access through whitelisted URLs that are checked before entry
Bench: Restricted Bench Workspaces allow for Open or Restricted workspaces. Restricted workspaces do not have access to the internet except for user-entered whitelist URLs
Fixed Issues
Data Management: Upload for files names including spaces is now consistent for connector and browser upload. We do still advise not to use spaces in file names in general
Data Management: Fixed search functionality in Activity > Data Transfers screen
Data Management: Improved performance on opening samples
Data Management: Fixed an issue where reference data in download tab initiates an unexpected download
Data Management: Fixed intermittent issue where the Storage configuration within a Project can go into Error status and can block users from creating records such as folders and files
Data Management/API: Improved error handling for API endpoints: Delete/api/projects/ {projectId}/bundles/{bundleId} and POST/api/projects/{projectId}/bundles/{bundleId}
Data Management/API: Improved error handling for POST/api/projects/{projectId}/base:ConnectionDetails
Bundles: Fixed an issue where the Table view in Bundles is not available when linking to a new Bundle version
Bundles/API: Improved error handling for DELETE/api/bundles/{bundleId}/tools/{toolId} and POST/api/bundles/{bundleId}/tools/{toolId}
Bundles/API: Improved error message for POST/api/bundles/{bundleId}/samples/{sampleId}
Bundles: Fixed an issue where linking/unlinking a Bundle with Base Tables could result in errors
Service Connector: Improved error message for DELETE/api/connectors/{connectorId}/downloadRules/{downloadRuleId}
Notifications/API: Custom subscriptions with empty filter expressions will not fail when retrieving them via the API
Notifications/API: Improved error handling for POST/api/projects/{projectId}/notificationSubscriptions
Notifications/API: Improved notification for Pipeline success events
Flow: When the input for a pipeline is too large, ICA will fail the Analysis and will not retry
Flow: Fixed issue where analysis list does not search-filter by ID correctly
Flow: Improved error handling when issues occur with provisioning resources
Flow: When retry succeeds in a Nextflow pipeline, exit code is now '0' instead of '143'
Flow/API: Fixed an issue causing API error when attempting to launch an Analysis with 50,000 input files
Flow/API: Improved pipeline error code for GET/api/projects/{projectId}/pipelines/{pipelineId} when already unlinked pipeline Id is used for API call
Flow/API: Fixed an issue where Analyses could not be retrieved via API when the Pipeline contained reference data and originated from a different tenant
Flow/API: Fixed filtering analyses on analysisId. Filtering happens via exact match, so part of the Id won't work
Bench/CLI: Fixed issue where the latest CLI version was not available in Bench workspace images
Navigation: Updated URLs for Correlation Engine and Emedgene in the waffle menu
Authentication: Using POST /api/tokens:refresh for refreshing the JWT is not possible if it has been created using an API-key.
Authentication: Improved error handling when there is an issue reaching the authentication server
Authentication: Improved usability of "Create OAuth access token" screen
Cohorts: Issue fixed where CNV data converted from hg19 to hg38 do not show up in Base table views.
Cohorts: Issue fixed accounting for multiple methods of referring to the alternate allele in a deletion from Nirvana data.
Cohorts: Fixed intermittent issue where GWAS ingestions not working after Base enabled in a project.
Fixed Issue
Fixed an issue causing incorrect empty storage configuration dropdown during Project creation when using the “I want to manage my own storage” option for users with access to a single region
Features and Enhancements
General availability of sequencer integration for Illumina sequencing systems and analysis auto launch
General usability improvements in the graphical interface, including improved navigation structure and ability to switch between applications via the waffle menu in the header
Storage Bundle field will be auto-filled based on the Project location that is being chosen if multiple regions are available
Event Log entries will be paged in the UI and will contain a maximum of 1,000 entries. Exports are limited to the maximum number of entries displayed on the page.
Read-only temporary credentials will be returned when you are not allowed to modify the contents of a file
Creating Project notifications for BaseSpace externally managed projects is now supported
The ICA UI will only allow selection of storage bundles belonging to ICA during Project creation, and the API will only return storage bundles for ICA
Flow: Allow attached storage for Pipeline steps to be set to 0 to disable provisioning attached storage and improve performance
Cohorts: GRCh37/hg19-aligned molecular data will get converted to GRCh38/hg38 coordinates to facilitate cross-project analyses and incorporating publicly available data sets.
API: Project list API now contains a parameter to filter on (a) specific workgroup(s)
API: Two new API endpoints are added to retrieve regular parameters from a pipeline within or without a Project context
Fixed Issues
Optimized price calculations resulting in less overhead and logging
Improved error handling:
during Project creation
of own storage Project creation failures.
to indicate connection issue with credentials
for graphical CWL draft Pipelines being updated during an Analysis.
Fixed an issue causing data records to remain incorrectly in Unarchiving status when an unarchive operation is requested in the US and Germany regions
Improved error messaging in cases where the AWS path contains (a) special character(s)
Fixed an issue causing errors when navigating via deep link to the Analysis Details view
CLI: Unarchival of folders is supported when archive or unarchive actions are not in progress for the folder
CLI: Improved error message to indicate connection issue with credentials
Flow: Fixed an issue causing incorrect naming of Analysis tasks generated from CWL ExpressionTools
Flow: Fixed an issue when cloning Pipelines linked from Entitled Bundles to preserve the original Tenant as the Owning Tenant of the cloned Pipeline instead of the cloning user’s Tenant
Flow: Fixed an issue causing outputs from CWL Pipelines to not show in the Analysis Details despite being uploaded to the Project Data Analysis output folder when an output folder is empty
Flow: When a Contributor starts an Analysis, but is removed afterwards, the Analysis still runs as expected
Flow: Fixed an issue where Analyses fail where Nextflow is run a second time
Flow: Fixed an issue causing API error when attempting to launch an Analysis with up to 50,000 input files
Flow: Fixed an issue causing degraded performance in APIs to retrieve Analysis steps in Pipelines with many steps
Flow: Fixed an issue causing Analysis failure during output upload with error “use of closed network connection”
Flow: Fixed an issue causing disk capacity alter log to not show when an Analysis fails due to disk capacity and added error message
Flow: Fixed an issue preventing cross-tenant users from being able to open a shared CWL pipeline
Base: Improved target Table selection for schedulers to be limited to your own Tables
Bench: Fixed an issue causing Workspaces to hang in the Starting or Stopping statuses
Cohorts: Now handles large VCFs/gVCFs correctly by splitting them into smaller files for subsequent annotation by Nirvana
API: Fixed returning list of unlinked data in a sample that was linked before in GET/api/projects/{projectId}/data
API: Fixed error for getSampleCreationBatch when using status filter
Features and Enhancements
Added a limit to Event Log and Audit UI screens to show 10,000 records
API: Parent output folder can be specified in URN format when launching a Workflow session via the API
Flow: Reduced Analysis delays when system is experiencing heavy load
Flow: Improved formatting of Pipeline error text shown in Analysis Details view
Flow: Users can now start Analyses from the Analysis Overview screen
Flow: Superfluous “Namespace check-0” step was removed to reduce Analysis failures
Flow: Number of input files for an Analysis is limited to 50,000
Flow: Auto launched Workflow sessions will fail if duplicate sample IDs are detected under Analysis Settings in the Sample Sheet
Base: Activity screen now contains the size of the query
Cohorts
Detect and Lift Genome Build: Cohorts documentation provides set-up instructions to convert hg19/GRCh37 VCFs to GRCh38 before import into Cohorts.
Attribute Queries: Improved the user experience choosing a range of values for numerical attributes when defining a cohort
Export Cohort to ICA Project Data: Improved the user experience exporting list of subjects that match cohort definition criteria to their ICA project for further analysis
Ingest Structural Variants into database
The Cohorts ingestion pipeline supports structural variant VCFs and will deposit all such variants into an ICA Base table if Base is enabled for the given project
Structural variants can be ingested and viewed in base tables
Needle Plot Enhancements
Users can input a numerical value in the Needle Plot legend to display variants with a specific gnomAD frequency percentage or sample count
The needle plot combines variants that are observed among subjects in the current project as well as shared and public projects into a single needle, using an additional shape to indicate these occurrences
Needle Plot legend color changes for Variant severity; pathogenic color coding is the same as the color coding in the visualization; differentiating hue between proteins and variants; and other color coding changes.
Needle plot tool tips that display additional information on variants and mutations are now larger and modal
The needle plot now allows to filter by gnomAD allele frequency and sample count in the selected cohort. Variants include links to view a list of all subjects carrying that variant and export that list.
Remove Samples Individually from Cohorts
Exclude individual subjects from a cohort and save the refined list
The subjects view allows users to exclude individual subjects from subsequent analyses and plots and save these changes
Subject exclusions are reset when editing a cohort
Subject Selection in Analysis Visualization: Users can follow the link for subject counts in the needle plot to view a list of subjects carrying the selected variant or mutation.
UI/UX: Start and End time points are available as a date or age with a condition attribute in the subject data summary screen.
Fixed Issues
Improved resilience against misconfiguration of the team page when there is an issue with Workgroup availability
Improved handling of multi-file upload when system is experiencing heavy loads
Fixed an issue to allow upload of zero-byte files via the UI
Fixed issue where other Bundles would not be visible after editing and saving your Bundle
Removed ‘IGV (beta)’ button from ‘View’ drop down when selecting Project Data in UI
API: Improved error handling for API endpoint: POST /api/projects/{projectId}/analysisCreationBatch
API: Improved performance of API endpoint: getbestmatchingfornextflow
Flow: Fixed an issue causing Analysis output mapping to incorrectly use source path as target path
Flow: Fixed an issue where the UI may display incorrect or invalid parameters for DRAGEN workflows which do not accurately show the true parameters passed. Settings can be confirmed by looking at the DRAGEN analysis log files.
Base: “Allow jagged rows” setting in the Scheduler has been replaced with “Ignore unknown values” to handle files containing records with more fields than there are Table columns
Base: Improved Base Activity view loading time
Base: Fixed an error message when using the API to load data into a Base Table that has been deleted
Bench: Fixed an issue resulting in incorrect Bench compute pricing calculations
Bench: Fixed an issue preventing building Docker images from Workspaces in UK, Australia, and India regions
Bench: Fixed an issue where /tmp path is not writeable in a Workspace
Cohorts:
Fixed issue where the bubble plot sometimes failed to display results even though the corresponding scatter plot showed data correctly.
The order of messages and warnings for ingestion jobs was not consistent between the UI and an error report sent out via e-mail.
The UI now displays any open cohort view tabs using shortened (“…”) names where appropriate
Issue fixed where ingestions with multiple errors caused halting to the ingestion queue.
The needle plot sometimes showed only one source for a given variant as opposed to all projects in which the variant had been observed in.
Issue fixed with unhandled genotype index format in annotation file to base database table conversion
Status updates via e-mail sometimes contained individual error messages or warnings without a text.
Fixed issue where items show in needle plot with incorrect numbering on the y-axis.
Fixed performance issue with subject count.
Widget bar-chart counts are intermittently cut off over four digits.
Fixed slowness when switching between tabs in query builder
Fixed Issue
Fixed issue with BaseSpace Free Trial and Professional users storing data in ICA
Fixed Issue
Fixed an issue resulting in analysis failures caused by a Kubernetes 404 timeout error
Features and Enhancements
API: POST /api/projects/{projectId}/data:createDownloadUrls now supports a list of paths (in addition to a list of IDs)
Each tenant supports a maximum of 30,000 Projects
.MAF files are now recognized as .TSV files instead of UNKNOWN
Added VCF.IDX as a recognized file format
General scalability optimizations and performance improvements
Fixed Issues
After creating a new Bundle the user will be taken to the Bundle Overview page
Fixed an issue preventing the ‘Owning Project’ column from being used outside of Project
Fixed an issue which prevented changing the format of a file back to UNKNOWN
Fixed an issue causing inaccurate upload progress to be displayed for UI uploads. The Service Connector or CLI are recommended for large file uploads.
Fixed an issue showing an incorrect status for data linking batch jobs when data is deleted during the linking job
Fixed an issue allowing the region of a Project to be changed. Changing the region of a resource is not supported.
Strengthened data separation and improved resilience against cross-Project metadata contamination
Service Connector: Fixed an issue allowing download of a Service Connector when no operating system is set
Service Connector: Cleaned up information available on Service Connectors by removing empty address information fields
API: Fixed date formatting for GET /api/eventLog (yyyy-MM-dd’T’HH:mm:ss.SSS’Z’)
API: Fixed an issue where the GET users API was not case sensitive on email address
API: Fixed an issue causing the metadata model to be returned twice in PSOT /api/projects/{projectId}/samples:search
API: Fixed the listProjects API 500 response when using the pageoffset query parameter
API: The searchProjectSamples API returns Sample metadata for Samples shared via a Bundle
API: Fixed an issue causing createProjectDataDownloadUrls API 400 and 502 errors when server is under load
Flow: Fixed analysis failures caused by kubernetes 404 timeout error
Flow: Fixed an issue where Workflwos would prematurely report completion of an Analysis
Flow: Improved Pipeline retry logic to reduce startup delays
Flow: Fixed an issue where Nextflow pipelines were created with empty files (Nextflow config is allowed to be empty)
Flow: Removed the 1,000 input file limitation when starting an Analysis
Flow: Improved the performance of status update messages for pipelines with many parallel steps
Flow: Fixed an issue with overlapping fields on the Analysis Details screen
Flow: Deactivated the Abort button for Succeeded analyses
Base: Fixed an issue where Pipeline metadata was not captured in the metadata Table generated by the metadata schedule
Base: Error logging and notification enhancements
Bench: Fixed an issue where Workspaces could be started twice
Bench: Fixed an issue where the system checkpoint folder was incorrectly created in Project data when opening a file in a Workspace
Features and Enhancements
Analysis system infrastructure updates
Features and Enhancements
Added ability to refresh Batch Jobs updates without needing to leave the Details screen.
Projects will receive a job queuing priority which can be adjusted by an Administrator.
The text "Only showing the first 100 projects. Use the search criteria to find your projects or switch to Table view." when performing queries is now displayed both on the top and bottom of the page for more clarity.
API: Added a new endpoint to retrieve download URLs for data: POST/api/projects/{projectId}/data:createDownloadUrls
API: Added support for paging of the Project Data/getProjectDataChildren endpoint to handle large amounts of data.
API: Added anew endpoint to deprecate a bundle (POST /api/bundles/{bundleId}:deprecate)
API: If the API client provides request header "Accept-Encoding: gzip", then the API applies GZIP compression on the JSON response. This way the size of the response is significantly smaller which improves the download time of the response, resulting in faster end-to-end API calls. In case of compression the API also provides header "Content-Encoding: gzip" in the response, indicating that compression was effectively applied.
Flow: Optimized Analysis storage billing, resulting in reduced pipeline charges.
Flow: Internal details of a (non-graphical) pipeline marked ‘Proprietary’ will not be shared with users from a different tenant.
Flow: A new grid layout is used to display Logs for Analyses with more than 50 steps. The classic view is retained for analyses with 50 steps or less, though you can choose to also use the grid layout by means of a grid button on the top right on the Analysis Log tab.
CLI: Command to launch a CWL and Nextlfow Pipeline now contains the mount path as a parameter.
CLI: Version command now contains the build number.
CLI: Added support for providing the nextflow.config file when creating a Nextflow pipeline via CLI.
API: HTML documentation for aPipeline can now be returned with the following requests:
GET /api/pipelines/{pipelineId}/documentation/HTML
GET /api/projects/{projectId}/pipelines/{pipelineId}/documentation/HTML
API: Added a new endpoint for creating and starting multiple analyses in batch: POST /api/projects/{projectId}/analysisCreationBatch
Flow: Linking to individual Analyses and Workflow sessions is now supported by /ica/link/project//analysis/ and /ica/link/project//workflowSession/
Cohorts: Users can now export subject lists to the ICA Project Data as a file.
Cohorts: Users can query their ingested data through ICA Base. For users who already have ingested private data into ICA Cohorts, another ingestion will need to happen prior to seeing available database shares. Customers can contact support to have previously ingested data sets available in Base.
Cohorts: Correlation bubble plot counts now link to a subject/sample list.
Fixed Issues
Tooltip in the Project Team page provides information about the status of an invite
‘Resend invite’ button in the Project Team page will become available only when the invite is expired instead of from the moment the invite is being send out
Folders, subfolders and files all contain information about which user created the data
Files and folders with UTF-8 character are not supported. Please look at the documentation on how to recover from it in case you already have used them.
Improved performance for creating or hiding a Project in a tenant with many Projects
Service Connector: Updated information in the Service Connector screen to reflect the name change from "Type of Files" to the more accurate "Assign Format"
Service Connector: Folders within a Bundle can be downloaded via the Service Connector
Service Connector: Upload rules can only be modified in the Project where they apply
Service Connector: A message describes when a file is skipped during upload because it already exists in the Project
Service Connector: Fixed an issue where opening the Connectivity tab occasionally results in a null pointer error
Service Connector: Fixed an issue causing excessive logging when downloading files with long file paths
Service Connector: Fixed an issue where the Service Connector log may contain spurious errors which do not impact data transfers
Existing storage configurations are displayed and accessible via API and UI
Newly added storage configurations do no longer remain in ‘Initializing’ state
Fixed error when creating a storage configuration with more than 63 characters
Clicking on a Data folder in flat mode will now open the details of the folder
Only Tools in Released state can be added to a Bundle
Fixed issue preventing new Bundle versions to be created from Restricted Bundles
Deprecated Bundles are displayed upon request in card and table view
Bundles view limited to 100 Bundles
API: Fixed the API spec for ProjectDataTransfer.getDataTransfers
API: Fixed an issue with the projectData getChildren endpoint which returned incorrect object and pagination
API: Fixed an issue where multiple clicks on Create sample batch API endpoint resulted in multiple requests
API: POST /api/projects/{projectId}/data/{dataId}:scheduleDownload can now also perform folder downloads
API: Improved information on the Swagger page for GET /api/pipelines, GET/api/projects/{projectId}/pipelines, and GET/api/projects/{projectId}/pipelines/{pipelineId}
API: Fixed and issue when a user provides the same input multiple times to a multi-value input on an analysis run, that input is only passed to the pipeline once instead of multiple times: POST /api/projects/{projectId}/analysis:nextflow
CLI: Copying files in the CLI from a local directory on MacOS to your Project can result in both the desired file and the metadata file (beginning with ‘./’) being uploaded. The metadata file can safely be deleted from the Project
CLI: Hardened protection against accidental file overwriting
CLI: Improved handling for FUSE when connection to ICA is lost
CLI: icav2 projectdata mount –list shows updated list of mounted Projects
CLI: Paging improvements made for project list, projectanalyses list, and projectsdata list
CLI: When there is no config or session file the user will not be asked to create one for icav2 config reset and icav2 config get
CLI: Fixed an issue where Bundle data could not be seen through FUSE in Bench
CLI: Fixed an error message when missing config file upon entering the Project context
CLI: The unmount is possible without a path and will work via the stored Project ID or with a directory path resulting in an unmount of that path
CLI: Fixed an error when creating a Pipeline using URN for Project identifier
CLI: Attempting to delete a file from an externally-managed project returns an error indicating this not allowed
CLI: Fix to delete session file when config file is not detected
CLI: Paging option added to projectsamples list data
CLI: Fixed “Error finding children for data” error in CLI when downloading a folder
CLI: projectdata list now returns the correct page-size results
Flow: Fixed handling of special characters in CWL pipeline file names
Flow: Fixed an issue where task names exceeding 25 characters cause analysis failure in CWL pipelines
Flow: Fixed an issue which prevented requests for economy tier compute
Flow: Fixed an issue limiting CWL workflow concurrency to two running tasks
Flow: Fixed an issue where analysis file inputs specified in the input.json with ‘location’set to an external URL cause to CWL pipelines to fail
Flow: Fixed an issue resulting in out of sync Pipeline statuses
Flow: Improved Nextflow engine resiliency, including occurrences where Nextflow pipelines fail with ‘pod 404 not found’ error
Flow: Fix issue with intermittent system runtime failures incorrectly causing analysis failures
Flow: Fixed an issue where links to Analysis Details returned errors
Flow: Enabled scrolling for Pipeline documentation
Flow: Improved performance for handling analyses with large numbers of inputs
Flow: Improved handling of hanging Analyses
Flow: Improved error messages for failed Pipelines
Flow: Added documentation on how to use XML configuration files for CWL Pipelines
Flow: Duplicate values for multi-value parameters are no longer automatically removed
Flow: Correct exit code 0 is shown for successful Pipeline steps
Base: Fixed an issue so that only users with correct permissions are allowed to retrieve a list of Base tables
Base: Fixed an issue with metadata scheduler resulting in a null pointer
Base: An empty Table description will not return an error when requesting to list all Tables in a Project
Base: Jobs failed with an error containing 'has locked table' are not shown on the Base Job activity list. They can be displayed by selecting the 'Show transient failures' checkbox at Projects > Activity > Base Jobs.
Base: Users can see Schedulers and their results for the entire tenant if created by a tenant administrator in their project, but not create, edit or run them
Base: Fixed an issue preventing data format change in a schedule
Base: Fixed an issue preventing exporting data to Excel format
Bench: Improved handling to prevent multiple users in a single running Workspace
Bench: Fixed an issue causing Workspaces to be stuck in "Starting" state
Bench: Fixed an issue where usage does not showing up on usage CSV-based report
Bench: Fixed an issue where Bundle data could not be seen via the Fuse driver
Bench: Users can now consistently exit Workspaces with a single click on the ‘Back’ button.
Bench: After leaving a Workspace by clicking on the ‘Back’ button, the Workspace will remain in a ‘Running’ state and become available for a new user to access
Bench: Workspaces in a ‘Stuck’ state can be manually changed to ‘Error’ state, allowing users to restart or delete them
Cohorts: Fixed issue where file system cleanup not occurring after delete.
Cohorts: Fixed sign in and authentication issues in APN1 region.
Cohorts: Fixed issue where gene filter missing when editing a cohort and removing the edited filter and cancelling. The filter was preserved and should not have been.
Cohorts: Fixed issue where users see an application tile in the Illumina application dashboard selection screen called "Cohort Analysis Module".
Cohorts: Correlation: Fixed issue, Data type selections shows half when loading the search result
Cohorts: Fixed issue, Users will see an application tile on the Connected Platform home page screen called “Cohort Analysis Module” if the Cohorts module is added to the domain. Users should not enter the ICA Cohorts through this page. They should enter through ICA."
Features and Enhancements
New notifications added for data upload, data created, and analyses with ‘aborted’ state
Flow: Progress on batch jobs can now be monitored in the Activity screen under the ‘Batch Jobs’ tab
Base: Shared database activity is available in the Base Activity Logging Screen
Cohorts:
Enhanced Attribute Queries: Filter logic now allows users to include or exclude filter criteria.
Ingestion Wizard: Human readable project name added to email notification of ingestion status
Ingestion Wizard: Notification email now includes the project name and job failure details
Ingestion Wizard: File lists are now sortable and filterable by type
Ingestion Wizard: UI enhancements on ingestion wizard selections
Ingestion Wizard: Based on data type chosen, the list of files for ingestion will be filtered for easier navigation of files to ingest
Share Cohorts with Collaborators - Cohort definitions are now sharable in a project
Statistical significance on Cohort Analysis - A p-value calculation is added to the Survival Comparison data points
Fixed Issues
Resolved an error when creating a Bucket Name with more than 63 characters
Fixed an issue causing failure when attempting to link "complex" folders (i.e., contains 100 sub-folders each with sub-folders) via GUI throws a “…transaction is not active…" error
Fixed an issue causing intermittent failure of "complex" folder (i.e., contains 100 sub-folders each with sub-folders) downloads
Fixed an issue so that the API to create data download URL correctly returns a 503 instead of a 404 response when the underlying system is experiencing network-related issues
Fixed an issue where the user is not redirected to the Project overview view upon clicking ‘Hide a Project’
Fixed an issue which prevented linking and unlinking multiple Bundles to and from a Project in a single operation via the GUI
Fixed an issue to show records in Bundle Data Transfers when data from a Bundle is accessed via View Tab and when downloaded
Fixed an issue showing stack trace when attempting to add Secondary Data to Data when not allowed
Fixed issue where the File Upload selection disappears after navigating back to Data after navigating to the Data History Tab
Fixed an issue where create folder session and get upload URL APIs are allowed for data accessible via an entitled Bundle
Fixed an issue where OpenAPI spec for delete project bundle API does not include proper "accept" header when using "Try It" functionality
Fixed an issue where write-access can be acquired for data accessible via an entitled Bundle
Fixed an issue where Project owner of an externally managed Project could not open Analysis Overview
Fixed an issue where multiple OS versions of the Service Connector are incorrectly downloaded at once when a single version is selected
Fixed an issue when selecting the "empty" option in the Operating System drop-down when downloading a connector
Fixed an issue where users with Contributor role and no data upload access can add data to samples without error
Added an event log entry for Service Connector log uploads initiated by Illumina Support
Flow: Fixed an issue to support Aborting an Analysis immediately after starting it
Flow: Fixed an issue when publishDir in Nextflow pipelines contains symlinks to data, those data are not included when uploading analysis outputs to a Project
Flow: Fixed an issue preventing closure of tabs in the Logs section for technical steps in the Analysis Details view
Flow: Cancel button added to the define CWL Pipeline selection dialogue window, and the order of buttons has been changed
Fixed an issue causing pipelines to stall for hours after all steps are completed
Flow: Fixed an issue where Nextflow pipelines fail with ‘pod 404 not found’ error
Base: Linked files from a Bundle can be inserted in a Base table
Base: Fixed an error on previewing Base tables that have been shared from a Bundle
Base: Fixed an issue where Base API to retrieve a list of Base jobs is not paging correctly
Bench: Fixed an issue where Bench Workspaces could not be opened with a double-click
Bench: After the “Save and Start” of a new Bench Workspace, the breadcrumb shows the actual name of the Workspace
Bench: Clarified error messages when creating new Workspaces with file permissions
Cohorts: Issue fixed where cohort is created during filtering process for RNA Expression on Apply
Cohorts: Issue fixed where order of items in the Survival Curve comparison legend are shuffled when reloaded
Cohorts: Issue fixed where long comparison names caused by a string concatenation between cohorts were causing the comparison feature to malfunction
Cohorts: Issue fixed where comparison name updates not reflected in the UI
Cohorts: Issue fixed where option still remains checked after Molecular filter removed from the right panel while on the project tab
Cohorts: Issue fixed where TCGA data not showing Molecular breakdown for selected genes
Cohorts: Issue fixed where data selections in the correlation feature where the second option visually shows half of the result in the UI
Cohorts: Issue fixed where "missense_and_ptv_all" missing from PHWAS/GWAS variant types
Features and Enhancements
Added format auto-detection for VCF (.hard-filtered.vcf.gz, .cnv.vcf.gz) and TAR (.v8.tar, .tar.gz) files
Pipeline documentation will be available if present when starting a new Analysis
Added Workflow Session event topics
Base: Shared databases are visible in Base tables. Shared databases are read-only and queryable tables originating from ICA modules such as Cohorts
Reduced pipeline spin up time delays due to shared files system provisioning
Added support for Illumina applications to add custom attributes to data, samples, workflow sessions, and analyses
API: "creatorId" for Project data records returned in the API
API: Added instructions for paging parameters on list endpoints in API specification
API: The edit and delete API endpoint for Terms of Use are no longer available
Updated ICA CLI to version 2.10.0
Fixed Issues
Fixed an issue allowing unauthorized users to access external-managed Project contents via a direct link
Fixed an issue in the Docker Repository view causing region text and check boxes to be misaligned
Fixed an issue causing storage configuration status to incorrectly report an error
Fixed an error preventing the creation of an AWS S3 storage configuration in a Project
Fixed an issue where the Service Connector status incorrectly showing ‘Currently Connected’ when the associated Connector is not connected
Fixed an intermittent error when clicking to expand a Connector rule details in the Connectivity View
Removed red exclamation mark icon next to "Unlinked Files" tab in the Samples view; improperly indicates an error
Fixed an issue where Bundle Terms of Use acceptance flag unset upon saving bundle version
Fixed an issue preventing users from viewing details of an Entitled Bundle
Fixed an issue resulting in an error message when creating Bundle Terms of Use with an existing version
API: Fixed an issue where the API endpoint to link data to a Bundle to responds with errors when data is part of a Reference Data
API: Fixed an issue causing 500 API error response when retrieving sample details
Fixed an issue causing incomplete list of users when adding users within the same tenant to the Project Team
API: Updated the API spec description for create upload URL and create inline view URL for Project Data
Fix to prevent editing of an existing active terms of use
Improved the Terms of Use UI flow for forcing re-acceptance upon update
API: Added a warning response for API to create custom events when event code exceeds 50 characters
API: Fixed the create data download URL API to correctly return a 503 instead of a 404 response when the underlying system is experiencing network-related issues
API: Added API endpoint to hide a Project
Fixed an issue with Boolean inputs incorrectly displayed as false on Analysis Details view (correct value is actually passed in)
Fixed an issue where CWL parameters containing only a white space are passed incorrectly to underlying pipeline runner
API: Fixed an issue where launch Nextflow analysis API returns a 500 error response when including duplicate parameters
Fixed an issue causing analysis failures due to docker image authorization check timeout
Fixed a shared storage issue causing analyses to remain In Progress until timeout
Fixed an intermittent issue causing Analyses to be stuck in progress with inability to view logs in the UI
Fixed an issue where Analysis output data is not immediately shown in the Analysis Details view upon completion
Fixed an issue causing Analysis failures showing “…Connection timed out (Connection timed out)…” in stdout logs
Fixed an issue causing CWL pipeline runner restarts during an Analysis which may lead to unexpected results
Fixed an issue resulting in stuck analyses when invalid Docker image is specified for a task; timeout reduced to 15 minutes
Fixed the name of Pipeline running logs tab from "workflow_runner" to "pipeline_runner" in Analysis Details view in the UI
Fixed missing start time on Analysis tasks and incorrectly reported costs for such tasks
Fixed an issue resulting in an error message when attempting to create a graphical CWL pipeline with a Tool with a Docker image that is not available in the Project's region
Fixed an issue causing errors when saving a new Bench Workspace
Fixed an issue causing out of sync Bench table and card views when a Workspace is created or deleted
Fixed an issue causing an error when starting Bench Workspace when enabling service account on the Workspace from tenant receiving the Workspace via cross-tenant share
Features and Enhancements
ICA Cohorts is released in production and added to the ICA user documentation
Clicking on a folder in the Data view will open the folder. Once a Folder is opened, the Folder details are available on the top right of the screen.
Added XML Configuration documentation and examples
General ICA robustness enhancements
Fixed Issues
Limit the Projects Overview view to 100 Projects with an indication that filtering must be used to see excluded Projects to improve UI performance
Fixed an issue preventing removal of permissions on externally managed projects
API – Fixed a 500 error response from Search Project Samples API when using a page token parameter
Fixed an issue allowing Bundles to be saved without mandatory Release Version when back-navigating from the Create New Bundle screen in the graphical UI
Fix to prevent data added to Bundle via secondary data upload on Data Details view
API – Fixed an error message when attempting to link data to a Bundle from a Project with Data Sharing disabled
API – Fixed an issue with tag filtering on getBundleSamples API
Fixed an error when filtering on size in Docker Repository where size filter is ignored
Fixed an issue where Analysis stuck in Aborting status when abort request sent during final stages of Analysis (such as Output, Upload)
Fixed an issue where intermittent Analysis failures are caused by system scaling errors under high load
Reduced repetitive Nextflow runner log entries from internal system logs to improve readability
API – Fixed an error in the Launch Nextflow Pipeline API when invalid parameter names are requested
API – Correctly support multi-value input parameters in launch Analysis APIs
Deprecated Tools are only visible when ticking the box "Show deprecated Tools" box in both table and card views
Fixed an issue preventing files linked to a Project from a Bundle as available to load into a Base Table
When exporting a query result from Base the file receives the name specified by the user and contains the header from the table
Removed unsupported AVRO and PARQUET formats from Available Formats to load data into Base tables
Fixed an issue where Bench service account API key is not revoked when a Workspace is stopped
Features and Enhancements
Added support for providing a nextflow.config file with a Nextflow Pipeline to improve pipeline building experience (see Nextflow Configuration docs)
Each tenant is limited to 100 in-progress Analyses at any given time to prevent resource exhaustion; additional analyses will be queued. Note: For assistance with planning batch jobs which require greater than 100 concurrent analyses, please contact Illumina Tech Support.
Added support for deep link URLs for direct navigation to child views in UI
API – Added an API to link Data to Projects in batch
API - Limit number of tags on data to 100; Excess tags in request will be ignored
API - Added "slug" attribute to Pipeline, Project, and Data asset types
API - Added support for the use of Project slug in place of ID in API request bodies
API - Added “Uniform Resource Name” (URN) attribute to Pipelines and Project object types
API - Added Project ID to notification payload for Pipeline Succeeded/Failed events
API - Added owning Project ID for data records in the getAnalysisOutput API response
API - Indicate originating Workflow/Workflow Session on Analyses returned via API, when applicable
Added support to use bundled Pipelines and data to create Analyses via API without first linking to the Project
Added support to automatically mount project data in Bench Workspace with ICA CLI
Fixed Issues
Renamed “Delete” to “Manage” in the Samples page dropdown menu
Removed the IGV button under View when selecting a VCF file
API – Fixed an issue causing the List Files API to incorrectly respond with an error when passing a folder or file path parameter value containing a leading or training space
Fixed an issue where files linked through a Project Connector will appear on root level as folder linking is not supported yet in the Project Connector
Fixed an issue when deleting files from S3 along with records from ICA via either volume delete (with purgeobjectstore=true) or folder delete API calls, if there are more than 1,000 items to delete the system deleted only the first 1,000 items, however, all the data records got deleted from ICA. In this case the data in the user-managed S3 bucket and ICA was out of sync.
Fixed an incorrect count displayed in UI when scheduling a download for a folder containing sub-folders
Fixed an issue where scheduling a download of a folder containing sub-folders creates duplicate data transfer operations
Disabled support for "fileNameExpression" on download rules; if provided when creating a download rule, the value will be ignored
Fixed an issue allowing public domain users to incorrectly access ICA
API – Fixed an issue where API to retrieve folder upload session intermittently responding incorrectly with error
API – Fixed a 500 response from the API to add a pipeline to a bundle when pipeline has already been added
Fixed an issue where user with "Viewer" permission for Flow in a project cannot link Samples from that Project
Return the correct error response when creating a new Nextflow or CWL pipeline through the API or CLI when an existing pipeline with the same name already exists
Improved UI messaging when setting up a Storage Configuration with permission error
Fixed an issue when attempting to edit a saved incomplete Tool (i.e., missing Docker image)
API - Fixed API to create notification channel to properly respond with error when attempting to create a duplicate notification channel
Fixed an issue where boolean Analysis input settings are incorrectly set to false when passed to the Pipeline runner
Fixed an issue when launching a Nextflow pipeline via API with multi-value inputs the order of the listed multi-value input is not preserved when passed to the pipeline runner
Fixed an issue where "Re-run" analysis button is broken for Nextflow pipeline analyses
When different versions of DRAGEN are run on the same host, reset the DRAGEN state so that the next task can correctly configure the FPGA compute instance
Fixed an issue causing intermittent analysis failures due to system error when evaluating permission to pull Docker image
Fixed empty start time attribute for Analysis tasks with a short duration (within a few seconds)
Fixed an issue where analysis start time is set when in fact the analysis is still queued; analysis start time will be set when compute resources (i.e., shared file system) have been requested
Fixed an issue where log file content for the pipeline runner is overwritten if system error occurs during an Analysis causing a pod restart; multiple log files may be created for the pipeline runner task if such an error occurs
Fixed an issue causing UI log stream view for Analysis tasks to incorrectly show errors or blank views
Fixed an issue causing incorrect Analysis task log to be displayed on the UI Logs view
Fixed an issue where analyses get stuck in Running status if input file paths contain leading or trailing spaces in path for analysis stuck in running status if input files contain leading or trailing spaces
Fixed an issue causing Analysis to be incorrectly marked as Success due to incorrect evaluation of exit code of an analysis task
Fixed an issue preventing the same file to be passed to multiple input fields when launching an Analysis
Workflow failures due to unavailable compute resources will fail quickly with a more descriptive error
Fixed an issue where analysis output data is not immediately shown in the Analysis Details view after the analysis completes
Removed the “Print” and “Save” buttons in the Analysis Report
Fixed an issue preventing Pipeline creation and Tool editing when an invalid Tool is saved in the Tool Repository
API – Fixed a 500 response from API to list Projects to properly return a 400 response when providing a search filter value with invalid characters
API - Fixed a 500 response from create download URL API to properly return a 400 response when data is not in "Available" status (ie, Partial/Archived)
Fixed an issue with loading data into a Base Table created directly in Snowflake with fields containing quotes
Fixed an issue where data was added to Base Table twice when multiple Scheduled Jobs run simultaneously
Fixed an issue with previewing Base Tables created directly via Snowflake
Fixed an issue where status is stuck at "In Progress" for Base Copy Table operations
Fixed an issue where Base Tables linked to a project from a bundle are shown as Views (read-only)
Fixed an intermittent issue causing duplicate data in Base Tables
API - Fixed a 500 response when sending a duplicate requests to load data for a Base Table to properly return a 400 response
API - Fixed a 500 response from API to load data into Base Table to correctly return a 400 response when attempting to load data with a type mismatch
API - Corrected Swagger UI responses
Fix error "Unable to save workspace" when attempting to start a Bench Workspace
Fixed Issues
Fixed an intermittent issue where analyses fail due to "Connection timed out" system error (seen in analysis error logs)
Added retry logic to decrease rate of pipeline failures due to internal system error (e.g. errors in pipeline runner log such as connection timeout, 404 pod not found, 500 errors)
Fixed an intermittent issue where "Cannot access folder" error is shown in pipeline error logs (most common in long-running analyses)
Fixed an issue resulting in analysis failures due to system error under high load
Fixed a compute instance configuration issue impacting DRAGEN analyses
Features and Enhancements
API - Added method to retrieve an analysis JSON input data
Added Analysis Details views in the UI
Added Analysis ID to the project analyses
Support deep links for direct navigation to Analysis Details view
Added support for multiple Nextflow versions when creating Nextflow pipeline
API - Added method to create multiple samples in batches
API - Improved response time performance of API that launches analysis with many data inputs
Fixed Issues
Repaired issue where Pipeline-generated Sample Metadata is not picked up by platform
Restored Time to delete function; now executes file deletes at the specified time
Bench example code now lists project data and pagination correctly
Restored ability to link secondary data to primary data
Fixed issue where batch sample creation jobs get stuck in PARTIALLY_SUCCEEDED status
Resolved issue where batch sample creation is not completed if data items could not be linked
API - Fixed issue where mountPaths were ignored when launching analyses via API
Restored function so that bundle contributors can link assets to their bundle
Restored data linking to a sample in an ICA project
Resolved issue where child workflow configurations prevent termination of parent workflow
Removed incorrect error reported in cases where previous failure occurred
Restored pipeline linking for bundle contributors
Improved UI messaging for Storage Configuration permission errors
API - Updated GET /api/projects/{projectId}/data to returns correct file details
Features and Enhancements
Re-labeled "Run" and "Runs" to “Analysis” and “Analyses” in the UI
Allow storage configuration to be disconnected from a hidden Project
Added support for a read-only view of Entitled Bundles and Bundles shared from another Tenant
Changed analysis timeouts for setting up environment, downloading inputs, pipeline runner (inclusive of all child steps), and finalizing output steps to 96 hours
API – Add idempotency key header to createAnalysis API
API – Analysis logs are available via getAnalysisSteps API; includes streamable via websocket endpoint during execution and downloadable S3 URL upon completion
JSON data with array field can be added in a Base table
Cohorts – Additional submenu items now available
CLI – Added support for setting the current Project ID to the value set in the ICA_PROJECT_UUID environment variable upon starting the CLI
CLI – Added support for mounting Project data to local operating system (Read-Only)
CLI – Added support for the outputParentFolderId flag for the ‘projectpripeline start’ command
CLI – Added support for customizing the filename/path of additional Pipeline files with the pipeline create commands
CLI – Added owning Project name/id attributes to the project data list output
CLI – Added owning Project name attribute to output for projectdata list command
CLI – Added commands to create and refresh API tokens (JWT)
CLI – Added support for CLI use without writing to disk by enabling API Key and Server flags for all commands
CLI – Added support for linking data to a Sample with projectdata upload command
Fixed Issues
Fixed an issue with UI stability leading to slow response times
Fixed an issue causing data uploads via browser to hang without error
Fixed an issue with UI automatically refreshing upon load when many Projects are present in the Projects view
Fixed an issue where Accept Share pop-up remains on screen after pressing to Accept the share
Fixed an issue with uploading nested folders using the Windows Connector
Fixed an issue where path separators are misinterpreted in upload rules for Windows Connectors.
Fixed an issue where modifying a local file uploaded (tracked) using the Connector causes a Connector failure
A User with Contributor Role for Flow and a Viewer Role for a Project can start an Analysis using input data from a Sample
API – Fixed an issue where sampleDeepDelete API fails with 500 error response when Sample contains a linked folder in archived status
Fixed an issue where ICA_STARTING_USER_TENANT_NAME is not set to the correct value on initial start
API – Fixed an issue where upload rules are not deleted via the deleteUploadRule API despite a success response
Sensitive fields in metadata models are now shown in italics
Fixed an issue where filtering on the Data view gives incorrect results
All Contributors in a Bundle are allowed to link assets to the Bundle
Content of a Bundle shared via an entitlement in another Tenant is now visible
Fixed an externally managed Project Team to only allow adding members with "Viewer" and "Allow Download" permissions
API – Fixed an issue with project sample APIs returning 500 error response when sample is not linked to the project specified in the request
API – Fixed issues with cursor-based paging on listData and listBundleData APIs returning expected results
Fixed an issue where a User can create empty folders inside an externally managed Project with Viewer permission
Fix auto-detection of AVRO and PARQUET data formats
API – Fixed an issue where the API to link Samples to a Bundle incorrectly returns a 204 response when the Sample is not in Available status and the Sample is not actually linked
API - Fixed an issue where changing the Project owner causes unexpected error responses from the listProjects, listProjectPermissions, and getProjectPermission APIs
API – Fixed an issue where deleteDownloadRule API returns 204 responses but does not delete the specified download rule
Fixed an issue causing delays in operations performed on data
Fixed an intermittent permission error when writing to /scratch from a Pipeline
Fix to make tags optional (not required) with Pipeline launch requests
Fixed an issue causing Analyses to be incorrectly set to SUCCEEDED status when failure occurs
API – Possible to create a Pipeline with a metadata model via API
Fixed an issue where logs are not streamed during Analysis
Fixed an issue when a User tries to start an Analysis with missing required parameters, the analysis is stuck in "awaiting input"
Fixed an issue preventing renaming a pipeline after attempting to save and receiving error that a pipeline exists with the same name
API – Fixed an issue when a Pipeline is launched via API with no input value provided for a nonmandatory input parameter the analysis status remains in "Awaiting Input" indefinitely
API – Fixed an issue causing unpredictable error responses from createAnalysis API when many Analyses are launched in rapid succession
API – Added operationID to API spec for the listAnalysisStorages and getAnalysisRawOutput APIs
Fixed an issue when Analyses are queued and Aborted before they are Running, the status is not correctly updated in the API/UI, and they appear stuck in “Aborting”
API – Fixed an issue with createAnalysis API causing error responses when launching many Analyses in rapid succession
Possible to load JSON in Base with a repeated field (array)
Tables created by the meta and admin data Schedules are updated after a sync
Fixed an issue causing Base Table field names to be changed from lowercase to UPPERCASE during a manual sync
Fixed an intermittent issue with Base Administrative/Meta-data Schedules creating files in project Data when schedules are run
Fixed an intermittent issue with Base Administrative/Meta-data Schedules creating files in project Data when scheduled
Fixed an issue where the User receives an obscure error message when creating a Tool from a Bench Workspace with an image name that already exists
Fixed an error message upon attempting to save Bench Workspace when input values are invalid
Fixed an issue where the Bench Workspace is temporarily inaccessible and stuck in Starting status
CLI – Fixed an issue with list samples command output limited to 20 unique records
Features and Enhancements
The value of the "Show Hidden Projects" filter on the Project Overview page will be saved for each user
Change analysis task log names to be more human-readable
Analyses will queue when system load is high
Add owning project tenant name (ICA_PROJECT_TENANT_NAME) and the tenant name of the user which last started the Workspace (ICA_STARTING_USER_TENANT_NAME) environment variables to Bench Workspaces
Add Cohorts URL environment variable to Bench Workspaces
API - Add support for Cross Origin Resource Sharing (CORS) to the API
Fixed Issues
Fixed an error message in UI when attempting to link a Sample that cannot be linked
API - Fix issue where API to update Sample metadata fields does not update the fields
Fixed error message when creating a Bundle with a name conflict
Fixed error message when a user attempts to view Bundle details without permission
API - Fix Project Bundle APIs to properly link, retrieve, and unlink entitled Bundles
Fixed to block ability to link data with Contributor role and no upload permission on the target project
Fixed an issue where the Owner of an externally managed Project cannot add Team members with Viewer or Download permission
Fixed field for providing an e-mail address when adding a Team member to a Project to be required
Removed temporary directory creation by service connector
Fix for configuring service connector to use proxy settings on Linux clients
Fixed an issue where data downloaded through Service Connector is incomplete
Fixed an issue causing errors in Service Connector when a directory is deleted in the UI but still present on the local machine. Data added to the folder on the local machine will not be uploaded if the corresponding folder is deleted in the UI.
Fix to display an error message when saving a Pipeline with invalid fields
Fix issue where CWL file/directory inputs are not mounted when provided within nested structures (ie, CommandInputRecordSchema, CommandInputArraySchema)
Fix to show error when attempting to run a graphical CWL pipeline with an invalid definition
Fix error when attempting to save a graphical CWL Pipeline with a Tool with an output folder and no specified format
API - Fix to ignore input JSON field when launching graphical CWL pipelines via API
API - Fix ignored default compute type (machine profile) defined in the Pipeline definition when launching a graphical CWL Pipeline via API
Fixed Base Query result export file format set to UNKNOWN
Fixed Base activity log to show Table associated with log event
API - Ignore unknown attributes in request bodies
API - Fixed API error responses formatted as HTML instead of JSON in error scenarios
API - Fix list analyses API not returning the "items[].analysisStorage" field
Features and Enhancements
Samples may only be linked when status is Available
Added Data ID to grid view in UI
Added ability to select and copy tag values to the clipboard
Added an indication of source domain in list of reference data sets when adding new data to Reference Data
Updated the service connector client icon
Added ability to filter projects by workgroup, including projects where user is a member of a workgroup added to the project
Added ability to specify output directory within a project when launching a pipeline
Added ability to Favorite Projects and filter the Projects view by Favorites
Remove obsolete commented text from auto-generated CWL Tool definitions
Scheduler displays the number of files to be loaded into a Base Table
Added an indication that Base Table previews display up to 50 columns and 100 rows
API - New endpoint to manage SecondaryData on Data objects
API - Added support for "mounts" parameter when launching a pipeline to provide input mount paths for file and folder inputs
API - Added fields to API error response schema: HTTP Method, User ID, Tenant Name, Tenant ID
API – Added new endpoints for retrieving Entitled Bundles and to manage Project Bundles
Fixed Issues
Docker images available via entitled bundles are listed in green text in the Docker Repository view
Fixed an issue in OpenAPI spec causing invalid cURL requests in 'Try it Out' functionality for the create upload URL API
Fixed an issue that resulted in empty Workgroup listing via the API and when adding Workgroups to a Project Team
Fixed an intermittent error when navigating to Bundles view
Fixed an issue where Tools available via Bundles are not displayed in the Tool Repository
Fixed an issue where example JSON generation for root Metadata Model fails when the root Metadata Model is modified
Fixed an issue where incorrect reporting of compute start times resulted in incorrect compute billing
Fixed an issue where Nextflow analyses get stuck "In Progress" when output folder is deleted during an ongoing pipeline run
Fixed an issue where incorrect casing on the scatter method 'dotproduct' in CWL auto-generated from graphical pipelines results in failures in pipelines with scatter-gather steps
CLI - Fixed an issue with startAnalysis APIs no longer accepting the 'Accept' header with value "application/problem+json,application/vnd.illumina.v3+json”
API/CLI - Fixed issue where graphical CWL pipelines launched via API/CLI do not use machine profiles (compute types) specified in the pipeline definition
Fixed an issue where pipelines within shared bundles could not execute across regions
Fixed an issue where exported Base Tables with null values populate a CSV with 'N' values instead of empty values
Fixed an error when loading data into Base Tables with an invalid number of columns
Fixed an issue causing View Results in Base Query History to return an expiration error
Fixed an issue that caused JSON import into Base Tables to fail when sub-field (record type) is used in the Table schema
Updated the error message when invalid SQL is used in a Base Query
Fixed an issue with JSON import to Base Tables failing when reference data fields are enabled
Fixed an issue where updated Base Table fields in the UI do not update the underlying Snowflake Table
Fixed an issue resulting in "Unable to create file error" message when creating a Tool in a Bench Workspace
Fixed an issue where Bench Workspaces cannot be deleted
Fixed an intermittent issue where Bench Workspaces unexpectedly fail in an error state
Updated messaging for users without access to Variant Viewer and Cohorts
API - Fixed an issue where the API did not accept multiple 'Accept' header values
API - Fixed an issue where the create folder upload sessions API returns date and time with incorrect formatting
API - Fixed the 500 response on APIs when the user has not yet accessed the UI
API - Fixed missing version header in OpenAPI spec for APIs to link a pipeline to a Project or Bundle
API - Fixed the 500 response from create project API when the 'storageConfigurationSubfolder' value is invalid
API - Fixed a 500 response from the API to link data to a bundle when linking a Folder with child data records
API - Fixed an issue where the API to create File data record creates the data record at the root of the project data when a trailing forward slash ('/') is missing after the folder name
API - Fixed an issue where API to retrieve pipeline reference sets does not display linked reference data
API - Fixed an issue where APIs to initiate analysis perform validation of required properties in request body
Fixed Issues
Fix issue where pipeline launch is blocked when a single file is used as the input to multiple pipeline inputs
Features and Enhancements
A Base query is now possible with null marker 'nan'
A user can search on Secondary Data tab
A user is allowed to run a pipeline from a shared project if the Billing mode is set as 'Tenant'
Add "application" field to Project API model to indicate whether a project is externally managed.
An error is shown when a user tries to build a workspace that is not running.
An error is thrown when the user uses the same name of an existing project connector rule in the same project
Base query can be exported as a view
Bug fix for allowing multiple files with same name but different paths to be passed as input; causes conflict when downloading inputs to pipeline filesystem, one file will be overwritten (random) causing unpredictable results
Data filters and columns now also apply to technical tags
Externally managed projects are now supported via API
Meta data schedule can be created on tenant level
New ICA notification for Data/Sample linked
Preview of the content of a Base table is available with null marker nan
Project connector permissions are aligned with manual linking permissions
Remove obsolete "BlueVantage" option from the Source drop down selections during Base schedule creation
Revert cwltool upgrade
Samples linked via a project connector are visible in the 'receiving' project
The 'Type of Files' field in the configuration of Upload Rules in a Connector is changed to 'Assign Format' so that its purpose is clear
The project connector rules are available across tenants when the user has Contributor rights
The user is able to use the filters 'Type' and 'Status' and sort the files
The yaks-scheduler used to check the pod-to-node fit, but will now defer to the kubernetes default-scheduler as the pod-placement authority. The assumption is that when the pod is not scheduled, there are no nodes available, and a node will be provisioned.
When a user links a sample from another project to the current project, the data will not be automatically linked to the current project as well. The user should explicitly add the data to the current project as well if the user wants to use it.
[Base] When selecting 'Include reference', you will see the record first and when you unfold the record, you will see the fields belonging to the record
When trying to populate the Sample Metadata, we now always use the Project Metadata Model if there is no Pipeline Metadata Model.
[API] Add ability to filter by region for list projects API
[API] Add psToken authorization type to enable creating JWT from psToken via the create token API
[Bench] Add ICA_PROJECT_UUID environment variable to the Jupyter Notebook Bench image.
Fixed Issues
Fix 400 error response when launching an analysis with a pipeline from a bundle
Fix Base Query results file export set to format UNKNOWN
Fix Bench pricing calculation accuracy
Fix for Base table export unexpected failure
Fix for auto-linking pipeline outputs files to a sample when input files provided are linked to the sample
Fix for failed Base table import when imported data contains "nan" (null marker) for a column of type integer
Fix for linking Base Query templates to Bundles
Fix for modifying Bench Workspace image unless Workspace is in Stopped state
Fix for more informative API error response when EULA has not been accepted
Fix for preserving sample data links for samples linked from shared bundles
Fix for retrieving download URL for data in an externally managed project.
Fix for silent error when running Base schedule on Administrative Data or Metadata
Fix intermittent UI issue where checkbox for file selection during docker image add is partially visible
Fix issue where Base schedule for Metadata does not work when source is set to "Tenant"
Fix issue where Base table schema definition changes from record to string after sync
Fix issue where Project Connectors cannot link samples where the source project is owned by a different tenant
Fix issue where an analysis cannot be started when a metadata model is defined for the pipeline
Fix issue where data cannot be removed from a Sample using the "Remove Data" button from the sample details view.
Fix issue where pipelines from linked bundles do not show as available selections in the "Generated by Pipelines" drop down when creating a Base schedule
Fix issue where samples cannot be updated after configuring a Project Connector rule with the sample.
Fix issue where scroll bar is not shown in the metadata models list view
Fix issue where secondary data is not shown in the reference data view
Fix issue where team members added to eternally managed projects cannot be granted Contributor permissions, and cannot link data to other projects
Fix issue where user cannot use sample filtering when samples and files are manually linked to their project and the files do not appear associated with their sample in that project
Fix issue where user is unable to see their input files when using sample filtering if they are linked from another project without selecting the flat view option
Fix issue where workgroups cannot be added to project team of externally managed projects with Contributor permission
Handle K8s api timeouts by retrying query.
Improve pipeline analysis startup times to approximately 10 mins maximum (times may vary)
[API] Fix 500 response for getProjectDataLinkedProjects API
[API] Fix error response when invalid API Key is provided for authorization
[API] Fix get bundles API error response when user has access to a project with a linked bundle that the user does not have access to
[API] Fix getProjectData API to return correct results when using the "id" query param
[API] Fix incorrect error response for Project Data API to create folder upload session
Features and Enhancements
the cwl workflow engine logs now also show debug logs, these includes the javascript expressions evaluated etc. these logs are similar to running cwltool --debug locally
Show "username" of pipeline creator in Pipelines overview view
Removed excessive logging from the "Audit Trail" logging.
Various performance improvements - reuse of connections to Base instances
A particular storage configuration should only be available during project creation in case it is your own storage configuration, or when the owner (or a tenant administrator) gives explicit access to it for other users in their tenant.
Bench - when trying to get an access token through the API, access was denied in case Base was not enabled within the project.
Base - 'You are not allowed to modify data' was shown to the user when clicking the Sync button as a Contributor
Base - importing data with format JSON in a table was not possible.
Files within a sample will only show up in case the data is also explicitly linked to the project/bundle.
Bundles - a user with Contributor access did not have the necessary access rights for managing all resource of the bundle.
Project billing modes can be 'project' or 'tenant'. In case 'tenant', the users from another tenant than the owner of the project was not able to upload files.
Within the Activation Codes view, activation codes for access to the Variants/Cohorts modules were not listed.
It was not possible to add a bundle at project creation. The project had to be saved before the bundle could be added.
The roles explanation in the Team page on a permission record did not show the entire information.
Various stability improvements - In case of a database outage, ICA was not able to automatically reconnect.
When managing docker images, a partial docker image was not allowed to be deleted nor updated resulting in a stuck image.
When exporting the contents of a grid from the UI, the exported file contained tenant ids instead of tenant names.
Base - the export of query results to a json format resulted in a failure in case the query ended with ';'
A non-tenant-admin user was not able to create a metadata model.
Base - exporting query results, specifying a custom delimiter did not work.
When a user clicks the bundle card in the "Bundles" view, and the user has no access to manage the bundle because it is shared with the user by another tenant, the following notification will be displayed: "Details for a bundle from another tenant are not accessible" instead of "Bundle details unavailable"
When a user links a sample from another project to the current project, the data will not be automatically linked to the current project as well. The user should explicitly add the data to the current project as well if the user wants to use it.
Various performance improvements - linking a folder with a lot files to another project or bundle timed out.
Base - loading metadata into a table using a schedule resulted in the same data being added over and over again.
When trying to populate the Sample Metadata, we now always use the Project Metadata Model if there is no Pipeline Metadata Model.
Limit usable character set for the user reference of an analysis run
When an analysis was aborted, the output files which were generated so far were not visible in the details of the analysis, but only in the data view.
Base - schedule metadata population in a table did not work.
Added support for Docker Images to be added to Bundles
Added Project Connectors in the project connectivity section to enable creation of automatic sample and data linking rules between projects.
Added Queue Date to analysis Logs view to indicate when the specified job was submitted to the underlying infrastructure.
Added support for externally-managed projects for Illumina software applications built on top of ICA to manage resources inside ICA projects while limiting tenant users' direct access to the project to read-only capacity.
Improved performance of API calls when using API Key for authentication
Added tenantName to API resource DTOs
Add API to abort analysis run
Added APIs to share a storageConfiguration, share a storageCredential, update a storageCredential secret
Added mime-type field to the DataFormat DTO for the data APIs
Added "objectEtag" field to DataDto for Data APIs to expose e-tag from the cloud provider object store
Added Base APIs to retrieve Jobs, retrieve Tables, and load data into Table
API - POST‚/api/projects/{projectId}/samples:search was missing request parameters to browse through samples in a paged manner.
API - POST /api/projects/{projectId}/analysis:cwl - tags were not persisted when starting a CWL analysis run
The PUT /project/{projectid}/data/{dataid} gives the user the impression that all fields can be updated on the data, but it is a different set for files as for folders. This distinction is not properly documented.
API - GET /api/projects/{projectId}/analyses - when the pipeline did not belong to the tenant, but the user still has access to it (eg: through a bundle), the analysis details could not be retrieved through the API.
API - GET /api/projects/{projectId}/analyses/{analysisId} returned the details of an analysis, even if the projectid did not match the actual project the analysis ran in.
API - GET /projects/{projectId}/samples/{sampleId} did not expose metadata information
API - GET /projects/{projectsId}/data presented the user with an unclear error message in case providing invalid request parameters.
API - An unclear error was presented in the problem response in case an invalid json was provided as request body.
API - An unlcear error was presented in the problem response when creating a project using invalid uuids in the request body.
API - Removed deprecated "sample matching strategy" endpoint
CLI - Added command to update project data
CLI - Added command to retrieve a list of project samples
CLI - Added command to create a project sample
CLI - Added command to list available data formats
CLI - API Key encryption modified to use local value on machine where API Key is configured to prevent hijacking config.
CLI - Added command to get details of a project sample
CLI - Added command to list data linked to a project sample
CLI - Added command to update a project analysis
CLI - Added command to update a project sample
CLI - Added command to unlink data from a sample
CLI - Added commands to delete a project sample
CLI - Added command to link data to a sample
CLI - Added command to complete a sample
CLI - Removed unnecessary flags on the config command.
CLI - For starting CWL and Nextflow pipelines, the flag 'storage-size' was ignored by CLI, and the default storage size of the pipeline was taken.
Fixed Issues
[bugfix] when CWL workflows failed or produced null as one of the outputs, all the outputs were incorrectly processed. in this case, the user may not be able to see the details of the analysis output in UI/API.
[Base] Fix error message when viewing a query from query history that no longer returns results
[bugfix] in some cases after aborting an analysis, it may remain stuck in aborting state indefinitely. Now more resources are explicitly cleaned to avoid this issue from happening.
Fixed bug when showing logfiles of steps in an analysis.
[bugfix] cwl workflow sometimes fails in spite of all steps in the workflows have succeeded. Made the workflow engine robust to detect and retry in this case
When linking a folder containing +2000 files to another project, only 2000 files were actually linked.
When a user did not choose a country when filling out the user account details, login to ICA could fail.
Fixed bug when handling custom output objects in CWL.
Fixes issue with compute type selection when launching CWL graphical pipelines
The Project > Team permissions showed a little pencil in front of the row indicating a modification. With this fix, the pencil will only show when there were actual modifications done to the row in particular.
API - All dates in API responses were formatted in 12 hours notation without AM/PM notation. This way a time in PM is presented as AM. With this fix, all time notations are in 24 hours.
When starting an analysis through the sample page, the start window is not closed after pressing start. Users would click it again, which would start the analysis multiple times. With this fix, the start window is closed after pressing start.
When starting an analysis fails because the backend service returns an error, the status of the analysis is "Awaiting Input". With this fix, the analysis goes to "Failed".
Base - in case the export of query results failed, an empty file was created and never cleaned up. With this fix, there are no leftovers in case of export failure.
When a sample was linked from another project to the current project, the details view of the sample shows all data linked to it, even of the current project does not have access to it. With this fix, only the data which is also linked to the current project will be shown on the details of the sample.
API - PUT /project/{projectid}/data/{dataid} update sometimes had to be done twice for it to persist. With this fix, all editable attributes can be updated at once.
Features and Enhancements
[CLI] Storage size can be entered by name
[CLI] Details of a project analysis can be retrieved by entering the analysis Id
[CLI] A list of metadata models can be retrieved
[CLI] A list of regions can be retrieved
[CLI] A list of storage configurations can be retrieved
[CLI] A pipeline can be linked to a project by using the name/id of the pipeline as an argument
[CLI] A pipeline can be unlinked from a project by defining the name/id as an argument
[CLI] A list of storage bundles can be retrieved
[API] Usability improvements for the list project data API:
Modify filePath query param to behave as prefix instead of exact match by default
Add filePathMatchMode query param
Add parentFolderPath query param
[API] Add API endpoint to refresh a JWT (POST /api/tokens:refresh)
[CLI] Add CLI command to create projects
[API] Support search by tags on samples search API
Renew macOS connector certificate
When a filter is applied and you select a folder, you can see the content from that folder without th* Connector rules for a project can only be changed when you are active in that project
It is possible to link more than 2000 files to a bundle or another project
Data coming from a project in another tenant can be linked when you are invite for collaboration
The error from a failed pipeline analysis is visible on the details page
Species and type are unique within a tenant, not across tenants
Sqs and sns test messages are aligned with the email test message
A collaborating user from another tenant is able to see the details view of a pipeline analysis run.
Bundle data can be used via API
When executing api/projects/{projectId}/data and a file already exist, there is an indication that thresponse.
Connector cancel button is renamed to delete to make its functionality more clearly
[CLI] Modify pipeline start command to use pipeline name or ID as an argument instead of a flag parameter
[API] Removed endpoint to POST on a sample metadata model
Bugs Fixed
[Base] Fix error message when viewing a query from query history that no longer returns results
[CLI] Fix for CLI upload command when uploading a folder with 0 bytes files
[API] Fix create bundle API to return error when bundle with existing name is attempted to be created
[API] Fix for non-functional If-Match header on PUT requests
[CLI] Fix folder upload "panic: close of closed channel" error
[API] Fix API for project creation returning success with invalid storage configuration provided
[Bench] Fix for docker build logs not showing when build is interrupted
Fix population of sample metadata from multiple analyses on the same sample
Fix error message for missing docker image on tool creation
[CLI] Fix pipeline create command to make pipeline code an argument instead of a flag parameter
Fix UI element for tags to remove active button appearance
Fix error when attempting to link pipeline with no pipeline selected
Fix error message when attempting to add field to Base table schema with invalid characters
Fix issue with analysis outputs files and logs not showing after analysis completion
Fix UI issue with pipeline selection in the Generated by Pipelines field of the view to create a Base s
Fix issue with a duplicate workspace being created while editing a workspace
Fix issue for table creation using schedule for metadata
Fix Base viewer role when applied to workgroups
Fixed Issue: optional parameters for Nextflow pipeline cause error when value not provided
Fixed issue when selecting data records to export from the Data overview page on a Bundle
Fix for Base saved queries not visible in shared project
Fix minor price calculation error (rounding) for Base storage
Features and Enhancements
API cursor-based and offset-set paging
Authorize calls using Illumina API key
Multiple reference data elements are possible when starting an analysis
APIs added for:
Account, User, Workgroup
Connector, Download Rule, Upload Rule
Data, Data Format, Data Transfer
Region
Download
Entitlement
Event Log
Project, Project Analysis, Project Data, Project Permission, Project Sample, Project Pipeline
Metadata model
Pipeline
Bundle
Notifications, Event Codes
Base
OpenAPI v3 API Spec published at https://ica.illumina.com/ica/api/swagger/index.html
Added sort parameter to several APIs
Custom event payload is defined as a json
Separate call to retrieve metadata model fields available
Add rclone temporary credentials support in the API to retrieve folder upload session details
Tokens API POST also contains basic authentication
Activation code endpoints are aligned with start analysis endpoints
Base templates cannot be deleted
Information can be written from Bench to Base
Viewer/Contributor permissions are introduced in Snowflake
Saved Template details can be edited
Access via Illumina authentication is available
Snowflake access token can be created
Added new Bundles feature for sharing data with other users
ICA allows the user to connect an S3 bucket as storage instead of an Illumina managed storage
An overview of all storage configurations is available
Secrets can be managed for the storage configuration
A new storage configuration can be added
The health status of a configuration can be monitored
A storage configuration can be deleted when not used in a project
Storage type and health are displayed in the overview
When selecting own storage ICA will return you a list of all BYOBs in the chosen region
A default bucket can be selected for every region
Data is automatically archived/deleted when the grace period (if present) is over
Data is archived/deleted automatically by a housekeeping mechanism
An admin user can change the date to be archived/deleted at any time
Data has a status
Data format is automatically detected at creation
Files can be grouped in a sample
Data can have additional linked projects
The data screen shows project data and data linked via a bundle
3 new columns are present in the data grid: "Owning Project", "Linked Projects" and "Linked Bundles"
The record is displayed in the data grid depending on the data being linked or not
Filter on Status with checkboxes on status values is possible
Total Data Size is available
Data can be linked/unlinked to/from a project
Only root parent files and folders are shown when opening the data grid
Deleted status does not exist anymore
Data from projects with Data Sharing disabled cannot be linked
Project is hidden instead of (soft-)delete
Metadata models can be created for linking to a project
Add support for Nextflow pipeline language
A pipeline input can have multiple formats
Output JSON for CWL analysis is shown
Ongoing logfiles are available for the user
You have two options to define a CWL pipeline: Graphical editor or code
Test connection button is available for SQS
Checkbox is added to toggle view between all rules or only your own rules
Notification channels can be validated
Subscription can be edited
Custom subscriptions are possible
Notifications can be received via SQS, SNS, Mail, and Webhook
All users have an overview of all the notifications used in their tenant
Notification channels are only visible for tenant admins
Notifications can have a filter expression to only receive specific events
A project administrator can grant permission to a single user on a project
A project administrator can invite somebody unknown to ICA to his project
A contributor can manage reference data
A project administrator can define who pays for using my project
A project administrator can invite a workgroup to a project
When multiple project roles are given to a user/workgroup the correct project role is applied
Team-page in projects and bundles are aligned
Reference data can exist in multiple regions
Species and type are being populated per tenant
Data can become reference data
Data tree shows all the regions in which the reference data is available