Illumina Connected Software is becoming Illumina BioInsight Platform. For this and other updates, you can
read more here
Ctrlk
Illumina KnowledgeIllumina SupportSign In
For the complete documentation index, see llms.txt. This page is also available as Markdown.

Data Manager

Overview

DataManager is a utility software from Illumina Connected Annotation software suite for user to easily pick and choose prebuilt annotation databases compatible with Illumina Connected Annotation. Previously, we provided the Downloader software that the user can run to download the latest annotation database automatically. With the new DataManager utility, users can choose the exact version of an annotation database they want.

DataManager utility requires credential file. Please follow the guide to generate this file from Prerequisite.

Downloader.dll utility will no longer be provided and instead DataManager.dll should be used.

Commands

Datamanager utility has several commands. Below are those commands:

List

The first command is list. This command is to list all available data annotation and version.

dotnet DataManager.dll list
---------------------------------------------------------------------------
DataManager                                         (c) 2024 Illumina, Inc.
                                                                     3.25.0
---------------------------------------------------------------------------

USAGE: dotnet DataManager.dll list [options]
List available data

OPTIONS:
      --credentials-file, -l <VALUE>
                             Path to credential file. The default file
                               location is ~/.ilmnAnnotations/credentials.json
      --ref, -r <VALUE>      Assembly reference (GRCh38/GRCh37)
      --dir, -d <VALUE>      Local directory
      --sources, -s <VALUE>  data source to show
      --help, -h             displays the help menu
      --version, -v          displays the version

##### Supported Annotation Sources #####
Basic Tier: DECIPHER, GME, GERP, DANN, REVEL, ClinGen, gnomAD, phyloP, TOPMed, DGV, 1000 Genomes, CliinVar, dbSNP, FusionCatcher, MITOMAP, MultiZ100Way

Professional Tier: PrimateAI(GRCh37), PrimateAI-3D(GRCh38), SpliceAI, COSMIC, OMIM.

##### Contact #####
Professional content licensing, feedback and technical support: annotation_support@illumina.com.

There are several parameters that you can pass.

  • Param --ref or -r is a required parameter. This is to filter data based on genome assembly reference. The value is either GRCh37 or GRCh38

  • Param --dir or -d is an optional parameter. This param is the path to your local data directory. If you provide this param, the list will mark data source version that you already have in the directory.

  • Param --credentials-file or -l is an optional parameter. This param is the path to your credential file.

  • Param --sources or -s is a filter based on data source. Without this param, the list will display all data source but only display the latest version available. If you use this param, you will see all available version for that particular data source.

Below are some example of the output:

  • List all latest data source version and providing local directory

  • Display all available GnomAD GRCh38 version

Create config

The second command is make-config. This command will generate a default config file that will be used to define which annotation data and which version to download and to use during annotation. The generated config file is basically a template which you can edit manually in case you want to download it with some specific data source and version.

There are several parameters for make-config command.

  • Param --credentials-file or -l is an optional parameter. This param is the path to your credential file.

  • Param --ref or -r is a required parameter. This is to filter data based on genome assembly reference. The value is either GRCh37 or GRCh38 or all to generate config for both assembly.

  • Param --dir or -d is an optional parameter. This is to generate a config file from your existing directory. It will read the content of your directory and generate config file. It is useful to generate config file from existing directory when you previously use old Downloader to download.

  • Param --outDir or -o is an optional parameter. This param is the path where the config file will be written to. If not specified, it will write to ~/.ilmnAnnotation. The file name would be {assembly}_annotation_config.json.

Below is example of the generated config file.

Download file

The third command available is download. This command will download the database files. Files that already exist in the output folder will be skipped (if the file size match). The DataManager will perform md5sum checks against the downloaded files to ensure file integrity and completeness. If you see that a file was marked Mismatched checksum in the Download Summary, try resolving any known root cause and run the DataManager again.

There are several parameters for the download command.

  • Param --credentials-file or -l is an optional parameter. This param is the path to your credential file.

  • Param --ref or -r is a required parameter. This is to filter data based on genome assembly reference. The value is either GRCh37 or GRCh38

  • Param --versions-config is an optional parameter. This is the file config that contains database and version that you want to download. If not specified, it will use ~/.ilmnAnnotation/GRChXX_annotation_config. json

  • Param --dir or -d is a required pqrameter. It specified the directory that will be used to store all downloaded files.

  • Param --thread or -t is an optional parameter. It is the number of thread that will be used to download file simultaneously. By default, the value is 4.

Below is an example where you download new release of reference file and Ensembl and RefSeq cache. The downloader will first detect that you have an existing reference and cache files with different version. The downloader then will delete those files and download the new files.

Note that adding additional sources so versions config file will only download those additional files.

Version validate

The fourth command is version-validate. This command is used to check whether the directory provided has all data source file and version that is defined in the version config.

There are several parameters that you can pass:

  • Param --versions-config is a required param to the version config file path that you want to check against your file.

  • Param --cache or -c is an optional param to the cache directory path that you want to use. This will be the same path that you pass when running Illumina Connected Annotation to annotate.

  • Param --ref or -r is a required param to the genome assembly reference file path that you want to use. This will be the same path that you pass when running Illumina Connected Annotation to annotate.

  • Param --sd is an optional param to the supplementary annotation directory path that you want to use. This will be the same path that you pass when running Illumina Connected Annotation to annotate.

This command will return successfully all data sources and matching versions that are defined in the version config exist.

The default path for credential file and version config is located in ~/.ilmnAnnotation which is the home folder of the user that run the command. In case you are using docker to run DataManager utility, make sure the credential file is mounted in the docker container. By default, docker container is run as root, so the default directory path the DataManager will use would be /root/.ilmnAnnotation. You can mount your local .ilmnAnnotation to this folder by adding parameter:

-v /local/path/.ilmnAnnotation:/root/.ilmnAnnotation

PreviousSAUtilsNextJist

Last updated

Was this helpful?