arrow-left

All pages
gitbookPowered by GitBook
1 of 6

Loading...

Loading...

Loading...

Loading...

Loading...

Loading...

Check Losslessness

DRAGEN ORA Compression is a lossless compression.

If you wish to verify that no data was lost during the compression of the fastq.ora file, compare the MD5 checksum of the decompressed fastq.ora file and the MD5 checksum of the decompressed fastq.gz file.

1. Compute the md5 checksum of the uncompressed FASTQ.ORA content as follows

md5sum <(orad myfile.fastq.ora --raw -c )

2. Compute the md5 checksum of the uncompressed FASTQ.GZ content as follows

md5sum <(gzip -d -c myfile.fastq.gz)

DRAGEN ORA Decompression v2.7

circle-exclamation

Warning Do not use this documentation for decompression on the DRAGEN Server. The DRAGEN Server has its own integrated DRAGEN ORA Decompression component.

DRAGEN ORA Decompression Software decompresses fastq.ora files into fastq.gz files. Fastq.ora files are generated using lossless compression technology as part of DRAGEN. Orad is the executable file that runs the DRAGEN ORA Decompression Software, which is a standalone piece of software.

With DRAGEN v4.3, compression of FASTQ files derived from supported non-human species and from human bisulfite data (methylated DNA application) is available. DRAGEN ORA decompression v2.7 allows for the decompression of FASTQ.ORA compressed with all supported references. Refer to the and sections.

The DRAGEN ORA Decompression Software is available for the following operating systems in three separated executable:

  • Linux

  • Mac

  • Windows

Decompression of FASTQ.ORA stored on local storage is supported on Linux, Mac, and Windows. Decompression of FASTQ.ORA stored on AWS S3 is only supported on Linux.

Decompression of FASTQ.ORA stored on Azure Blob storage is only supported on Linux.

When FASTQ.ORA is located on AWS S3 or Azure Blob storage, the decompression occurs on a streaming mode: the FASTQ.ORA file does not need to be fully transferred before decompression can start.

References Installation
Supported References

Software Installation

hashtag
Installation Requirements

The following are the minimum requirements for the DRAGEN ORA Decompression Software:

Component
Minimum Requirement

System memory

8 GB RAM

hashtag
Installing the DRAGEN ORA Decompression Software

Use the following steps to install the DRAGEN ORA Decompression Software once DRAGEN ORA Decompression has been downloaded from the .

hashtag
Linux or Mac

1. Extract the archive files using the following command:

2. Navigate to the Orad directory as follows:

3. Move the executable to your preferred location as follows:

4. Add Orad to your path as follows:

5. Move the oradata folder content into the home repository as follows:

To store the folder in a different location, use the following command:

When oradata has been moved in another location, you can:

  • either point to the reference by using the ORA_REF_PATH environment variable as follows:

  • or use the following command at decompression

hashtag
Windows

1. Extract the downloaded archive with a software that can handle gziped tarballs, such as 7-Zip. Right-click on the archive and select Extract with. The following two files are extracted:

  • orad.exe

  • refbin

The following steps use C:\Users\user1 as an example location. Change C:\Users\user1 to the location where you extracted the archive.

2. Open the Command Prompt application.

3. Set the environment variables to use the orad.exe and the refbin file with the set command or the setx command. The set command configures the variables temporarily (for the current console window) while the setx command configures the variables permanently.

4. Set the path to the orad.exe to the PATH environment variable as follows:

or

5. Set the path to the refbin file to an ORA_REF_PATH environment variable as follows:

or

Free disk space

From 2 GB to 15 GB depending on how many different specific references are needed for the decompression. Specific references are available for downloads in the ORA Support Sitearrow-up-right. For human data, 2 GB are required. The default human reference is already pre-installed within the software and does not need to be downloaded.

Compatible Linux distributions

  • CentOS 7 and later

  • Ubuntu 14 and later

  • Oracle 8 and later

  • Fedora 26

  • Debian 8

Compatible Mac distribution

Mac 10.15 and later (Apple silicon and Intel CPU processors)

Compatible Windows distribution

Windows 10 and later

ORA Support Sitearrow-up-right
tar -xzvf orad.2.7.0.linux.tar.gz (Linux) 
tar -xzvf orad.2.7.0.mac.tar.gz (Mac)
cd orad.2.7.0.linux
mv orad your_preferred_location/
echo 'PATH=$PATH: your_preferred_location/'» ~/.bashrc source ~/.bashrc
mv oradata ~
mv oradata ~/otherlocation/
export ORA_REF_PATH=~/otherlocation/oradata/
--ora-reference ~/otherlocation/
set PATH=%PATH%; C:\Users\user1
setx PATH=%PATH%; C:\Users\user1
set ORA_REF_PATH= C:\Users\user1
setx ORA_REF_PATH= C:\Users\user1

References Installation

For decompression of FASTQ.ORA derived from specific species/models, the reference that has been used at compression needs to be downloaded from the ORA Support Sitearrow-up-right. The whole reference database can also be downloaded.

circle-info

Info For the default human reference, there is no need to add this extra step. The default human reference is already included in the DRAGEN ORA Decompression Software.\

hashtag
Linux or mac

hashtag
Extract a specific reference

1. Move the downloaded archive to the oradata directory

2. Navigate to the oradata directory

3. Extract the archive file using the following command:

When the extraction of the specific reference is completed on Linux OS, the orad.2.7.0.linux folder should be structured as follows with example on gallus_gallus reference:

orad.2.7.0.linux

   |___orad

   |___oradata

          |___refbin

          |___Gallus_gallus

               |___refbin

               |___readme_gallus_gallus

circle-info

Note The oradatafolder can be moved to another location but should keep its structure.\

hashtag
Extract the full database

1. Move the downloaded archive to the orad.2.7.0.linux directory

2. Navigate to the orad.2.7.0.linux directory

3. Delete existing oradata folder

4. Extract the archive file using the following command:

When the extraction of the full database is completed the orad.2.7.0.linux folder should be structured as follows:

orad.2.7.0.linux

   |___orad

   |___oradata

     |___Homo_sapiens

          |___refbin

          |___readme_homo_sapiens

     |___Homo_sapiens_bisulfite

          |___refbin

          |___readme_homo_sapiens_bisulfite

     |___<Genus_specificname>

          |___refbin

          |___ readme_<Genus_specificname>

circle-info

Note The oradatafolder can be moved to another location but should keep its structure.\

hashtag
Windows

Extract the downloaded archive with a software that can handle gziped tarballs, such as 7-Zip. Right-click on the archive and select Extract with.

When a specific reference is downloaded, a folder with name <Genus_specifcname> is extracted. This folder contains the corresponding refbin and readme files. Move this folder in the location where orad and the default human refbin file has been extracted during the installation of the DRAGEN ORA Decompression Software procedure.

When the full database is downloaded, a folder with name oradata is extracted. This folder contains subfolders for each specific species which contains the corresponding refbin and readme files. Move this oradata folder in the location where orad and the default human refbin has been extracted during the installation of the DRAGEN ORA Decompression Software procedure.

mv yourdownloadpath/<genus_specificname>.tar.gz yourpath/oradata
cd yourpath/oradata
tar -xzvf <genus_specificname>.tar.gz (Linux)
tar -xzvf <genus_specificname>.tar.gz (Mac)
mv yourdownloadpath/oradata.tar.gz yourpath/orad.2.7.0.linux
cd yourpath/ orad.2.7.0.linux
rm -r oradata
tar -xzvf oradata.tar.gz (Linux)
tar -xzvf oradata.tar.gz (Mac)

Combine with downstream analysis

For a fully transparent usage of fastq.ora files (no changes in the command, no overhead, no additional footprint) with third-party bioinformatics software, DRAGEN ORA Helper Suite Software is recommended and available for download on the ORA Support Sitearrow-up-right. This software is only supported on Linux.

For a semi-transparent usage of fastq.ora files with third-party bioinformatics software, use DRAGEN ORA Decompression with the pipe function or process substitution. This method improves system performance by reducing reads and writing to the disk versus a full decompression step.

  • If the analysis software can read from the standard input, such as BWA, use the following command:

orad file.fastq.ora -c --raw | bwa mem humanref.fasta - > resu.sam

The -c option decompresses to standard output. The result is sent | to BWA, which uses the dash option - to read from standard input. This also works for paired reads, which uses the -p option of BWA to specify that the input contains interleaved paired reads.

  • If the analysis software cannot read from the standard input, you can use process substitution:

bwa mem humanref.fasta <(orad file.fastq.ora -c --raw) > resu.sam

For the file name, use the <( ) syntax containing the command that generates the file to standard output. In this case, orad with the -c option as in the command above. This method does not work when the third-party software checks the input file name or when the third-party software does not read the file sequentially.

circle-info

Info On Windows, replace orad with orad.exe\

Commands

hashtag
Required commands

Use the following commands to decompress the files:

orad FILE [args]or orad [args] FILE

A folder that contains FASTQ.ORA files can also be provided for batch decompression of FASTQ.ORA files at top level directory:

orad ./pathtofolder/ [args]

On Windows, replace orad with orad.exe.

orad.exe FILE [args].

To decompress FASTQ.ORA compressed with a reference other than the default human reference, ensure the specific reference is available locally.

No change applies in the command line. The decompression software automatically detects which species/model is used.

hashtag
Command Line Options

Command
Description

hashtag
Command Examples

Using Windows, replace orad with orad.exe. Example is: orad.exe myfile.fastq.ora --check

Command
Description

-v --version

Prints software version.

-i --info

Prints information about the compressed ORA file. The following information is included:

  • Software version used to compress the file.

  • Total number of sequences in the file.

  • Total number of bases in the file.

-c --stdout

Prints the decompressed file to the default standard output stdout. This is useful to share the results with another application without writing the decompressed file to disk.

-

Reads an input fastq.ora file from the default standard input stdin. This option is not supported for Windows OS.

-P --path

Sets the path location of the output file. The default file name is used. If a path is not specified, the file is created in the same location as the input file.This option overwrites the path if it is used with the -o option.

-o --out

Sets the name of the output file and the path when used with -P. The default is the name of the input fastq.ora file.

-N --name

Restores the original name saved in the fastq.ora file, at the time it was compressed to a fastq.ora file.

-I --interleave

Decompresses the output file into a single interleaved file. By default, when the input is a single interleaved fastq.ora file, the decompression automatically decompresses into two separate paired read files. If the interleaved fastq.ora file was generated with DRAGEN ORA Decompression v.4.0 or later, -interleaved is included in the file name.

--ora-reference

Changes the directory of the ORA reference file refbin.By default the software looks for the reference file in the following locations:

  • $HOME/oradata/refbin

  • $HOME/oradata/Genus_specificname/refbin

--check-ora-reference-path

Verifies if the ORA reference file refbin is accessible and prints the refbin path. Decompression does not occur when this option is added.

--quiet

Sets decompression to quiet mode. In quiet mode, nothing is written to the standard output and standard error. This mode is ignored when used with -c (--stdout) or -C (--check).

--empty-third-line

Outputs the third line in the FASTQ format, (which is, the line that starts with +) as an empty line. By default this line is preserved.

-r --repeat-header

Adds the read header to the third line in the FASTQ format, (which is, the line that starts with +).

-C --check

Checks the integrity of the specified ORA file. This option decompresses the file in memory and verifies that the checksum of the decompressed data and the checksum of the original data are identical. The decompressed file is not saved.

--raw

Decompresses the ORA file into an uncompressed FASTQ file. By default, the DRAGEN ORA Decompression Software decompresses to gzip format.

--rm

Deletes the input file after successful execution. By default, the input file is not deleted. This option is not supported for files in AWS S3 or Azure Blob Storage.

-t --threads

Sets the maximum number of threads allowed by the system. The default value is 8.

-f --force

Overwrites the output file without prompting. By default, if the output file exists, the software exits without overwriting.

-h --help

Prints help and exits.

orad myfile.fastq.ora --check

Checks the integrity of an ora file.

orad myfile.fastq.ora

Decompresses myfile.fastq.ora to myfile.fastq.gz

orad myfile.fastq.ora --info

Prints information summary of an ora file.

orad -c --raw myfile.fastq.ora | head

Prints the first lines of the corresponding .FASTQ file in the terminal.

orad --check-ora-reference-path

Verifies the accessibility of the ora reference file and print its path.

If the file contains interleaved data.

  • The original name of the file if it was saved in the fastq.ora file.

  • The name of the species/reference it has been compressed with, and the related xxhash checksum of the reference. When nothing is specified the FASTQ.ORA file has been compressed with the default human reference.

  • This option is not supported for files in AWS S3 or Azure Blob Storage. Although the ORA file format supports concatenation of fastq.ora files, using this command on a concatenated fastq.ora file prints erroneous information.

    ./oradata/refbin

  • ./oradata/Genus_specificname/refbin

  • If you specified a location in the environment variable, the software also looks in the location ORA_REF_PATH. For example, set with export ORA_REF_PATH=/some/path/.