Why do I get "?" for FDR p-values in my Deseq2 result?

When a feature (gene) has low expression, it will be filtered by automatic independent filtering. To avoid this, you can either filter features to exclude low expression features before Deseq2, or in the Deseq2 advanced options, choose apply independent filtering setting. Details about independent filtering can be found at the Deseq2 documentation.

Click here for troubleshooting other differential analysis models and "?" results

What is fold change?

Fold change indicates the extent of increase or decrease in feature expression in a comparison. In Partek Flow, fold change is in linear scale (even if the input data is in log scale). It is converted from ratio, which is the LSmean of group one divided by LSmean of group two in your comparison. When the ratio is greater than 1, fold change is identical to ratio; when the ratio is less than 1, fold change is -1/ratio. There is no fold change value between -1 to 1. When ratio/fold change is 1, that means there is no change between the two groups.

Log ratio option in Partek Flow is converted from ratio, this is a value comparable to log fold change in some other tools.

Can I label a Volcano plot with gene names?

Yes, go to Style in the Data Viewer and make sure Gene name is selected under "Labeling". Next, go to the in plot selection tools (right side of the graphic) and use any of the selection tools to select the cells that you would like to label. You can use ctrl or shift to select multiple populations at once. For more information on the Volcano plot click here.

In Volcano plot, what is inconclusive group mean?

By default, Flow is using the p value <= 0.05 and |fold change|>=2 as the significance cutoff. If genes meet both p value and fold change cutoff, they are significantly up or down regulated genes. If they only meet one criteria, they are called inconclusive. If genes won't pass either criteria, they are not significant. Click on the Statistics button in the Configure section in the left control panel, you can change the cutoff. Click on the Style button to change the color of significance categories.

What is the difference between FDR and FDR step up?

FDR is the expected proportion of false discoveries among all discoveries. FDR Step-up is a particular method to keep FDR under a given level, alpha, that was proposed in this paper. In Partek Flow, if one calls all of the features with p-values 0.02 or less, the FDR is less or equal to 0.41.

How to perform a paired t-Test in Flow

You should have at least the following two attributes in the Metadata, treatment (including two subgroups) and subject ID (to pair the two samples). When performing differential analysis, choose ANOVA and include both attributes into the ANOVA model, the two-way ANOVA is mathematically equivalent to paired t-Test.

Can I compare one attribute at a time versus all of the others combined?

Yes, you can use the Compute biomarkers task to compare one subgroup at a time to all of the others combined. An alternative option is to set up the differential analysis model in this way; for more information please see the information here for each model.

I downloaded gene counts from the output data node generated by the Quantify to annotation model task, why can't I find my genes of interest?

In the Quantifying to an annotation model dialog, by default, Partek Flow filters features based on the total count across all of the samples and features with a total count greater than 10 will be reported. If you want to report all of the genes in the annotation file, change the Filter features value to 0.

Partek software enables researchers to easily perform genomic data analysis without ever needing to write a single line of code or sacrificing statistical power or advanced functionality. From alignment to pathway analysis, Partek provides a seamless, integrated analysis solution on a single platform that provides the power of a cloud or cluster when needed, and the convenience of desktop software for less compute intensive tasks.

Here you will find documentation on how to use and administer our products.

Partek Flow is a web-based application for genomic data analysis and visualization. It can be installed on a desktop computer, computer cluster or cloud. Users can then access Partek Flow from any browser-enabled device, such as a personal computer, tablet or smartphone.

Read on to learn about the following installation topics:

• Minimum System Requirements

• Single Cell Toolkit System Requirements

• Single Node Installation

• Single Node Amazon Web Services‎ Deployment

• Multi-Node Cluster Installation

• Creating Restricted User Folders within the Partek Flow Server

• Updating Partek Flow

• Uninstalling Partek Flow

• Dependencies

• Docker and Docker-compose

• Java KeyStore and Certificates

• Kubernetes

Additional Assistance

If you need additional assistance, please visit our support page to submit a help ticket or find phone numbers for regional support.

How to create a project?

To create a project, you first need to , and then import the files into your project using the import data wizard, here is the and .

Can I change my user avatar?

Yes, navigate to and click the "Change image" button. Do this by clicking your avatar at the top right corner of the interface, select , then choose .

How do I add and use my own lists?

Click your avatar in the top right corner of the Partek Flow interface, choose in the menu, and select from the left panel of the section. Lists can also be generated from result tables using the "Save as managed list" button. For more information please click .

Can I repeat a task and everything downstream of it, while changing only one/a few parameters?

Yes, click on the rectangular task that you want to change the parameters. On the context-specific menu on the right, under Task actions, select ‘Rerun with downstream tasks’, this will bring you to the task set up page where you can edit the parameters for the task, then click Finish to run the task with the new parameters. The tasks downstream of it will be initiated automatically.

What can I use to identify cells that are actively expressing genes within a gene list?

Use to identify cells with active gene sets; this task calculates a value for each cell by ranking all genes by their expression level in the cell and identifying what proportion of the genes from the gene list fall within the top 5% (default cutoff) of genes. An alternative option is to use the Gene score for a feature list to select and filter populations based on the distribution; .

Can I build and use pipelines for my analysis?

Yes, click on on the bottom of the tab dashboard. This will help you import either our hosted pipelines or your own saved pipeline which can be found under -> -> . Click for steps to save and run a pipeline. For more information related to navigating pipelines click .

How do I classify cells?

Classification in Partek Flow can be performed manually or with automatic cell classification which is explained in more detail . Users often want to classify cells by gene expression threshold(s), for details on classification by marker expression click . Automatic classification needs to be performed on a non-normalized single cell data node; once complete, then use this classification in visualizations and tasks. You may choose to perform and to help identify biomarkers that can then be used to identify the clusters and we also provide hosted lists for different cell types.

My server is full, how do I make more space?

How do I add library files if I am not studying human or mouse?

The Annotation model is a file containing feature location. This file can be used to quantify to annotation model in RNA-Seq analysis, or annotate variant or peaks in a DNA-Seq or ATAC-Seq/ChIP-Seq data analysis pipeline. The file format should be .gtf/.gff/.bed.

Are Genome coordinates 1-based or 0-based?

Genome coordinates for annotation models stored in Partek Flow are 1-based, start-inclusive, and stop-exclusive. This means that the first base position starts from one, the start coordinate for a feature is included in the feature and the stop/end coordinate is not included in the feature. These are the genome coordinates that are printed in various task reports and output files when an annotation model is involved in the task. When custom annotation files are added to Partek Flow, the genome coordinates are converted into this format. The coordinates are converted back if necessary for a specific task. shows how the genome coordinates vary between different annotation formats.

Can I add transgenes to my reference files?

Yes, to add transgenes (including gfp or related) to the references files, first choose an assembly, create the transgene reference, and merge the references together (e.g. combine mm10 with dttomato). This is the same process for the annotation file.

How do I export data from the result nodes?

Why can’t I find the RPKM method on the normalization menu (I see FPKM)?

Why don't you have RPM in your normalization?

RPM (reads per million) is the same as Total Count. Please use Total Count.

What is a canonical transcript?

For genes with multiple transcripts, one of the transcripts is picked as the canonical transcript. Based on the UCSC definition from the table browser,

knownCanonical - identifies the canonical isoform of each cluster ID, or gene. Generally, this is the longest isoform.

we define the canonical transcript as either the longest CDS (coding DNA sequence) if the gene has translated transcripts, or the longest cDNA.

Why are there decimal values in the Partek E/M quantification output?

The Partek E/M quantification algorithm can give decimal values because of multi-mapping reads (the same read potentially aligning to multiple locations) and overlapping transcripts/genes (a read that maps to a location with multiple transcripts or genes at that location). In these scenarios, the read count will be split.

For example, if a read maps to two potential locations, then that read contributes 0.5 counts to the first location and 0.5 counts to the second location. Similarly, if a read maps to one location with two overlapping genes, then that read contributes 0.5 counts to the first gene and 0.5 counts to the second gene.

If you need to remove the decimal points for downstream analysis outside of Partek Flow, you can round the values to the nearest integer.

Why is the number of variants listed in the variant reports and summarize cohort mutations report different?

For variants with multiple alternative alleles, the variant has one row for all alternative alleles, while the summarize cohort mutations report lists each alternative allele on a separate rows. The number of variants listed at the top of the each report is calculated from the number of rows in the report.

General

How to create a project?

To create a project, you first need to , and then import the files into your project using the import data wizard, here is the and .

Can I change my user avatar?

Yes, navigate to and click the "Change image" button. Do this by clicking your avatar at the top right corner of the interface, select , then choose .

How do I add and use my own lists?

Click your avatar in the top right corner of the Partek Flow interface, choose in the menu, and select from the left panel of the section. Lists can also be generated from result tables using the "Save as managed list" button. For more information please click .

Can I repeat a task and everything downstream of it, while changing only one/a few parameters?

Yes, click on the rectangular task that you want to change the parameters. On the context-specific menu on the right, under Task actions, select ‘Rerun with downstream tasks’, this will bring you to the task set up page where you can edit the parameters for the task, then click Finish to run the task with the new parameters. The tasks downstream of it will be initiated automatically.

What can I use to identify cells that are actively expressing genes within a gene list?

Use to identify cells with active gene sets; this task calculates a value for each cell by ranking all genes by their expression level in the cell and identifying what proportion of the genes from the gene list fall within the top 5% (default cutoff) of genes. An alternative option is to use the Gene score for a feature list to select and filter populations based on the distribution; .

Can I build and use pipelines for my analysis?

How do I classify cells?

My server is full, how do I make more space?

How do I add library files if I am not studying human or mouse?

The Annotation model is a file containing feature location. This file can be used to quantify to annotation model in RNA-Seq analysis, or annotate variant or peaks in a DNA-Seq or ATAC-Seq/ChIP-Seq data analysis pipeline. The file format should be .gtf/.gff/.bed.

Are Genome coordinates 1-based or 0-based?

Genome coordinates for annotation models stored in Partek Flow are 1-based, start-inclusive, and stop-exclusive. This means that the first base position starts from one, the start coordinate for a feature is included in the feature and the stop/end coordinate is not included in the feature. These are the genome coordinates that are printed in various task reports and output files when an annotation model is involved in the task. When custom annotation files are added to Partek Flow, the genome coordinates are converted into this format. The coordinates are converted back if necessary for a specific task. shows how the genome coordinates vary between different annotation formats.

Can I add transgenes to my reference files?

Yes, to add transgenes (including gfp or related) to the references files, first choose an assembly, create the transgene reference, and merge the references together (e.g. combine mm10 with dttomato). This is the same process for the annotation file.

How do I export data from the result nodes?

Why can’t I find the RPKM method on the normalization menu (I see FPKM)?

Why don't you have RPM in your normalization?

RPM (reads per million) is the same as Total Count. Please use Total Count.

What is a canonical transcript?

For genes with multiple transcripts, one of the transcripts is picked as the canonical transcript. Based on the UCSC definition from the table browser,

knownCanonical - identifies the canonical isoform of each cluster ID, or gene. Generally, this is the longest isoform.

we define the canonical transcript as either the longest CDS (coding DNA sequence) if the gene has translated transcripts, or the longest cDNA.

Why are there decimal values in the Partek E/M quantification output?

The Partek E/M quantification algorithm can give decimal values because of multi-mapping reads (the same read potentially aligning to multiple locations) and overlapping transcripts/genes (a read that maps to a location with multiple transcripts or genes at that location). In these scenarios, the read count will be split.

For example, if a read maps to two potential locations, then that read contributes 0.5 counts to the first location and 0.5 counts to the second location. Similarly, if a read maps to one location with two overlapping genes, then that read contributes 0.5 counts to the first gene and 0.5 counts to the second gene.

If you need to remove the decimal points for downstream analysis outside of Partek Flow, you can round the values to the nearest integer.

Why is the number of variants listed in the variant reports and summarize cohort mutations report different?

For variants with multiple alternative alleles, the variant has one row for all alternative alleles, while the summarize cohort mutations report lists each alternative allele on a separate rows. The number of variants listed at the top of the each report is calculated from the number of rows in the report.

Visualization

How do I order my heatmap by the cell types?

How do I display UMAP for each sample in the Data Viewer?

For a multi-sample project, all of the downstream tasks will be run separately if 'Split by sample' was checked when performing the PCA task. Visualization of different samples can be displayed by 'Sample' using the 'Misc' section in the Axes card. To show different samples side by side, one can click 'Duplicate plot' first, then use the 'Sample' option to switch the samples.

Can I visualize fold change values on a heatmap without using a z-score?

Why don't I see Flip mode on the heatmap? Why can't I download all of the data after zooming?

The Flip mode and download all data options are disabled if there are more than 2.5 million values (rows x columns) in the heatmap.

How to label gene names on volcano plot?

By default, genes are selected if the p-value is <=0.05 and |fold change| >=2 and when the number of selected genes is less than 2000 genes, they will be labeled. You can click on Style button in Configure section, choose a gene annotation field from the Label by drop-down list to change the label. If you number of selected genes is select less than or equal to 100, Partek Flow will try to spread out labels as much as possible to clearly display the labels. If number of selected genes is more than 100, labels will be next to the selected genes, there will be overlaps where genes are close together. If there are more than 2000 genes selected, no label will be displayed.

If you click any blank space, you can turn off select and use different selection mode button on the vertical bar on the upper-right corner of the plot to manually select dots on the plot.

Statistics

Why do I get "?" for FDR p-values in my Deseq2 result?

What is fold change?

Fold change indicates the extent of increase or decrease in feature expression in a comparison. In Partek Flow, fold change is in linear scale (even if the input data is in log scale). It is converted from ratio, which is the LSmean of group one divided by LSmean of group two in your comparison. When the ratio is greater than 1, fold change is identical to ratio; when the ratio is less than 1, fold change is -1/ratio. There is no fold change value between -1 to 1. When ratio/fold change is 1, that means there is no change between the two groups.

Log ratio option in Partek Flow is converted from ratio, this is a value comparable to log fold change in some other tools.

Can I label a Volcano plot with gene names?

In Volcano plot, what is inconclusive group mean?

By default, Flow is using the p value <= 0.05 and |fold change|>=2 as the significance cutoff. If genes meet both p value and fold change cutoff, they are significantly up or down regulated genes. If they only meet one criteria, they are called inconclusive. If genes won't pass either criteria, they are not significant. Click on the Statistics button in the Configure section in the left control panel, you can change the cutoff. Click on the Style button to change the color of significance categories.

What is the difference between FDR and FDR step up?

How to perform a paired t-Test in Flow

You should have at least the following two attributes in the Metadata, treatment (including two subgroups) and subject ID (to pair the two samples). When performing differential analysis, choose ANOVA and include both attributes into the ANOVA model, the two-way ANOVA is mathematically equivalent to paired t-Test.

Can I compare one attribute at a time versus all of the others combined?

I downloaded gene counts from the output data node generated by the Quantify to annotation model task, why can't I find my genes of interest?

Biological Interpretation

What is the difference between GSEA and Gene Set Enrichment?

In Partek Flow, GSEA should be performed on a sample/cell and feature matrix data node (e.g. normalization count data). GSEA is used to detect a gene set/a pathway which is significantly different between two groups. Gene set enrichment should be performed on a filtered gene list; it is used to identify overrepresented gene set/pathway based the filtered gene list using Fisher's exact test. The input data is a filtered list using gene names.

What is the enrichment score shown in the Gene Set Enrichment report?

The enrichment score shown in the enrichment report is the negative natural log of the enrichment p-value derived from Fisher Exact test. The higher the enrichment score, the more overrepresented our list of genes in the gene set of a GO/pathway category.

In KEGG pathway, genes can be colored by Fold change and p-value etc, how are the gene statistics calculated?

For Gene set enrichment analysis, only genes from the input data node (filtered gene list) will be colored in the KEGG pathway gene network, using the statistics in the data node.

During GSEA (or Gene set ANOVA) computation, we also perform ANOVA on each gene based on the attributed selected independent from GESA computation (at gene set level). The results of ANOVA is only used to color the genes in the KEGG gene network. If GSEA is computed using another other database, e.g. GO, we don't compute ANOVA on each gene since GO databased doesn't have gene network information.

When should I use GSEA or Gene set ANOVA?

What is the difference between GSEA and Gene Set Enrichment?

In Partek Flow, GSEA should be performed on a sample/cell and feature matrix data node (e.g. normalization count data). GSEA is used to detect a gene set/a pathway which is significantly different between two groups. Gene set enrichment should be performed on a filtered gene list; it is used to identify overrepresented gene set/pathway based the filtered gene list using Fisher's exact test. The input data is a filtered list using gene names.

What is the enrichment score shown in the Gene Set Enrichment report?

The enrichment score shown in the enrichment report is the negative natural log of the enrichment p-value derived from Fisher Exact test. The higher the enrichment score, the more overrepresented our list of genes in the gene set of a GO/pathway category.

In KEGG pathway, genes can be colored by Fold change and p-value etc, how are the gene statistics calculated?

For Gene set enrichment analysis, only genes from the input data node (filtered gene list) will be colored in the KEGG pathway gene network, using the statistics in the data node.

During GSEA (or Gene set ANOVA) computation, we also perform ANOVA on each gene based on the attributed selected independent from GESA computation (at gene set level). The results of ANOVA is only used to color the genes in the KEGG gene network. If GSEA is computed using another other database, e.g. GO, we don't compute ANOVA on each gene since GO databased doesn't have gene network information.

When should I use GSEA or Gene set ANOVA?

Both methods should be performed on a normalized matrix data node, and requires gene symbol in feature annotation. Both methods are detecting a differentially expressed Gene set (pathway) instead of each individual gene. The algorithms are different. GSEA is a popular method from the Broad institute. Gene Set ANOVA is based on generalized linear model, here are the details.

How do I order my heatmap by the cell types?

If you would like specific groups (e.g. cell types) in a certain order, do not perform Hierarchical clustering on these cells and instead choose to assign order, then use click and drag to reorder the groups. If you want to remove a group, you can choose to exclude this group in the filtering section. You can still perform Hierarchical clustering on the features if you would like to. Hierarchical clustering will force the heatmap to cluster and you would need to click the dendrogram nodes to switch the order. Click here for more information.

How do I display UMAP for each sample in the Data Viewer?

For a multi-sample project, all of the downstream tasks will be run separately if 'Split by sample' was checked when performing the PCA task. Visualization of different samples can be displayed by 'Sample' using the 'Misc' section in the Axes card. To show different samples side by side, one can click 'Duplicate plot' first, then use the 'Sample' option to switch the samples.

Can I visualize fold change values on a heatmap without using a z-score?

Yes, the default settings can be modified by clicking "Configure" in the Advanced settings during task set-up, then change the "feature scaling" option to "none" to plot the values without scaling. For more information related to the heatmap click here.

Why don't I see Flip mode on the heatmap? Why can't I download all of the data after zooming?

The Flip mode and download all data options are disabled if there are more than 2.5 million values (rows x columns) in the heatmap.

How to label gene names on volcano plot?

By default, genes are selected if the p-value is <=0.05 and |fold change| >=2 and when the number of selected genes is less than 2000 genes, they will be labeled. You can click on Style button in Configure section, choose a gene annotation field from the Label by drop-down list to change the label. If you number of selected genes is select less than or equal to 100, Partek Flow will try to spread out labels as much as possible to clearly display the labels. If number of selected genes is more than 100, labels will be next to the selected genes, there will be overlaps where genes are close together. If there are more than 2000 genes selected, no label will be displayed.

If you click any blank space, you can turn off select and use different selection mode button on the vertical bar on the upper-right corner of the plot to manually select dots on the plot.

Overview

This guide gives the basics of Partek Flow usage. Partek Flow can be installed in either a server, computer cluster or on the cloud. Regardless of where it's installed, it can be viewed using any web browser. We recommend using Google Chrome.

This guide covers:

• Overview

• Starting a new project

• Basic Partek Flow layout

• Saving visualizations

• Downloading your data

• Partek Flow in action

Logging in to your Partek Flow account will bring up the Home page (Figure 1). This page will show recent activities you've performed, recent projects you've worked on and pertinent details about each project.

Starting a new project

Uploading your dataset

Select the type of data (Single cell, Bulk, Other), choose the assay type, and select the data format. Partek Flow accepts various data types. Use the Next button to proceed with import.

There are three ways you can upload the data:

1. From your Partek Flow server (click here for more information)

2. From a URL

3. From a GEO / ENA Bioproject (click here for more information)

Because genomics datasets are generally large, it is ideal to have the data copied in a folder directly accessible to the Partek Flow server. Make sure that the directory has the appropriate permissions for Partek Flow to read and write files in that folder. You may wish to seek assistance from your system administrator in uploading your data directly.

Select the files you would like to create samples from. Once they've been created, assign the corresponding sample attributes for each sample using the Metadata tab. The most efficient way to assign sample attributes is by clicking Assign sample attributes from a file and uploading a tab delimited text file. The file should contain a table with the following:

• The first row lists the attribute names (e.g. Treatment, Exposure) and

• The first column of the table lists the sample names (the sample names in the file must be identical to the ones listed in the Sample name column in the Data tab)

• List the corresponding attributes for each sample in the succeeding columns

Basic Partek Flow layout

The Analyses Tab

After samples have been added and associated with valid data files, a data node will appear in the Analyses tab (Figure 3). The Analyses tab is where you can invoke tasks, using the context sensitive menu on the right, and view the results of your analysis.

To add more data use the Add data task in the menu on the right or Add data in the Metadata tab. Once a task is performed, data can no longer be added to the project.

Data and task nodes

The Analyses tab contains two elements: data nodes (circles) and task nodes (rounded rectangles) connected by lines and arrows . Collectively, they represent a data analysis pipeline (Figure 4).

Performing tasks

Clicking a data node brings up a context sensitive menu on the right (Figure 5). This menu changes depending on the type of data node. It will only present tasks which can be performed on that specific data type. Hover over the task to obtain additional information regarding each option.

Depending on the task, a new data node may automatically be created and connected to the original data node. This contains the data resulting from the task. Tasks that do not produce new data types, such as Pre-alignment QA/QC, will not produce an additional data node.

To view the results of a task, click the task node and choose the Task report option on the menu.

Saving visualizations

Downloading your data

Data associated with any data node can be downloaded by clicking the node and choosing Download data at the bottom of the task menu (Figure 8). Compressed files will be downloaded to the local computer where the user is accessing the Partek Flow server. Note that bigger files (such as unaligned reads) would take longer to download. For guidance, a file size estimate is provided for each data node. Downloaded files can be seamlessly imported in Partek® Genomics Suite®.

Partek Flow in action

Watch a webinar of how to set-up a simple RNA-Seq project.

Additional Assistance

If you need additional assistance, please visit our support page to submit a help ticket or find phone numbers for regional support.

Ready to start work on your Partek Flow hosted trial? This page has some helpful videos to get you started!

• Uploading Your Data to a Hosted Instance of Partek Flow

• How to Get Started on your First Project

Don't have a trial server yet? Request one on our website.

Uploading Your Data to a Hosted Instance of Partek Flow

This short video shows you how to import your data into a hosted instance of Partek Flow. Adjust your device's volume for optimal sound.

Note: When upload large size of data, it might take a while, please turn off the computer sleep mode settings!

How to Get Started on your First Project

In this short video, we'll give you an overview of the interface and how to get started with your analysis.

Additional Assistance

If you need additional assistance, please visit our support page to submit a help ticket or find phone numbers for regional support.

Web Browser Requirements

Regardless of whether Partek Flow is installed on a server or the cloud, users will be interacting with the software using a web browser. We support the latest Google Chrome, Mozilla Firefox, Microsoft Edge and Apple Safari browsers. While we make an effort to ensure that Partek Flow is robust, note that some browser plugins may affect the way the software is viewed on your browser.

• Web Browser Requirements

• Hardware Requirements (Single-node Linux)

• Hardware Requirements (Cluster or Cloud)

• Storage Recommendations

Hardware Requirements (Single-node Linux)

If you are installing Partek Flow on your own single-node server, we require the following for successful installation:

• Linux: Ubuntu® 18.04, Redhat® 8, CentOS® 8 or later versions of these distributions

• 64-bit 2GHz quad-core processor1

• 48GB of RAM2

• > 2TB of storage available for data

• > 100GB on the root partition

• A broadband internet connection

We support Docker-based installations. Please contact support@partek.com for more information.

1Note that some analyses have higher system requirements for example to run the STAR aligner on a reference genome of size ~3 GB (such as human, mouse or rat), 16 cores are required.

2Input sample file size can also impact memory usage, which is particularly the case for TopHat alignments.

Increasing hardware resources (cores, RAM, disk space, and speed) will allow for faster processing of more samples.

If you are licensed for the Single Cell Toolkit, please see Single Cell Toolkit System Requirements for amended hardware requirements.

Hardware Requirements (Cluster or Cloud)

Please contact Partek Technical Support if you would like to install Partek Flow on your own HPC or cloud account. We will assist in assessing your hardware needs and can make recommendations regarding provisioning sufficient resources to run the software.

Storage Recommendations

Proper storage planning is necessary to avoid future frustration and costly maintenance. Here are several DO's and DO NOT's:

DO:

• Plan for at least 3 to 5 times more storage than you think is necessary. Investing extra funds in storage and storage performance is always worth it.

• Keep all Flow data on a single partition that is expandable, such as RAID or LVM.

• Back up your data, especially the Partek Flow database.

DO NOT:

• Store data on 'removable' USB drives. Partek Flow will not be able to see these drives.

• Store data across multiple partitions or folder locations. This will increase the maintenance burden substantially.

• Use non-Linux file systems like NTFS.

Additional Assistance

If you need additional assistance, please visit our support page to submit a help ticket or find phone numbers for regional support.

This document describes how to set up and configure a single-node Partek Flow license.

• Installing on Linux

Additional Assistance

If you need additional assistance, please visit our support page to submit a help ticket or find phone numbers for regional support.

• Up to 100,000 cells per analysis

• More than 100,000 cells per analysis

Because of the large size of single cell RNA-Seq data sets and the computationally-intensive tools used in single cell analysis, we have amended our system requirements and recommendations for installations of Partek Flow with the Single Cell toolkit.

Up to 100,000 cells per analysis

Required

• Linux: Ubuntu® 18.04, Redhat® 8, CentOS® 8, or newer

• CPU: 64-bit 2 GHz quad-core processor

• Memory: 64 GB of RAM

• Local scratch space*: 1 TB with cached or native speeds of 2GB/s or higher

• Storage: > 2 TB available for data and > 100 GB on the root partition

Recommended

• Linux: Ubuntu® 18.04, Redhat® 8, CentOS® 8, or newer

• CPU: 64-bit 2 GHz quad-core processor

• Memory: 128 GB of RAM

• Local scratch space1: 2 TB with cached or native speeds of 2GB/s or higher

• Storage: > 2 TB available for data and > 100 GB on the root partition

More than 100,000 cells per analysis

Required

• Linux: Ubuntu® 18.04, Redhat® 8, CentOS® 8, or newer

• CPU: 64-bit 2 GHz quad-core processor

• Memory: 256 GB of RAM

• Local scratch space1: 2 TB with cached or native speeds of 2GB/s or higher

• Storage: > 4 TB available for data

Recommended

• Linux: Ubuntu® 18.04, Redhat® 8, CentOS® 8, or newer

• CPU: 64-bit 2 GHz quad-core processor

• Memory: 512 GB of RAM

• Local scratch space1: 10 TB with cached or native speeds of 2GB/s or higher

• Storage: 10 TB available for data

For fastest performance:

• Newer generation CPU cores with avx2 or avx-512 are recommended.

• Performance scales proportionality to the number of CPU cores available.

• Hyper thread cores (threads) scales performance for most operations other than principal component analysis.

*Contact Partek support for recommended setup of local scratch storage

Additional Assistance

If you need additional assistance, please visit our support page to submit a help ticket or find phone numbers for regional support.

Partek Flow is a genomics data analysis and visualization software product designed to run on compute clusters. The following instructions assume the most basic setup of Partek Flow and must only be attempted by system administrators who are familiar with Linux-based commands. These instructions are not intended to be comprehensive. Cluster environments are largely variable, thus there are no 'one size fits all' instructions. The installation procedure on a computer cluster is highly dependent on the type of computer cluster and the environment it is located. We can to support a large array of Linux distributions and configurations. In all cases, Partek Technical Support will be available to assist with cluster installation and maintenance to ensure compatibility with any cluster environment. Please consult with Partek Licensing Support (licensing@partek.com) for additional information.

Prior to installation, make sure you have the license key related to the host-ID of the compute cluster the software will be installed in. Contact licensing@partek.com for key generation.

• Installation on a Computer Cluster

• Integration with your queueing system

• Bringing up workers

• Shutting down workers

• Updating Partek Flow

Installation on a Computer Cluster

Make a standard linux user account that will run the Partek Flow server and all associated processes. It is assumed this account is synced between the cluster head node and all compute nodes. For this guide, we name the account flow

1. Log into the flow account and proceed to the cd to the flow home directory

cd home/flow

1. Download Partek Flow and the remote worker package

wget --content-disposition http://packages.partek.com/linux/flow

wget --content-disposition http://packages.partek.com/linux/flow-worker

1. Unzip these files into the flow home directory /home/flow. This yields two directories: partek_flow and P_artekFlowRemoteWorker_

2. Partek Flow can generate large amounts of data, so it needs to be configured to the bulk of this data in the largest shared data store available. For this guide we assume that the directory is located at /shared. Adjust this path accordingly.

3. It is required that the Partek Flow server (which is running on the head node) and remote workers (which is running on the compute nodes) see identical file system paths for any directory Partek Flow has read or write access to. Thus /shared and /home/flow must be mounted on the Flow server and all compute nodes. Create the directory /shared/FlowData and allow the flow linux account write access to it

4. It is assumed the head node is attached to at least two separate networks: (1) a public network that allows users to log in to the head node and (2) a private backend network that is used for communication between compute nodes and the head node. Clients connect to the Flow web server on port 8080 so adjust the firewall to allow inbound connections to 8080 over the public network of the head node. Partek Flow will connect to remote workers over your private network on port 2552 and 8443, so make sure those ports are open to the private network on the flow server and workers.

5. Partek Flow needs to be informed of what private network to use for communication between the server and workers. It is possible that there are several private networks available (gigabit, infiniband, etc.) so select one to use. We recommend using the fastest network available. For this guide, let's assume that private network is 10.1.0.0/16. Locate the headnode hostname that resolves to an address on the 10.1.0.0/16 network. This must resolve to the same address on all compute nodes.

6. For example:

host head-node.local yields 10.1.1.200

Open /home/flow/.bashrc and add this as the last line:

export CATALINA_OPTS="$CATALINA_OPTS -Djava.awt.headless=true
-DflowDispatcher.flow.command.hostname=head-node.local
-DflowDispatcher.akka.remote.netty.tcp.hostname=head-node.local"

Source .bashrc so the environment variable CATALINA_OPTS is accessible.

NOTE: If workers are unable to connect (below), then replace all hostnames with their respective IPs.

1. Start Partek Flow

~/partek_flow/start_flow.sh

1. You can monitor progress by tailing the log file partek_flow/logs/catalina.out. After a few minutes, the server should be up.

2. Make sure the correct ports are bound

netstat -tulpn

1. You should see 10.1.1.200:2552 and :::8080 as LISTENing. Inspect catalina.out for additional error messages.

2. Open a browser and go to http://localhost:8080 on the head node to configure the Partek Flow server.

3. Enter the license key provided (Figure 1)

Figure 1. Setting up the Partek Flow license during installation

1. If there appears to be an issue with the license or there is a message about 'no workers attached', then restart Partek Flow. It may take 30 sec for the process to shut down. Make sure the process is terminated before starting the server back up:

~/partek_flow/stop_flow.sh

Then run:

~/partek_flow/start_flow.sh

1. You will now be prompted to setup the Partek Flow admin user (Figure 2). Specify the username (admin), password and email address for the administrator account and click Next

Figure 2. Setting up the Partek Flow 'admin' account during installation

1. Select a directory folder to store the library files that will be downloaded or generated by Partek Flow (Figure 3). All Partek Flow users share library files and the size of the library folder can grow significantly. We recommend at least 100GB of free space should be allocated for library files. The free space in the selected library file directory is shown. Click Next to proceed. You can change this directory after installation by changing system preferences. For more information, see Library file management.

Figure 3. Selecting the library file directory

1. To set up the Partek Flow data paths, click on Settings located on the top-right of the Flow server webpage. On the left, click on Directory permissions then Permit access to a new directory. Add /shared/PartekFlow and allow all users access.

2. Next click on System preferences on the left menu and change data download directory and default project output directory to /shared/PartekFlow/downloads and /shared/PartekFlow/project_output respectively

Note: If you do not see the /sharedfolder listed, click on the Refresh folder list link that is toward the bottom of the download directory dialog

1. Since you do not want to run any work on the head node, go to Settings>System preferences>Task queue and job processing and uncheck Start internal worker at Partek Flow server startup.

2. Restart the Flow server:

~/partek_flow/stop_flow.sh

After 30 seconds, run:

~/partek_flow/start_flow.sh

This is needed to disable the internal worker.

1. Test that remote workers can connect to the Flow server

2. Log in as the flow user to one of your compute nodes. Assume the hostname is compute-0. Since your home directory is exported to all compute nodes, you should be able to go to /home/flow/PartekFlowRemoteWorker/

3. To start the remote worker:

./partekFlowRemoteWorker.sh head-node.local compute-0

1. These two addresses should both be in the 10.1.0.0/16 address space. The remote worker will output to stdout when you run it. Scan for any errors. You should see the message woot! I'm online.

2. A successfully connected worker will show up on the Resource management page on the Partek Flow server. This can be reached from the main homepage or by clicking Resource management from the Settings page. Once you have confirmed the worker can connect, kill the remote worker (CTRL-C) from the terminal in which you started it.

3. Once everything is working, return to library file management and add the genomes/indices required by your research team. If Partek hosts these genomes/indices, these will automatically be downloaded by Partek Flow

Integration with your queueing system

1. In effect, all you are doing is submitting the following command as a batch job to bring up remote workers:

/home/flow/PartekFlowRemoteWorker/partekFlowRemoteWorker.sh head-node.local compute-0

1. The second parameter for this script can be obtained automatically via:

$(hostname -s)

Bringing up workers

Bring up workers by running the command below. You only need to run one worker per node:

/home/flow/PartekFlowRemoteWorker/partekFlowRemoteWorker.sh head-node.local compute-0

Shutting down workers

Go to the Resource management page and click on the Stop button (red square) next to the worker you wish to shut down. The worker will shut down gracefully, as in it will wait for currently running work on that node to finish, then it will shut down.

Updating Partek Flow

For the cluster update, you will get a link of .zip file for Partek Flow and remote Flow worker respectively from Partek support, all of the following actions should be performed as the Linux user that runs Flow. Do NOT run Flow as root.

1. Go to the Flow installation directory. This is usually the home directory of the Linux user that runs Flow and it should contain a directory named "partek_flow". The location of the Flow install can also be obtained by running ps aux | grep flow and examining the path of the running Flow executable.

2. Shut down Flow:

./partek_flow/stop_flow.sh

1. Download the new version of Flow and the Flow worker:

wget --content-disposition http://packages.partek.com/linux/flow-release

wget --content-disposition http://packages.partek.com/linux/flow-worker-release

1. Make sure Flow has exited:

ps aux | grep flow

The flow process should no longer be listed.

1. Unpack the new version of Flow install and backup the old install:

mv partek_flow partek_flow_prev

mv PartekFlowRemoteWorker PartekFlowRemoteWorker_prev

1. Backup the Flow database folder. This should be located in the home directory of the user that runs Flow.

tar -czvf partek-db-bkp-date.tgz ~/.partekflow

1. Start the updated version of Flow:

./partek_flow/start_flow.sh

tail -f partek_flow/logs/catalina.out

(make sure there is nothing of concern in this file when starting up Flow. You can stop the file tailing by typing: CTRL-C)

You may also want to examine the the main Flow log for errors:

~/.partekflow/logs/flow.log

Additional Assistance

If you need additional assistance, please visit our support page to submit a help ticket or find phone numbers for regional support.

• Creating a New Elastic Compute Cloud Instance for Partek Flow Software

• Enabling External Access to the Partek Flow Elastic Compute Cloud Instance)

• Attaching the Amazon Elastic Block Store Volume for Partek Flow Data Storage)

• Installing Partek Flow on a New Elastic Compute Cloud Instance

• Partek Amazon Web Services Support

• General Recommendations

• Amazon Web Services Instance Type Resources and Costs

• Elastic Block Store Volumes

Creating a New Elastic Compute Cloud Instance for Partek Flow Software

Note: This guide assumes all items necessary for the Amazon elastic Comput Clout (EC2) instance does not exist, such as Amazon Virtual Private Cloud (VPC), subnets, and security groups, thus their creation is covered as well.

Log in to the Amazon Web Services (AWS) management console at https://console.aws.amazon.com

Click on EC2

Switch to the region intended to deploy Partek Flow software. This tutorial uses US East (N. Virginia) as an example.

On the left menu, click on Instances, then click the Launch Instance button. The Choose an Amazon Machine Image (AMI) page will appear.

Click the Select button next to Ubuntu Server 16.04 LTS (HVM), SSD Volume Type - ami-f4cc1de2. NOTE: Please use the latest Ubuntu AMI. It is likely that the AMI listed here will be out of date.

Choose an Instance Type, the selection depends on your budget and the size of the Partek Flow deployment. We recommend m4.large for testing or cluster front-end operation, m4.xlarge for standard deployments, and m4.2xlarge for alignment-heavy workloads with a large user-base. See the section AWS instance type resources and costs for assistance with choosing the right instance. In most cases, the instance type and associated resources can be changed after deployment, so one is not locked into the choices made for this step.

NOTE: New instance types will become available. Please use the latest mX instance type provided as it will likely perform better and be more cost effective than older instance types.

On the Configure Instance Details page, make the following selections:

• Set the number of instances to 1. An autoscaling group is not necessary for single-node deployments

• Purchasing Option: Leave Request Spot Instances unchecked. This is relevant for cost-minimization of Partek Flow cluster deployments.

• Network: If you do not have a virtual private cloud (VPC) already created for Partek Flow, click Create New VPC. This will open a new browser tab for VPC management.

  • Use the following settings for the VPC:

    • Name Tag: Flow-VPC

    • IPv4 CIDR block: 10.0.0.0/16

    • Select No IPv6 CIDR Block

    • Tenancy: Default

  • Click Yes, Create. You may be asked to select a DHCP Option set. If so, then make sure the dynamic host configuration protocol (DHCP) option set has the following properties:

    • Options: domain-name = ec2.internal;domain-name-servers = AmazonProvidedDNS;

    • DNS Resolution: leave the defaults set to yes

    • DNS Hostname: change this to yes as internal DNS resolution may be necessary depending on the Partek Flow deployment

  • Once created, the new Flow-VPC will appear in the list of available VPCs. The VPC needs additional configuration for external access. To continue, right click on Flow-VPC and select Edit DNS Resolution, select Yes, and then Save. Next, right click the Flow-VPC and select Edit DNS Hostnames, select Yes, then Save.

  • Make sure the DHCP option set is set to the one created above. If it is not, right-click on the row containing Flow-VPC and select Edit DHCP Option Sets.

  • Close the VPC Management tab and go back to the EC2 Management Console.

• Click the refresh arrow next to Create New VPC and select Flow-VPC.

• Click Create New Subnet and a new browser tab will open with a list of existing subnets. Click Create Subnet and set the following options:

  • Name Tag: Flow-Subnet

  • VPC: Flow-VPC

  • VPC CIDRs: This should be automatically populated with the information from Flow-VPC

  • Availability Zone: It is OK to let Amazon choose for you if you do not have a preference

  • IPv4 CIDR block: 10.0.1.0/24

• Stay on the VPC Dashboard Tab and on the left navigation menu, click Internet Gateways, then click Create Internet Gateway and use the following options:

  • Name Tag: Flow-IGW

  • Click Yes, Create

• The new gateway will be displayed as Detached. Right click on the Flow-IGW gateway and select Attach to VPC, then select Flow-VPC and click Yes, Attach.

• Click on Route Tables on the left navigation menu.

• If it exists, select the route table already associated with Flow-VPC. If not, make a new route table and associate it with Flow-VPC. Click on the new route table, then click the Routes tab toward the bottom of the page. The route Destination = 10.0.0.0/16 Target = local should already be present. Click Edit, then Click Add another route and set the following parameters:

  • Destination: 0.0.0.0/0

  • Target set to Flow-IGW (the internet gateway that was just created)

• Click Save

• Close the VPC Dashboard browser tab and go back to the EC2 Management Console tab. Note that you should still be on Step 3: Configure Instance Details.

Click the refresh arrow next to Create New Subnet and select Flow-Subnet.

Auto-assign Public IP: Use subnet setting (Disable)

Placement Group: No placement group

IAM role: None.

Note: For multi-node Partek Flow deployments or instances where you would like Partek to manage AWS resources on your behalf, please see Partek AWS support and set up an IAM role for your Partek Flow EC2 instance. In most cases a specialized IAM role is unnecessary and we only need instance ssh keys.

Shutdown Behaviour: Stop

Enable Termination Protection: select Protect against accidental termination

Monitoring: leave Enable CloudWatch Detailed Monitoring disabled

EBS-optimized Instance: Make sure Launch as EBS-optimized Instance is enabled. Given the recommended choice of an m4 instance type, EBS optimization should be enabled at no extra cost.

Tenancy: Shared - Run a shared hardware instance

Network Interfaces: leave as-is

Advanced Details: leave as-is

Click Next: Add Storage. You should be on Step 4: Add Storage

For the existing root volume, set the following options:

• Size: 8 GB

• Volume Type: Magnetic

• Select Delete on Termination

  • Note: All Partek Flow data is stored on a non-root EBS volume. Since only the OS is on the root volume and not frequently re-booted, a fast root volume is probably not necessary or worth the cost. For more information about EBS volumes and their performance, see the section EBS volumes.

Click Add New Volume and set the following options:

• Volume Type: EBS

• Device: /dev/sdb (take the default)

• Do not define a snapshot

• Size (GiB): 500

  • Note: This is the minimum for ST1 volumes, see: EBS volumes

• Volume Type: Throughput optimized HDD (ST1)

• Do not delete on terminate or encrypt

Click Next: Add Tags

• You do not need to define any tags for this new EC2 instance, but you can if you would like.

Click Next: Configure Security Group

• For Assign a Security Group select Create a New Security Group

• Security Group Name: Flow-SG

• Description: Security group for Partek Flow server

• Add the following rules:

  • SSH set Source to My IP (or the address range of your company or institution)

  • Click Add Rule:

  • Set Type to Custom TCP Rule

  • Set Port Range to 8080

  • Set Source to anywhere (0.0.0.0/0, ::/0)

    • Note: It is recommended to restrict Source to just those that need access to Partek Flow.

Click Review and Launch

• The AWS console will suggest this server not be booted from a magnetic volume. Since there is not a lot of IO on the root partition and reboots are will be rare, choosing Continue with Magnetic will reduce costs. Choosing an SSD volume will not provide substantial benefit but it OK if one wishes to use an SSD volume. See the EBS Volumes section for more information.

Click Launch

Create a new keypair:

• Name the keypair Flow-Key

• Download this keypair, the run chmod 600 Flow-Key.pem (the downloaded key) so it can be used.

• Backup this key as one may lose access to the Partek Flow instance without it.

The new instance will now boot. Use the left navigation bar and click on Instances. Click the pencil icon and assign the instance the name Partek Flow Server

Enabling External Access to the Partek Flow Elastic Compute Cloud Instance

The server should be assigned a fixed IP address. To do this, click on Elastic IPs on the left navigation menu from the EC2 Management Console.

• Click Allocate New Address

• Assign Scope to VPC

• Click Allocate

On the table containing the newly allocated elastic IP, right click and select Associate Address

• For Instance, select the instance name Flow Test Server

• For Private IP, select the one private IP available for the Partek Flow EC2 instance, then click Associate

Note: For the remaining steps, we refer to the elastic ip as elastic.ip

SSH to the new Flow-Server instance:

$ chmod 600 Flow-Key.pem

$ ssh -i Flow-Testing.pem ubuntu@elastic.ip

Attaching the Amazon Elastic Block Store Volume for Partek Flow Data Storage

Attach, format, and move the ubuntu home directory onto the large ST1 elastic block store (EBS) volume. All Partek Flow data will live in this volume. Consult the AWS EC2 documentation for further information about attaching EBS volumes to your instance.

$ sudo su

$ mkfs -t ext4 /dev/xvdb

Note: Under Volumes in the EC2 management console, inspect Attachment Information. It will likely list the large ST1 EBS volume as attached to /dev/sdb. Replace "s" with "xv" to find the device name to use for this mkfs command.

Make a note of the newly created UUID for this volume

Copy the ubuntu home directory onto the EBS volume using a temporary mount point:

$ mount -t ext4 /dev/xvdb /mnt/

$ rsync -avr /home/ /mnt/

$ umount /mnt/

Make the EBS volume mount at system boot:

Add the following to /etc/fstab: UUID=the-UUID-from-the-mkfs-command-above /home ext4 defaults,nofail 0 2

$ mount -a

Disconnect the ssh session, then log in again to make sure all is well

Installing Partek Flow on a New Elastic Compute Cloud Instance

Note: For additional information about Partek Flow installations, see our generic Installation Guide

Before beginning, send the media access control (MAC) address of the EC2 instance to MAC address of the EC2 instance to licensing@partek.com. The output of ifconfig will suffice. Given this information, Partek employees will create a license for your AWS server. MAC addresses will remain the same after stopping and starting the Partek Flow EC2 instance. If the MAC address does change, let our licensing department know and we can add your license to our floating license server or suggest other workarounds.

Install required packages for Partek Flow:

$ sudo apt-get update

$ sudo apt-get install software-properties-common

$ sudo add-apt-repository -y ppa:openjdk-r/ppa

$ sudo apt-get install openjdk-8-jdk python python-pip python-dev zlib1g-dev python-matplotlib r-base python-htseq libxml2-dev perl make gcc g++ zlib1g libbz2-1.0 libstdc++6 libgcc1 libncurses5 libsqlite3-0 libfreetype6 libpng12-0 zip unzip libgomp1 libxrender1 libxtst6 libxi6 debconf 

$ sudo pip install --upgrade pip && pip install --upgrade --upgrade-strategy eager --force-reinstall virtualenv numpy pysam cnvkit

Install Partek Flow:

Note: Make sure you are running as the ubuntu user.

$ cd (we will install Partek Flow to ubuntu's home directory)

$ wget --content-disposition packages.partek.com/linux/flow-release

$ unzip PartekFlow*.zip

$ ./partek_flow/start_flow.sh

Partek Flow has finished loading when you see INFO: Server startup in xxxxxxx ms in the partek_flow/logs/catalina.out log file. This takes ~30 seconds.

Alternative: Install Flow with Docker. Our base packages are located here: https://hub.docker.com/r/partekinc/flow/tags

Open Partek Flow with a web browser: http://elastic.ip:8080/

Enter license key

Set up the Partek Flow admin account

Leave the library file directory at its default location and check that the free space listed for this directory is consistent with what was allocated for the ST1 EBS volume.

Done! Partek Flow is ready to use.

Partek Amazon Web Services Support

After the EC2 instance is provisioned, we are happy to assist with setting up Partek Flow or address other issues you encounter with the usage of Partek Flow. The quickest way to receive help is to allow us remote access to your server by sending us Flow-Key.pem and amending the SSH rule for Flow-SG to include access from IP 97.84.41.194 (Partek HQ). We recommend sending us the Flow-Key.pem via secure means. The easiest way to do this is with the following command:

$ curl -F "file=@FlowKey.pem" https://installfeedback.partek.com/fupload

We also provide live assistance via GoTo meeting or TeamViewer if you are uncomfortable with us accessing your EC2 instance directly. Before contacting us, please run $ ./partek_flow/flowstatus.sh to send us logs and other information that will assist with your support request.

General Recommendations

With newer EC2 instance types, it is possible to change the instance type of an already deployed Partek Flow EC2 server. We recommend doing several rounds of benchmarks with production-sized workloads and evaluate if the resources allocated to your Partek Flow server are sufficient. You may find that reducing resources allocated to the Partek Flow server may come with significant cost savings, but can cause UI responsiveness and job run-times to reach unacceptable levels. Once you have found an instance type that works, you may wish to use reserved instance pricing which is significantly cheaper than on-demand instance pricing. Reserved instances come with 1 or 3-year usage terms. Please see the EC2 Reserved Instance Marketplace to sell or purchase existing reserved instances at reduced rates.

The network performance of the EC2 instance type becomes an important factor if the primary usage of Partek Flow is for alignment. For this use case, one will have to move copious amounts of data back (input fastq files) and forth (output bam files) between the Partek Flow server and the end users, thus it is important to have as what AWS refers to as high network performance which for most cases is around 1 Gb/s. If the focus is primarily on downstream analysis and visualization (e.g. the primary input files are ADAT) then network performance is less of a concern.

We recommend HVM virtualization as we have not seen any performance impact from using them and non-HVM instance types can come with significant deployment barriers.

Make sure your instance is EBS optimized by default and you are not charged a surcharge for EBS optimization.

T-class servers, although cheap, may slow responsiveness for the Partek Flow server and generally do not provide sufficient resources.

We do not recommend placing any data on instance store volumes since all data is lost on those volumes after an instance stops. This is too risky as there are cases where user tasks can take up unexpected amounts of memory forcing a server stop/reboot.

Amazon Web Services Instance Type Resources and Costs

The values below were updated April 2017. The latest pricing and EC2 resource offerings can be found at http://www.ec2instances.info

Single server recommendation: m4.xlarge or m4.2xlarge

Network performance values for US-EAST-1 correspond to: Low ~ 50Mb/s, Medium ~ 300Mb/s, High ~ 1Gb/s.

Elastic Block Store Volumes

Choice of a volume type and size:

This is dependent on the type of workload. For must users, the Partek Flow server tasks are alignment-heavy so we recommend a throughput optimized HDD (ST1) EBS volume since most aligner operations are sequential in nature. For workloads that focus primarily on downstream analysis, a general purpose SSD volume will suffice but the costs are greater. For those who focus on alignment or host several users, the storage requirements can be high. ST1 EBS volumes have the following characteristics:

Max throughput 500 MiB/s

$0.045 per GB-month of provisioned storage ($22.5 per month for a 500 GB of storage).

Note that EBS volumes can be grown or performance characteristics changed. To minimize costs, start with a smaller EBS volume allocation of 0.5 - 2 TB as most mature Partek Flow installations generate roughly this amount of data. When necessary, the EBS volume and the underlying file system can be grown on-line (making ext4 a good choice). Shrinking is also possible but may require the Partek Flow server to be offline.

Additional Assistance

If you need additional assistance, please visit our support page to submit a help ticket or find phone numbers for regional support.

Partek Flow provides the infrastructure to isolate data from different users within the same server. This guide will provide general instructions on how to create this environment within Partek Flow. This can be modified to accommodate existing file systems already accessible to the server.

Go to Settings > Directory permissions and restrict parent folder access (typically /home/flow) to Administrator accounts only

Click the Permit access to a new directory button and navigate to the folder with your library files (typically /home/flow/FlowData/library_files). Select the All users (automatically updated) checkbox to permit all users (including those that will be added in the future) to see the library files associated with the Partek Flow server

Then go to System preferences > Filesystem and storage and set the Default project output directory to "Sample file directory"

Create your first user and select the Private directory checkbox. Specify where the private directory for that user is located

If needed, you can create a user directory by clicking Browse > Create new folder

This automatically sets browsing permissions for that private directory to that user

When a user creates a project. The default project output directory is now within their own restricted folder

More importantly, other users cannot see them

Add additional users as needed

Additional Assistance

Flow ships with tasks that do not have all of their dependencies included. On startup Flow will attempt to install the dependencies, but not every system is equipped to install them.

In the case of any difficulties, it is highly recommended to instead use a docker deployment (cluster installations may require singularity instead, which is somewhat still a work-in-progress)Z

CNVkit

Requires Python 2.7 or later.

On startup Flow will attempt to install additional python packages using the command

Requires R 3.2.3 or later.

On startup Flow will attempt to install additional R packages.

There are cascading dependencies, but you can view the core libraries in partek_flow/bin/cnvkit-0.8.5/install.R

If these packages can't be built locally, it may be possible for the user to download them from us (see below).

DESeq2

Requires R 3.0 or later.

On startup Flow will attempt to install additional R packages.

There are cascading dependencies, but you can view the core libraries in partek_flow/bin/deseq_two-3.5/install.R

If these packages can't be built locally, it may be possible for the user to download them from us (see below).

RcppArmadillo may also have dependencies on multi-threading shared objects that may not be on the LD_LIBRARY_PATH

The recommendation is to copy those .so files to a folder and make sure it is available from the LD_LIBRARY_PATH when the server/worker starts.

Additional dynamic libraries (such as libxml2.so) may be missing and we can provide a copy appropriate for the target OS.

HTSeq

Requires Python 2.7 or 3.4 or above

On startup Flow attempts to install using pip

MACS3

Requires python 3.0 or above

Python

If there are any conflicts with preinstalled python packages, Flow should be configured to run with its own virtual environment:

or

R

R can usually be installed from the package manager. If the user installs Flow via apt or yum it should already be installed.

Currently, we offer a set of R packages compatible with some versions of R

Extract this file in the home directory. (Make .R a symlink if the home directory doesn't have enough free space)

These packages include the dependencies for both CNVkit and DESeq2

When running R diagnostic commands outside flow, it can simplify things if the environment includes a reference to the ~/.R folder:

or load

in ~/.Rprofile

list loaded packages:

get the version:

Variant Effect Predictor

This is a compiled Perl script (so it has no direct dependency on Perl itself) we have had one report (istem.fr) of it failing to run.

DECoN

DECoN requires R version 3.1.2

It must be installed under /opt/R-3.1.2 or set the DECON_R environment variable to its folder

Download DECoN

and install it under /opt/DECoN or set the DECON_PATH environment variable to its folder

You may need to add

to Linux/packrat/packrat.opts

Additional Assistance

Below are the yaml documents which describe the bare minimum infrastructure needed for a functional Flow server. It is best to start with a single-node proof of concept deployment. Once that works, the deployment can be extended to multi-node with elastic worker allocation. Each section is explained below.

The Flow headnode pod

Pod metadata

On a kubernetes cluster, all Flow deployments are placed in their own namespace, for example namespace: partek-flow. The label app.kubernetes.io/name: flowheadnode allows binding of a service or used to target other kubernetes infrastructure to this headnode pod. The label deployment: dev allows running multiple Flow instances in this namespace (dev, tst, uat, prd, etc) if needed and allows workers to connect to the correct headnode. For stronger isolation, running each Flow instance in its own namespace is optimal.

Data storage

The Flow docker image requires 1) a writable volume mounted to /home/flow 2) This volume needs to be readable and writable by UID:GID 1000:1000 3) For a multi-node setup, this volume needs to be cross mounted to all worker pods. In this case, the persistent volume would be backed by some network storage device such as EFS, NFS, or a mounted FileGateway.

This section achieves goal 2)

The flowconfig volume is used to override behavior for custom Flow builds and custom integrations. It is generally not needed for vanilla deployments.

The Flow docker image

Partek Flow is shipped as a single docker image containing all necessary dependencies. The same image is used for worker nodes. Most deployment-related configuration is set as environment variables. Auxiliary images are available for additional supporting infrastructure, such as flexlm and worker allocator images.

Official Partek Flow images can be found on our release notes page: Release Notes The image tags assume the format: registry.partek.com/rtw:YY.MMMM.build New rtw images are generally released several times a month. The image in the example above references a private ECR. It is highly recommended that the target image from registry.partek.com be loaded into your ECR. Image pulls will be much faster from AWS - this reduces the time to dynamically allocate workers. It also removes a single point of failure - if registry.partek.com were down it would impact your ability to launch new workers on demand.

Flow headnode resource request

Partek Flow uses the head node to handle all interactive data visualization. Additional CPU resources are needed for this, the more the better and 8 is a good place to start. As for memory, we recommend 8 to 16 GiB. Resource limits are not included here, but are set to large values globally:

Relevant Flow headnode environment variables

PARTEKLM_LICENSE_FILE

Partek Flow uses FlexLM for licensing. Currently we do not offer or have implemented any alternative. Values for this environment variable can be:

@flexlmserveraddress

An external flexlm server. We provide a Partek specific container image and detail a kubernetes deployment for this below. This license server can also live outside the kubernetes cluster - the only requirement being that it is network accessible. /home/flow/.partekflow/license/Partek.lic - Use this path exactly. This path is internal to the headnode container and is persisted on a mounted PVC.

Unfortunately, FlexLM is MAC address based and does not quite fit in with modern containerized deployments. There is no straightforward or native way for kubernetes to set the MAC address upon pod/container creation, so using a license file on the flowheadnode pod (/home/flow/.partekflow/license/Partek.lic ) could be problematic (but not impossible). In further examples below, we provide a custom FlexLM container that can be instantiated as a pod/service. This works by creating a new network interface with the requested MAC address inside the FlexLM pod.

PARTEK_COMMON_NO_TOTAL_LIMITS

Please leave this set at "1". Partek Flow need not enforce any limits as that is the responsibility of kubernetes. Setting this to anything else may result in Partek executables hanging.

CATALINA_OPTS

This is a hodgepodge of Java/Tomcat options. Parts of interest:

It is possible for the Flow headnode to execute jobs locally in addition to dispatching them to remote workers. These two options set resource limits on the Flow internal worker to prevent resource contention with the Flow server. If remote workers are not used and this remains a single-node deployment, meaning ALL jobs will execute on the internal worker, then it is best to remove the CPU limit (-DFLOW_WORKER_CORES) and only set -DFLOW_WORKER_MEMORY_MB equal to the kubernetes memory resource request.

If Flow connects to a corporate LDAP server for authentication, it will need to trust the LDAP certificates.

JVM heap size. If the internal worker is not used, set this to be a little less than the kubernetes memory resource request. If the internal worker is an use, and the intent is to stay with a single-node deployment, then set this to be ~ 25% of the kubernetes memory resource request, but no less than ~ 4 GiB.

The Flow headnode service definition

The flowheadnode service is needed 1) so that workers have a DNS name (flowheadnode) to connect to when they start and 2) so that we can attach an ingress route to make the Flow web interface accessible to end users. The app.kubernetes.io/name: flowheadnode selector is what binds this to the flowheadnode pod.

• 80:8080 - Users interact with Flow entirely over a web browser

• 2552:2552 - Workers communicate with the Flow server over port 2552

• 8443:8443 - Partek executed binaries connect back to the Flow server over port 8443 to do license checks

Ingress to flowheadnode

This provides external users HTTPS access to Flow at host: flow.dev-devsvc.domain.com Your details will vary. This is where we bind to the flowheadnode service.

The flexlm service pod

The yaml documents above will bring up a complete Partek-specific license server.

Note that the service name is flexlmserver. The flowheadnode pod connects to this license server via the PARTEKLM_LICENSE_FILE="@flexlmserver" environment variable.

You should deploy this flexlmserver first, since the flowheadnode will need it available in order to start in a licensed state.

Partek will send a Partek.lic file licensed to some random MAC address. When this license is (manually) written to /usr/local/flexlm/licenses, the pod will continue execution by creating a new network interface using the MAC address in Partek.lic, then it will start the licensing service. This is why the NET_ADMIN capability is added to this pod.

The license from Partek must contain VENDOR parteklm PORT=27001 so the vendor port remains at 27001 in order to match the service definition above. Without this, this port is randomly set by FlexLM.

This image is currently available from public.ecr.aws/partek-flow/kube-flexlm-server but this may change in the future.

Before performing updates, we recommend Backing Up the Database.

Updates are applied using the Linux package manager.

Make sure Partek Flow is stopped before updating it.

To update Partek Flow, open a terminal window and enter the following command.

For Debian/Unbuntu, enter:

For Redhat/Fedora/CentOS, enter:

For the YUM package manager, if updating Partek Flow fails with a message claiming "package not signed," enter:

Note that our packages are signed and the message above is erroneous.

For tomcat build update, download the latest version from below:

Additional Assistance

If you need additional assistance, please visit to submit a help ticket or find phone numbers for regional support.

Java Keystore

JKS or Java KeyStore is used in Flow for some very specific scenarios where encryption is involved and there is a need for asymmetric encryption.

Partek Flow is shipped with a Java Keystore on its own, the file is found at .../partek_flow/distrib/flowkeystore where you may want to add your public and private certificates.

Adding a certificate to the KeyStore

If you already have a certificate please skip to the next step.

Create a certificate

Please place the key in a secure folder. (it is advisable to place in Flow's home directory. eg. /home/flow/keys

These commands above are meant to be used in a terminal. There are other ways to help you make a certificate but they will not going to be mentioned here.

If you wish to understand the flags used above please refer to the OpenSSL documentation.

Import a certificate into flowkeystore

For this step you will have to find where the cacerts file is located, it is under the Java installation, if you do not know how to do it contact us and we can help.

In the example the cacerts file is located at /usr/lib/jvm/java-11-openjdk-amd64/lib/security/cacerts

Tell the JVM where to find the key

We need to tell Partek Flow where the key is located, to do this we will edit a file which contains some of the Flow settings.

The file is usually located at /etc/partekflow.conf if you do not have this file we would advise to use the bashrc file from the system user that runs Partek Flow.

At the end of that file please add:

Additional Assistance

Docker

Docker can be used along Partek Flow to deploy an easy to maintain environment which will not have dependency issues and will be easy to relocate among different servers if needed.

One can follow the Docker documentation to install and get started.

“Docker is a platform for developers and sysadmins to build, run, and share applications with containers. The use of containers to deploy applications is called containerization. “

Useful commands

This command will output the details of the currently running containers including port forwarding, container name/id, and uptime.

This command will allow us to enter the running container’s environment to troubleshoot any issues we might have with the container. (the containers are not meant to be changed the correct way to deal with any issues is creating a new one after the troubleshot)

Docker-compose

“Compose is a tool for defining and running multi-container Docker applications. With Compose, you use a YAML file to configure your application’s services. Then, with a single command, you create and start all the services from your configuration.“

Below it is an example of a docker-compose.yml file which can be used to bring a Partek Flow server with an extra worker.

These are some of the important tags shown above:

• restart: whether you want the container to be restarted automatically upon failure or system restart.

• image: the image we distributed and the desired version, even though we always recommend the users to run the latest version of the software if you need any specific versions of Partek Flow please visit here.

• environment: here we set up any environment variables to be run along the container.

• port: the default port to Partek Flow is 8080 and if you wish to change what port it will be accessible please change the first part (left to the colon) of 8080:8080. So if you wish to access the server on port 8888 then the correct format will be 8888:8080

• mac_address: this needs to match your license file

• volumes: in this section we specify the folder on the server to be shared with the container, this way we can better persist and access the files we create and use in the container.

Additional Assistance

Linux

Open a terminal window and enter the following command.

Debian/Ubuntu:

RedHat/Fedora/CentOS:

The uninstall removes binaries only (/opt/partek_flow). The logs, database (partek_db) and files in the home/flow/.partekflow folder will remain unaffected.

MacOS

1. Stop and quit Partek Flow using the Partek Flow app in the menu.

2. Using Finder, delete Flow application from the Applications menu.

Missing image Figure 1. Control of Partek Flow through the menu bar

This process does not delete data or the library files. Users who wish to delete those can delete them using Finder or terminal. The default location of project output files and library files is the /FlowData directory under the user's home folder. However, the actual location may vary depending on your System or Project specific settings.

Additional Assistance

If you need additional assistance, please visit to submit a help ticket or find phone numbers for regional support.

Here you will find videos of past live training webinars.

• Bulk RNA-Seq Analysis Training

• Basic scRNA-Seq Analysis & Visualization Training

• Advanced scRNA-Seq Data Analysis Training

• Bulk RNA-Seq and ATAC-Seq Integration Training

• Spatial Transcriptomics Data Analysis Training

• scRNA and scATAC Data Integration Training

Additional Assistance

If you need additional assistance, please visit our support page to submit a help ticket or find phone numbers for regional support.

Webinar Breakdown

Topics have been broken out into bite-size videos for your convenience.

Webinar Breakdown

Topics have been broken out into bite-size videos for your convenience.

Webinar Breakdown

Topics have been broken out into bite-size videos for your convenience.

Using a web browser, log in to Partek Flow. From the Home page click the New Project button; enter a project name (Figure 1) and then click Create project.

The Project name is the basis of the default name of the output directory for this project. Project names are unique, thus a new project cannot have the same name as an existing project within the same Partek Flow server.

Once a new project has been created, the user is automatically directed to the Analysis tab of the Project View (Figure 2).

Additional Assistance

Partek Flow software manages separate experiments as projects. A complete project consists of input data, tasks used to analyze the data, the resulting output files, and a list of users involved in the analysis.

This chapter provides instructions in creating and analyzing a project and covers:

Additional Assistance

Partek Flow tutorials provide step-by-step instructions using a supplied data set to teach you how to use the software tools. Upon completion of each tutorial, you will be able to apply your knowledge in your own studies.

Additional Assistance

If you need additional assistance, please visit to submit a help ticket or find phone numbers for regional support.

Webinar Breakdown

Topics have been broken out into bite-size videos for your convenience.

Webinar Breakdown

Topics have been broken out into bite-size videos for your convenience.

Webinar Breakdown

Topics have been broken out into bite-size videos for your convenience.

The Partek Flow Metadata Tab has an option to import data, and is where sample/cell attributes are managed. This is also where users can modify the location of the project output folder.

Import data

The Metadata tab can be used to import data. To add samples to the project, click Add data under Import, different import options are displayed using the cascading menu (Figure 1).

Automatically create samples from files

This method adds samples by creating them simultaneously as the data gets imported into a project. The sample names are assigned automatically based on filenames.

Before proceeding, it is ideal that you have already transfer the data you wish to analyze in a folder (with appropriate permissions) within the Partek Flow server. Please seek assistance from your system administrator in uploading your data directly.

Select the Automatically create samples from files button. The next screen will feature a file browser that will show any folders you have access to in the Partek Flow server (Figure 2). Select a folder by clicking the folder name. Files in the selected folder that have file formats that can be imported by Partek Flow will be displayed and tick-marked on the right panel. You can exclude some files from the folder by unselecting the check mark on the left side of the filename. When you have made your selections, click the Create sample button.

Alternatively, files can also be uploaded and imported into the project from the user's local computer -only use this option if your file size is less than 500MB. Select the My computer radio button (Figure 3) and the options of selecting the local file and the upload (destination) directory will appear. Only one file at a time can be imported to a project using this method.

Multiple data files can be compressed a single .zip file before uploading. Partek Flow will automatically unzip the files and put them in the upload directory.

Please be aware that the use of the method illustrated in Figure 3 highly depends on the speed and latency of the Internet connection between the user's computer and the server where Partek Flow is installed. Given the large size of most genomics data sets, is not recommended in most cases.

After successful creation of samples from files, the Data tab now contains a Sample management table (Figure 4). The Sample name column in the table is automatically generated based on the filenames and the table is sorted in alphabetical order.

Clicking the on the** Show data files** link on the lower right side of the sample management table will expand the table and reveal the filenames of the files associated with each sample. Conversely, clicking on Hide data files will hide the file information.

The columns in the expanded view show the files associated with each sample. Files are organized by file type. Any filename extensions that indicate compression (such as .gz) are not shown.

Once a sample is created in a project, the files associated with it can be modified. In the expanded view, mouse over the +/- column of a sample. The highlighted icons will correspond to the options for the sample on that row.

Create a new blank sample

Samples can be added one at a time by selecting the Create a new blank sample option (Figure 5). In the following dialog box, type a sample name and click Create. This process creates a sample entry in the sample management table but there is no associated file with it, hence it is a "blank sample."

Expanding the Sample management table by clicking Show data files on the lower left corner of the table will reveal the option to associate files to the blank sample.

Importing count matrix data

Alternatively, if you have a matrix of data, such as raw read count data in text format, select Import count matrix. The requirements of this text file are listed below:

• The file contains numeric values in a tab-delimited format, samples can be on rows while features (e.g.gene names) are in columns, or vice versa

• The file contains unique sample IDs and feature IDs

• If the data contains sample attribute information, all these attributes have to be ether

  • The leftmost columns when samples are on rows (Figure 6)

  • The first few rows when samples are on columns (Figure 7)

Like all other input files, you can upload the file from the Partek Flow server, My Computer or via a URL. Uploading the file brings up a file preview window (Figure 8). The preview of the first few rows and columns of the text file should help you determine on which rows/columns the relevant counts are located (the preview will display up to 100 rows and 100 columns). Inspect the text preview and indicate the orientation of the text file under File format>Input format.

If the read counts are based on a compatible annotation file in Partek Flow, you can specify that annotation file under Gene/feature annotation. Select the appropriate genome build and annotation model for your count data. Select the Contain sample attributes checkbox if your data includes additional sample information.

The example above is showing an example text file with samples listed on rows. The gene ID is compatible with the hg19 RefSeq Transcripts - 2016-08-01 annotation model. Under the Column information and Row information sections, indicate the location of the Sample ID, which in this case is on Column 1. Indicate the sample attribute location by marking where it starts, which in the example is at Column 2. Mark the Feature ID, which in this case are gene IDs and starts at Column 4.

If the data has been log transformed, specify the base under Counts format.

Project output directory

The project output directory is the folder within the Partek Flow server where all output files produced during analysis will be stored.

The default directory is configured by the Partek Flow Administrator under the Settings menu (under System Preferences > Default project output directory).

If the user does not override the default, the task output will go to a subdirectory with the name of the Project.

Sample Annotation

After samples have been added in the project, additional information about the samples can be added. Information such as disease type, age, treatment, or sex can be annotated to the data by assigning the Attributes for each sample.

Certain tasks in Partek Flow, such as Gene-Specific Analysis, require that samples be assigned attributes in order to do statistical comparisons between groups or samples. As attributes are added to the project, additional columns in the sample management table will be created.

Sample attributes

Attributes can be managed or created within a project. Under the Data tab, click the button to open the Manage attributes page (Figure 9).

To prepare for later data analysis using statistical tools, attributes can either be categorical or numeric (i.e., continuous).

Adding a categorical attribute

For categorical attributes, there are two levels of visibility. Project-specific categorical attributes are visible only within the current project. System-wide categorical attributes are visible across all the projects within the Partek Flow server, and are useful for maintaining uniformity of terms. Importing samples in a new project will retain the system-wide attributes, but not the project-specific attributes.

A feature of Partek Flow is the use of controlled vocabulary for categorical attributes, allowing samples to be assigned only within pre-defined categories. It was designed to effectively manage content and data and allow teams to share common definitions. The use of standard terms minimizes confusion.

To add a categorical attribute in the Manage attributes page, click the Add new attribute (Figure 10). In the dialog box, type a Name for the attribute, select the Categorical radio button next to Attribute type, select the visibility of the attribute and then click the Add button.

Repeat the process for additional attributes of the samples in your study. When done, click Back to sample management table. Categorical attributes will default to Project-specific visibility.

Click an attribute name to drag and drop can change the order of the attributes displayed on the data tab. Click on group name to drag and drop vertically can change the order of the group name, which can be reflected on visualization.

Adding a numeric attribute

To add a numeric attribute in the Manage attributes page, click the Add new attribute. In the dialog box (Figure 13), type a Name for the attribute, select the Numeric radio button next to Attribute type, and then click the Add button. Some optional parameters for numeric attributes include the Minimum value, Maximum value, and Units. When done, click Add to return to the Manage attributes page. Repeat the process add more numeric attributes. When done, click Back to sample management table.

Adding a system-wide attribute

Since system-wide attributes do not have to be created by the current user, they only need to be added to the sample management table in a project.

In the Data tab, click Add a system-wide attribute button. In the dialog box that follows (Figure 14), a drop down menu is located next to Add attribute where you can select the System-wide attribute you would like to add to the project. Once selected, it will be recognized automatically as either Categorical, system-wide or the Numeric attribute.

For an System-wide categorical attribute, the different categories are listed and you have the option of pre-filling the columns with N/A (or any other category within the attribute). Click Add column and you will return to the Data Tab.

Assigning categories or values to attributes

After adding all the desired attributes to a project, the sample management table will show a new column for each attribute (Figure 15). The columns will initially as "N/A", as the samples have not yet been categorized or assigned a value. To edit the table, click Edit attributes. Assign the sample attributes by using a drop down for categorical attributes (controlled vocabulary) or typing with a keyboard for numeric attributes.

When all the attributes have been entered, click Apply changes and the sample management table will be updated. After editing the sample table, make sure there are no fields with blank or N/A values before proceeding. To rename or delete attributes, click Manage attributes from the Data tab to access the Manage attributes page.

Assigning attributes using a Sample Annotation Text File

Another way to assign attributes to samples in the Data tab is to use a text file that contains the table of attributes and categories/values. This table is prepared outside of Partek Flow using any text editing software capable of saving tab-separated text files.

Using a text editor, prepare a table containing the attributes. An example is shown in Figure 16. There should only be one tab between columns with no extra tabs after the last column. In this particular example, the first column contains the filename and the text file is saved as Sampleinfo.txt.

The first row of the table in the text file contains the attributes (as headers). The first column of the table in the text file, regardless of the header of the first column, should contain either the sample names or the file names of the samples already added in Partek Flow. The first column is the unique identifier that will match the samples to the correct values or categories.

To upload sample attributes, click Assign sample attributes from a file in the Data tab. Then indicate where the attribute text file is stored and navigate to it. Partek Flow will parse the text file and present attributes that will be available for import (Figure 17).

Select the attributes you want to import by clicking the Import check box. Imported attributes that do not currently exist in the project will create new project-specific attributes.

You can change the name of a specific attribute by editing the Attribute name text box. Columns containing letter characters are automatically selected as categorical attributes. Columns containing numbers are suggested to be numeric attributes and can be changed to categorical using the drop down menu under Attribute type.

Guidelines for preparing the sample annotation text file

• The first column is always the unique identifier and can refer only to File names or Sample names.

• If using Sample names in the first column, they must match the entries of the Sample name column in the Sample management table.

• If using File names in the first column, use the filenames shown in the fastq column of the expanded sample management table (see Figure 4) then add the extension .gz. All filenames must include the complete file extension (e.g., Samplename.fastq.gz).

• The header name of the first column of the table (top left cell of our text table) is irrelevant but should not be left blank. Whether the first column contains File names or Sample names will be chosen during the process.

• The last column cannot have empty values

• Missing data (blank cells) can only be handled if the attribute is numeric. If it is categorical, please put a character in it.

It is advisable to use Sample name as the first column identifier when:

• Samples are associated with more than one file (for instance, paired-end reads and/or technical replicates).

• The files were imported in the SRA format (from the Sequence Read Archive database). In Partek Flow, they are automatically converted to the FASTQ format. Consequently, their filenames would change once they are imported. The new file names can be seen by expanding the sample management table, the new extension would be .fastq.gz.

If attributes are assigned from two different text files, the following will happen:

• If the previous attributes have the same header and type (both are either categorical or numeric), the values are overwritten.

• If there are different/additional headers on the "second round" of assignment, these new attributes will be appended to the table.

• For numeric attributes, a "blank" value will not override a previous value.

Use of attributes as Optional columns in task report tables

The attributes assigned to the samples within the Data Tab will be associated with the samples throughout the project. During the course of analysis, Partek Flow tasks generate various tables and any attributes associated with a sample can be included in the table as optional columns. An example is shown in Figure 18 for a pre-alignment QA/QC report where the Optional columns link on the top left of the table reveal the different sample attributes.

Deleting or Renaming samples within a Project

In the Data tab, each sample can be renamed or deleted from the project by clicking the gear icon next to the sample name. The gear icon is readily visible upon mouse over (Figure 19). Sample can only be deleted if no analysis has been performed on the data yet. If any analysis has been performed on the data node, then delete sample operation is invisible. You can perform filter samples in downstream analysis if you want to exclude certain samples in further analysis. Deleting a sample from a project does not delete the associated files, which will remain on the disk.

Additional Assistance

The Attachments tab allows the project owner to add external (i.e. non-Partek Flow) files to a project (for instance, spreadsheets, word documents, manuscripts). To attach a file, go to the Attachments tab (Figure 1). Choose File button invokes the file browser showing the directory structure of the local computer. Select the file that you want to attach and then click on the Upload attachment button. For security reasons, Partek Flow will not allow you to add an executable file.

All added files will be listed in the table under the Attachments tab (Figure 2). The tab will also display file sizes, the user name of the person who uploaded the file and the time it was uploaded. Note that uploaded files will count towards the total size of the project, and thus, if available, to the disk quota of the project owner.

To remove a file, click the icon. To download the attachment, click the icon.

Additional Assistance

The Log tab contains a table of the tasks that are running, scheduled, or those that have been completed within the Partek Flow (Figure 1). It provides an overview of the task progress, enables task management, and links to detailed reports for each task.

Each row of the table corresponds to a task node in the Analyses tab. The list can be sorted according to a specific column using the sort icon .

The Task column lists the name of the tasks. On the left of the task name is a colored circle indicating the layer of this task. The column is searchable by task name. Clicking the task name will open the Task report page. If the task did not generate a report, the link will go to the Task details page.

The User column identifies the task owner. Aside from the user who created the project, only collaborators and users with admin privileges can start tasks in a project. Clicking a name in the User column will display the corresponding User profile.

The End column shows when the task was completed. It will show the actual time for completed tasks, and the estimated time for running tasks. These estimates improve in accuracy as more tasks are completed in the current Partek Flow instance.

The Status column displays the current status of the task, such as Waiting, Running, Done, Canceled. If the task is currently running, a status progress bar will appear in the column. Once completed, the status of a task will be Done and the End column will be updated with the completion time.

A waiting task may be waiting for upstream tasks to complete ( ) or waiting for more computing resources to be available ( ).

The Action column contains the cancel button ( ) while a task is queued or running. Clicking this button will cancel the task. A trash icon ( ) will appear in the Action column for completed, canceled or failed tasks, and will allow the task to be deleted from the project. Deleting a task in the Queue tab will remove the corresponding nodes in the Analyses tab. Unless the user has admin privileges, a user may only cancel and delete a task that he/she started. The User, End, and Status columns may be used to filter ( ) the table.

Additional Assistance

If you need additional assistance, please visit our support page to submit a help ticket or find phone numbers for regional support.

• Project details

• Members

Project details

The Project details section shows the Name of the project as well as (optional) a textual project Description and a Thumbnail (picture) (Figure 1).

The owner and collaborators (if any) can customize the Description and Thumbnail entries by pushing the orange Edit project details button (Figure 2). The fields can now be edited to:

• Rename the project (names are limited to 30 characters). The original project Name is the one selected when creating the project

• Add or change a project description (up to 2000 characters)

• Add or change a thumbnail of the project (supported formats are .jpg, .bmp, .gif, .png; the maximum size of the image file is 10 MB)

The Description accepts hyperlinks starting with "http://" or "https://" and if selected, will open a new tab browser to navigate to the website. This description will be also displayed to collaborators and administrators on the Partek® Flow® Home Page. Choose File button launches a file browser showing directory structure of the local computer from which the thumbnail image file will be uploaded. Alternatively, Clear thumbnail button removes the current thumbnail.

Once all the edits have been made, push Save to accept (or Cancel to reject).

If a thumbnail has been added, it will appear on the Project details tab (Figure 3) and on the home page of Partek Flow, on the Details tab of the project.

Members

The Members section provides an overview of users associated with a particular project and enables project creators (owners) and administrators to add collaborators (Figure 1). A user (without administrator status) has to be specified as a collaborator in a project to be able to access the project in his/her home folder and to perform tasks.

To add a collaborator, use the Add member drop-down menu. The drop-down menu will list users you are collaborating with on any project on the current instance of Partek Flow. Click a user name in the drop-down list and then click the button to add them as a collaborator. To add a user you have not collaborated with before, type their exact username (e.g., jsmith) and click the button to add them. Depending on the collaborator's preference settings, an email notification may be sent to the email address associated with their user account. To delete a collaborator, select the next to their username (you will be asked for confirmation).

Pushing the pencil icon (Pencil icon](../../../.gitbook/assets/pencil-icon.png)) by a project member can result in two dialogs, depending on the status of the member. For a collaborator or a viewer, the pencil icon changes the member's role (e.g. from a Viewer to a Collaborator) (Figure 4).

Moreover, the project owner can transfer the ownership to another user account (one of the accounts already available at the current instance of Partek Flow) using the New owner dropdown list (Figure 5). The previous (old) owner can remain as a project collaborator, with the help of the matching option.

If email notifications are turned on for project ownership transfer, an email dialog box appears. This can be used to add additional text to the notification email body (Figure 6).

After samples have been added and associated with valid data files, in a Partek Flow project, a data node will appear in the Analyses tab (Figure 1). The Analyses tab is where different analysis tools and the corresponding reports are accessed.

• Data and Task Nodes

• The Context Sensitive Menu

• Running a Task

• Cancelling and Deleting Tasks

• Task Results and Task Actions

• Layers

• Collapsing Tasks

• Downloading Data

Data and Task Nodes

The Analyses tab contains two elements: data nodes (circles) and task nodes (rounded rectangles) connected by lines and arrows. Collectively, they represent a data analysis pipeline.

Data nodes (Figure 2) may represent a file imported into the project, or a file generated by Partek Flow as an output of a task (e.g., alignment of FASTQ files generates BAM files).

Missing image Figure 2. The Analyses tab showing a data node of unaligned reads

Task nodes (Figure 3) represent the analysis steps performed on the data within a project. For details on the tasks available in Partek Flow, see the specific chapters of this user manual dedicated to the different tasks.

The Context Sensitive Menu

Clicking on a node reveals the context sensitive menu, on the right side of the screen.

Only the tasks that are available for the selected data node will be listed in the menu. For data nodes, actions that can be performed on that specific data type will appear.

In Figure 4, a node that contains Unaligned reads is selected (bold black line). The tasks listed are the ones that can be performed on unaligned data (QA/QC, Pre-analysis tools, and Aligners).

To hide the context sensitive menu, simply click the symbol on the upper left corner of the context sensitive menu. Clicking the triangles will collapse ( ) or expand ( ) the different categories of tasks that are shown.

After a task is performed on a data node, a new task node is created and connected to the original data node. Depending on the task, a new data node may automatically be generated that contains the resulting data. For details of individual tasks, see Task Menu.

In Figure 5, alignment was performed on the unaligned reads. Two additional nodes were added: a task node for Align reads and an output data node containing the Aligned reads.

Running a Task

To run a task, select a data node and then locate the task you wish to perform from the context sensitive menu. Mouse over to see a description of the action to be performed. Click the specific task, set the additional parameters (Figure 6), and click Finish. The task will be scheduled, the display will refresh, and the screen will return to the project's Analyses tab.

In Figure 6, the STAR aligner was selected and the choices for the aligner index and additional alignment options appeared.

Tasks that are currently running (or scheduled in the queue) appear as translucent nodes. The progress of the task is indicated by the progress bar within the task node. Hovering the mouse pointer over the node will highlight the related nodes (with a thin black outline) and display the status of the task (Figure 7).

If a task is expected to generate data nodes, expected nodes appear even before the task is completed. They will have a lighter shade of color to indicate that they have not yet been generated as the task is still being performed. Once all tasks are done, all nodes would appear in the same shade.

Cancelling and Deleting Tasks

Tasks can only be cancelled by the user that started the task or by the owner of the project. Running or pending tasks can be canceled by clicking the right mouse button on the task node and then selecting Cancel (Figure 8). Alternatively, the task node may be selected and the Cancel task selected from the context sensitive menu.

A verification dialog will appear (Figure 9) asking to confirm the task cancellation, the cancelled tasks will remain in the Analyses tab but will be flagged by gray x circles on the nodes (Figure 10).

Data nodes connected to incomplete tasks are also incomplete as no output can be generated (Figure 10).

To delete tasks from the project click the right mouse button on the task node and then select Delete (Figure 11). Alternatively, click the task node and select Delete task from the context sensitive menu.

A verification dialog will appear (Figure 12). A yellow warning sign will show up if there some downstream tasks performed by collaborators will be affected. Deleting the tasks output files optional. If this is not selected, the task nodes will disappear from the Analyses tab but the output files will remain in the project output directory.

Task Results and Task Actions

Selecting a task node will reveal a menu pane with two sections: Task results and Task actions (Figure 13).

Items from the Task results section inform on the action performed in that node. Certain tasks generate a Task report (Figure 14), which include any tables or charts that the task may have produced.

The Task details shows detailed diagnostic information about the task (Figure 15). It includes the duration and parameters of the task, lists of input and output files, and the actual commands (in the command line) that were run.

Additionally, the Task details page would contain the error logs of unsuccessful runs. The user can download the logs or send them directly to Partek. This page plays an important role in diagnosing and troubleshooting issues related to task.

Double clicking on a task node will show the Task report page. However, if no report was generated, the user will be directed to the Task details page.

In the Task actions sections, the selected task can be Re-run w/new parameters, and in case it is part of a pipeline that includes additional tasks after it, running the Downstream tasks is an option. Re-running tasks will result in a new layer being made in the Analyses tab.

Another action available for a task node is Add task description (Figure 16), which is a way to add notes to the project. The user can enter a description, which will be displayed when the mouse pointer is hovered over the task node.

Layers

It is common for next-generation sequencing data analysis to examine different task parameters for optimization. Users may want to modify an upstream step (e.g. alignment parameters) and inspect its effect on downstream results (e.g. percent aligned reads).

The implementation of Layers makes optimizations easy and organized. Instead of creating separate nodes in a pipeline, another set of nodes with a different color is stacked on top of previous analyses (Figure 17). To see the parameters that were changed between runs, hover the mouse icon over the set of stacked task nodes and a pop-up balloon will display them. The text color signifies the layer corresponding to a specific parameter.

Layers are formed when the same task is performed on the same data node more than once. They are also formed when a task node is selected and the Re-run it w/new parameters is selected in the context sensitive menu. This will allow the users to change the options only for the selected task. The user may choose to re-run the task to which the changes have been made, as well as all the downstream tasks until the analysis is completed. To do so, select Re-run w/new parameters, downstream tasks from the context sensitive menu.

To select a different layer, use the left mouse button to click on any node of the desired layer. All the nodes associated with the selected layer have the same color and when clicked will be displayed on the top of the stack.

Collapsing Tasks

Addition of task and resulting data nodes to project may lead to creation of long pipelines, extending well beyond the border of the canvas (Figure 18).

In that case, the pipeline can be collapsed, to hide the steps that are (no longer) relevant. For example, once the single-cell RNA-seq data has been quantified, Single cell counts data node will be a new analysis start point, as the subsequent analyses will not focus on alignment, UMI deduplication etc. To start, right-click on the task node which should become a boundary of the collapsed portion of your pipeline and select Collapse tasks (Figure 19).

All the tasks on that layer will turn purple. Then left-click the task which should be the other boundary of the collapsed portion. All the tasks that will be collapsed will turn green and a dialog will appear (Figure 19). In the example shown in Figure 19, the tasks between Trim tags and Quantify barcodes will be collapsed. Give the collapsed section a name (up to 30 characters) and select Save (Figure 20).

The collapsed portion of the pipeline is replaced by single task node, with a custom label ("Single cell preprocessing"; Figure 21).

To re-expand the pipeline double click on the task node representing the collapsed portion of the pipeline. Alternatively, single click on the node and select Expand... on the context-sensitive menu. Within the same menu, you can also preview the contents of the collapsed task by selecting View... (Figure 22).

Downloading data

Data associated with any data node can be downloaded using the Download data link in the context sensitive menu (Figure 23). Compressed files will be downloaded to the local computer where the user is accessing the Partek Flow server. Note that bigger files (such as unaligned reads) would take longer to download. For guidance, a file size estimate is provided for each data node. These zipped files can easily be imported by the Partek® Genomics Suite® software.

Additional Assistance

If you need additional assistance, please visit our support page to submit a help ticket or find phone numbers for regional support.

This tutorial gives an overview of RNA-Seq analysis with Partek Flow. It will guide you through creating an RNA-Seq analysis pipeline. The goals of the analysis are to create a list of differentially expressed genes, visualize these gene expression signatures by hierarchical clustering, and interpret the gene lists using gene ontology (GO) enrichment.

This tutorial will illustrate:

• Importing the tutorial data set

• Adding sample attributes

• Running pre-alignment QA/QC

• Trimming bases and filtering reads

• Aligning to a reference genome

• Running post-alignment QA/QC

• Quantifying to an annotation model

• Filtering features

• Normalizing counts

• Exploring the data set with PCA

• Performing differential expression analysis with DESeq2

• Viewing DESeq2 results and creating a gene list

• Viewing a dot plot for a gene

• Visualizing gene expression in Chromosome view

• Generating a hierarchical clustering heatmap

• Performing biological interpretation

• Saving and running a pipeline

Description of the Data Set

This tutorial uses a subset of the data set published in Xu et al. 2013 (PMID: 23902433). In the experiment, mRNA was isolated from HT29 colon cancer cells treated with the drug 5-aza-deoxy-cytidine (5-aza) at three different doses: 0μM (control), 5μM, or 10μM. The mRNA was sequenced using Illumina HiSeq (paired end reads). The goal of the experiment was to identify differentially expressed genes between the different treatment groups.

Additional Assistance

If you need additional assistance, please visit our support page to submit a help ticket or find phone numbers for regional support.

• Project Deletion

• Selecting Files for Deletion

• Project Import and Export

Project Deletion

A project may be deleted from the Partek Flow server using the button on the upper right side of the Project View page (Figure 1).

Alternatively, you can also delete your projects directly from the Homepage by clicking Delete project under the Actions column (Figure 2).

Selecting Files for Deletion

After clicking the Delete project, a page displaying all the files associated with the project appears. Clicking the triangle will expand the list. Select the files to be deleted from the server by clicking the corresponding checkboxes next to each file (Figure 3). By default, all output files generated by the project will be deleted.

If you wish to delete the input files associated with the project, you can do that as well by clicking the Input files checkbox. Note that a warning icon appears next to input files that are used in other projects (Figure 4). These cannot be deleted until all projects associated with them are deleted.

Project Import and Export

Every project can be exported before it is removed from the server. By exporting old projects, you can free up some storage on your server. You can import the exported project back into Partek Flow later if needed.

Exporting a Project

Open a project and on the analysis page, click on the gear button and choose Export project (Figure 1). You can also export the project directly from Partek Flow home page by clicking the icon under the Action column (Figure 2).

When you export a project, you will be asked whether to include library files to export or not. If you choose Yes, the current version of library files used in the project will be archived and you can reproduce the result when you later import the project and re-do the analysis. However, it will make the archive size bigger. If you choose No, the library files will not be exported. Note that when you import the project later, you can only use the available version of needed library files to re-do the same analysis, and the results might not be the same.

Importing a Project

The Import project option is under Projects drop-down menu on the top of the Partek Flow page (Figure 5). This can be accessed on any Partek Flow page.

The input of this option is the zipped file of the exported project. Browse to the file location which can either be the Partek Flow server, a local machine, or a URL. The zip file first needs to be uploaded to the Partek Flow server (if it is not on the server already), and then Partek Flow will unpack the zip file into a project. The project name will be the same as the exported project name. If the project with the same name already exists, the imported project will have a number appended to it (e.g., ProjectName_1).

The owner of the imported project will be the user that imported it.

Additional Assistance

If you need additional assistance, please visit our support page to submit a help ticket or find phone numbers for regional support.

How to import a study from GEO / ENA

If a project is publicly available in the Gene Expression Omnibus (GEO) and European Nucleotide Archive (ENA) databases, you can import associated FASTQ files, sample attributes, and project details automatically into Partek Flow.

• Click Projects at the top of the page

• Click Import project

• Choose GEO / ENA project for Select files from

• Type the BioProject ID or the GEO Accession number

The format of a BioProject ID is PRJNA followed by one to six numbers (e.g., PRJNA291540). The format of a GEO Accession number is GSE followed by one to five numbers (e.g., GSE71578).

• Click Import project at the bottom

The Analyses tab will include an Unaligned reads data node once the data download has started (Figure 3). It may take a while for the download to complete depending on the size of the data. FASTQ files are downloaded from the ENA BioProject page.

Common Issues

Error Message - The project did not yield any data. Double-check the project ID, or try importing the data manually

If the study is not publicly available in both GEO and ENA, project import will not succeed.

The project was imported, but the Analyses tab is empty and there are no FASTQ files

If there is an ENA project, but the FASTQ files are not available through ENA, the project will be created, but data will not be imported.

Something is missing or the import failed

A variety of other issues and irregularities can cause imports to not succeed or partially succeed, including, but not limited to, a BioProject having multiple associated GSE IDs, incomplete information on the GEO or ENA page, and either the GEO or ENA project not being publicly available.

FAQ

What are GEO and ENA?

The Gene Expression Omnibus (GEO) and the European Nucleotide Archive (ENA) are web-accessible public repositories for genomic data and experiments. Access and learn more about their resources at their respective websites:

• GEO - https://www.ncbi.nlm.nih.gov/geo/

• ENA - https://www.ebi.ac.uk/ena

How do I know if a GEO project is also in ENA?

You can search ENA using the GEO ID (e.g., GSE71578) to check if there is a matching ENA project (Figure 6).

Additional Assistance

Attributes describe samples. Examples of sample attributes include treatment group, age, sex, and time point. Attributes can be added individually in the Metadata tab or in bulk using a text file. In this tutorial, we will add one attribute, 5-AZA Dose, manually.

• Click the Metadata tab

• Click Manage under Sample attributes (Figure 1)

• Click Add new attribute (Figure 2)

• To configure a new attribute, at Name, type in 5-AZA Dose as the name of the attribute

• Click Add to add 5-AZA Dose as a categorical, project-specific attribute (Figure 3)

• Name the first New category 0uM

• Click the green plus icon to add category (Figure 4)

• Repeat for two additional categories, 5uM and 10uM (Figure 5)

• Click Back to metadata tab

The data table now includes an Attribute column for 5-AZA Dose (Figure 6). Next, we need to assign samples attribute categories for 5-AZA Dose.

• Select Assign values

The option to edit the 5-AZA Dose field for each sample will appear as a drop-down menu (Figure 7).

• Select the 5-AZA Dose text box for a sample to bring up a drop-down menu with the 5-AZA Dose attribute categories (0uM, 5uM, 10uM)

• Use the drop-down menus to add a treatment group for each sample

The first three samples (SRR592573-5) should be 0uM, the next three samples (SRR592576-8) should be 5uM, and the final three samples (SRR592579-81) should be 10uM (Figure 8).

• Click Apply changes

The data table will now show each a 5-AZA Dose attribute for each sample.

Additional Assistance

With attributes added, we can begin building our pipeline.

• Click the Analyses tab

In the Analysis tab, data are represented as circles, termed data nodes. One data node, mRNA, should be visible in the Analysis tab (Figure 1).

• Click the mRNA node

Clicking a data node brings up the context-sensitive task menu with tasks that can be performed on the data node (Figure 2).

Pre-alignment QA/QC assesses the quality of the unaligned reads and will help us determine whether trimming or filtering is necessary.

• Click Pre-alignment QA/QC in the QA/QC section of the task menu

• Click Finish to run the task with default settings

Running a task creates a task node, e.g. the blue rectangle labeled Pre-alignment QA/QC (Figure 3), which contains details on the task and a report.. While tasks have been queued or are in progress they have a lighter color. Any output nodes that the task will generate are also displayed in a lighter color until the task completes. Once the task begins running, a progress bar is displayed on the task node.

• Click the Pre-alignment QA/QC node

The context-sensitive task menu (Figure 4) shows the option to view the Task report and the Task details. You can also access a task report by double-clicking on a task node.

• Click Task report

Pre-aligment QA/QC provides information about the sequencing quality of unaligned reads (Figure 5). Both project level summaries and sample-level summaries are provided.

• Click sample SSR592573 in the data table of the report to open its sample-level report

The Average base quality score per position graph in the upper right-hand panel (Figure 6) gives the average Phred score for each position in the reads.

A Phred score is a measure of base call accuracy with a higher score indicating greater accuracy.

By convention, a score above 20 is considered adequate. As you can see, the standard error bars in the graph show that some reads have quality scores below 20 for some of their base pair calls near the 3' end.

Based on the results of Pre-alignment QA/QC, while most of the reads are high quality, we will need to perform read trimming and filtering. For more information about the information included in the task report, please see the Pre-alignment QA/QC user guide.

• Click RNA-Seq 5-AZA to return to the Analyses tab

Additional Assistance

The tutorial data set includes 9 samples equally divided into 3 treatment groups. Sequencing was performed by an Illumina HiSeq (paired-end reads), but the workflow can be easily adapted for data generated by other sequencers. Each sample has 2 fastq files for a total of 18 fastq files.

You can obtain the tutorial data set through Partek Flow.

• Click your avatar

• Click Settings in the drop-down menu (Figure 1)

At the top of the System information page, there is a section labeled Download tutorial data (Figure 2).

• Click RNA-Seq 5-AZA to download the tutorial data set

A new project will be created and you will be directed to the Analyses Tab. The data will be downloaded automatically (Figure 3) and imported into your project. Because this is a tutorial project, there is no need to click on Add data as it will be done automatically.

At first the project is empty, but the file download will start automatically in the background. You can wait a few minutes then refresh your browser or you can monitor the download progress using the Queue.

• Click Queue

• Click View Queued Tasks in the drop-down menu

The Queued tasks page will open (Figure 4).

• Click Projects

• Click RNA-Seq 5-AZA in the drop-down menu

The Analyses tab will open (Figure 5). If you download has completed, you will see a blue circle titled mRNA.

Once the download completes, the sample table will appear in the Metadata tab.

• Click the Metadata tab The Metadata tab includes the sample table with the names of each imported sample (Figure 6).

In the next section of the tutorial, we will add a sample attribute that indicates the treatment group of each sample.

Additional Assistance