Data Management

Managing Data for Local App Analysis

Before running an analysis with the Local App, it is important to organize your input files properly and transfer them to a dedicated directory on the server. The steps below outline recommended best practices for managing data efficiently and ensuring successful transfers.

Using a Persistent Terminal Session

When transferring large datasets, it is recommended to use a persistent terminal session tool (such as screen) to prevent interruptions. A persistent session ensures that your transfer continues to run even if you disconnect from the terminal or lose your local connection.

Why use a persistent session?

• Prevents data transfer interruption if your computer sleeps, disconnects, or the terminal closes.
• Allows long-running transfers to complete without requiring you to stay logged in.
• Enables you to reconnect to an ongoing session at any time.

Basic workflow for using a persistent session:

1. Start a new persistent session.
2. Run your data transfer commands within the session.
3. Detach from the session when you need to disconnect.
4. Reattach to the session later as needed.
5. Keep a list of all existing sessions to locate active or detached sessions.

If you are unfamiliar with Linux persistent session tools such as screen, your IT department can assist with setup and usage.

Creating a Dedicated Data Directory

Before transferring your input files, create a dedicated directory on the server to store all data required for a single analysis.

Best practices for your data directory:

• Create a unique folder for each analysis.
• Store only the input files required for the current run inside this folder.
• Avoid keeping older or unnecessary files on the server to conserve storage.

Your IT department can help you create the appropriate directory in the designated data storage area on the server.

Transferring Data Using Reliable File Transfer Tools

Large datasets should be transferred using reliable tools that support integrity checking and resumable transfers. Tools such as rsync ensure that all files are copied completely and efficiently.

Recommended rsync arguments:

• -a preserves file permissions and ensures a complete copy.
• -v shows the files that are being copied.
• -P combines two useful flags (--progress and --partial), which provides a progress bar and, if the transfer is interrupted, allows it to resume from where it left off.

Why use a robust transfer tool?

• Ensures full and accurate data transfer.
• Provides progress visibility during the copy process.
• Automatically resumes incomplete transfers if interrupted.
• Avoids partial or corrupted copies.

General Guidance

• Use a tool capable of handling large folders (such as sequencing run directories).
• Ensure file permissions and directory structure are preserved.
• Work with your IT department, as needed, to perform the transfer using your organization's approved methods.