Bench Clusters

Managing a Bench cluster

Introduction

Workspaces can have their own dedicated cluster which consists of a number of nodes. First the workspace node, which is used for interacting with the cluster, is started. Once the workspace node is started, the workspace cluster can be started.

The cluster consists of 2 components

• The manager node which orchestrates the workload across the members.

• Anywhere between 0 and up to maximum 50 member nodes.

Clusters can run in two modes.

• Static - A static cluster has a manager node and a static number of members. At start-up of the cluster, the system ensures the predefined number of members are added to the cluster. These nodes will keep running as long as the entire cluster runs. The system will not automatically remove or add nodes depending on the job load. This gives the fastest resource availability, but at additional cost as unused nodes stay active, waiting for work.

• Dynamic - A dynamic cluster has a manager node and a dynamic number of workers up to a predefined maximum (with a hard limit of 50). Based on the job load the system will scale the number of members up or down. This saves resources as only as much worker nodes as needed to perform the work are being used.

Configuration

You manage Bench Clusters via the Illumina Connected Analytics UI in Projects > your_project > Bench > Workspaces > your_workspace > Details.

The following settings can be defined for a bench cluster:

Field	Description
Web access	Enable or disable web access to the cluster manager.
Dedicated Cluster Manager	Use a dedicated node for the cluster manager. This means that an entire machine of the type defined at resource model is reserved for your cluster manager. If no dedicated cluster manager is selected, one core per cluster member will be reserved for scheduling. For example, if you have 2 nodes of standard-medium (4 cores) and no dedicated cluster manager, then only 6 (2x3) cores are available to run tasks as each node reserves 1 core for the cluster manager.
Type	Choose between Static and Dynamic cluster members
Scaling interval	For static, set the number of cluster member nodes (maximum 50), for dynamic, choose the minimum and maximum (up to 50) amount of cluster member nodes.
Resource model	The type of machine on which the cluster member(s) will run. For every cluster member, one of these machines is used as resource, so be aware of the possible cost impact when running many machines with a high individual cost.
Economy mode	Economy mode uses AWS spot instances. This halves many compute iCredit rates vs standard mode, but may be interrupted. See Pricing for a list of which resource models support economy pricing.
Storage size	How much storage space (1GB - 16 TB) should be reserved per node as dedicated scratch space, available at /scratch

Operations

Once the workspace is started, the cluster can be started at Projects > your_project > Bench > Workspaces > your_workspace > Details and the cluster can be stopped without stopping the workspace. Stopping the workspace will also stop all clusters in that workspace.

Managing Data in a Bench cluster

Data in a bench workspace can be divided into three groups:

• Workspace data is accessible in read/write mode and can be accessed from all workspace components (workspace node, cluster manager node, cluster member nodes ) at /data. The size of the workspace data is defined at the creation of the workspace but can be increased when editing a workspace in the Illumina connected analytics UI. This is persistent storage and data remains when a workspace is shut down.

• Project data can be accessed from all workspace components at /data/project. Every component will have their own dedicated mount to the project. Depending on the project data permissions you will be able to access it in either Read-Only or Read-Write mode.

• Scratch data is available on the cluster members at /scratch and can be used to store intermediate results for a given job dedicated to that member. This is temporary storage, and all data is deleted when a cluster member is removed from the cluster.

Fast Read-Only Access

All mounts occur in /data/mounts/, see data access and workspace-ctl data.

Managing these mounts is done via the workspace cli /data/.local/bin/workspace-ctl in the workspace. Every node will have his dedicated mount.

For fast data access, bench offers a mount solution to expose project data on every component in the workspace. This mount provides read-only access to a given location in the project data and is optimized for high read throughput per single file with concurrent access to files. It will try to utilise the full bandwidth capacity of the node.

All mounts occur in path /data/mounts/

Show mounts

workspace-ctl data get-mounts

Creating a mount

For fast read-only access, link folders with the CLI command workspace-ctl data create-mount --mode read-only.

workspace-ctl data create-mount --mount-path /data/mounts/mydata --source /data/project/mydata

This has the same effect as using the --mode read-only option because this is applied by default when using workspace-ctl data create-mount .

Removing a mount

workspace-ctl data delete-mount --mount-path /data/mounts/mydata