Sonic HPC

Research IT provides High Performance Computing through our Sonic HPC, for more details on getting an account, user guide, software provided and cluster hardware see below.

For help, advice or additional software requirements please contact us

Open All

Please note : If you cannot see a login box for this form, you must first log into UCD Connect mail in a separate browser tab.  Go to UCD Connect and click on the Mail icon.  Once logged in, return to and refresh this page and you will see the booking form.

 

 "Getting Started on Sonic HPC Cluster" is now available on Brightspace. This is a video led presentation suitable for all researchers who have large computational requirements.  This series of videos  which takes about an hour gives a brief overview of what HPC (High Performance Computing) is, how to access the UCD Campus cluster Sonic and some step by step tutorials on submitting jobs, transferring data and accessing GPU's (Graphics Processing Units) on Sonic. The course is available through Brightspace and bookable through UCD Booking Centre 

Book here (please be logged into Infohub)

Research IT HPC Cluster 

Cluster Name Sonic
Cluster Usage Real Time Information
No. of Compute Nodes 53
Total Number of Cores 1468 (hyperthreading disabled)
Processor Speed

20 nodes with 2 Intel Xeon 6152, 2.1GHz, 22 cores each

4 nodes with  2 Intel Xeon 6140, 2.3Ghz, 18 cores each

 24 nodes with 2 Intel E5-2660 v2, 2.2Ghz, 10 cores each

4 nodes with with 2 Intel E5-2620 v2, 2.1Ghz, 6 cores each

1 node with 2 Intel E5-2620 v3, 2,4Ghz 6 cores each

Memory per Nodes 128Gb (24) 256Gb (7)  384 (20) 786 (1) 1.5TB (1)
Interconnect Infiniband QDR 40GBs
Home Directory Quota 50GB
Scratch Space 180TB 
Additional Nodes Types 

MEM2 - High Memory - 765 GB RAM 4 * 2.4GHz (6 cores)

MEM3 - High Memory - 1.5TB RAM 

3 GPU Servers each with 2 Nvidia Tesla V100 . CPU  2.1Ghz RAM =256GB

 

 

 

Getting Started:

Access to the Sonic HPC Cluster is through an SSH command line interface on the Sonic login node - login.ucd.ie on port 22

ssh username@login.ucd.ie

If you are using windows, you'll need an SSH client such as Putty to SSH to the login node. You will also need to you a transfer client such as WinSCP. Mac OSX or Linux users can use any Terminal application.

The Sonic Cluster is a Linux environment. Users not familiar with working in a Linux environment should familiarise themselves with the basic file and directory operations of the Linux command line interface. There are a good series of introductory tutorials available on the University of Surrey's Electronic Engineering Department website

Once you are logged in to the cluster, you will find yourself in your home directory. On Sonic, your home directory is at /home/people/username (where usernameis your UCD Connect username).

Each shared group user has a disk quota of about 50GB. Within your home directory you will see a linked directory called "scratch" - this is an area that users can use to temporarily store working data. As this  Please note that this data is liable to be deleted at short notice and as such is not suitable for long term storage. Users of a contributory group have larger quotas

File Storage:

Each shared group user has a home directory at /home/people/username which can be used to store approximately 50GB of data - this storage area is intended only for Research Data relevant to the users cluster usage and should not be used as a private data store.

Additionally, users have access to extra storage via the 'scratch/' directory/symlink within your home directory. This storage area is for temporary short-term storage of data required for input into, or output from, your currently running jobs. This is not a long term stoage platform and files not modified  for 6 months will be deleted 

Submitting Jobs:

To submit a job to the Sonic Cluster, you will need to use the sbatch command to queue your job. sbatch is part of SLURM which controls the queues on Sonic.

Your job will first need to be saved in a script which will need to pass certain parameters to Slurm. These parameters should be specified at the beginning of your jobs script.

An example of a script using some of these parameters is shown below:

#!/bin/bash -l

# Set the number of nodes

#SBATCH -N 1

# Set the number of tasks/cores per node required 
#SBATCH -n 3


# Set the walltime of the job to 1 hour (format is hh:mm:ss)
#SBATCH -t 01:00:00

# E-mail on begin (b), abort (a) and end (e) of job
#SBATCH --mail-type=ALL

# E-mail address of recipient
#SBATCH --mail-user=myemailaddress@ucd.ie

# Specifies the jobname
#SBATCH --job-name=myjob

# Change working directory to current directory
cd $SLURM_SUBMIT_DIR
# Your code here! (This example just prints the current date and exits!)
date;

To run the script, type:

$sbatch myjob.sh

This will then return something like:

Submitted batch job 331

where 331 is your job ID.

See here for more detail on sbatch parameters. There are also two example scripts present in your home folder: samplejob.sh and samplempi.sh

To view your own queued jobs, you can use the 'squeue' command, which will show you your job id, the name of the job,the jobs current status, and the queue (called a partition in slurm) and the length of time in which your jobs is running.

sacct -j jobid will give detailed of your job once it is finished 

You can delete a job from the queue by using "scancel <jobid>". 

Specifying the GPU nodes 

If you wish to avail of the GPU enabled worker nodes (equipped with 2 Nvidia Telsa V100's each). To request a GPU server and all its cores you should submit to the GPU queue/partion with this command .  There is a sample GPU job in your home directory called samlegpu.sh which has all the setting you to need to use a GPU nodes and that you should amend as required. To submit enter the following 

sbatch --partition=gpu   samplegpu.sh If you are a computer science user please use this command

sbatch --partition=csgpu samplegpu.sh

Specifying the High Memory nodes

If you wish to avail of the high memory server sonicmem2 equipped with 768 GB of RAM and sonicmem3 equipped with 1.5TB of RAM you should add this line to your job submission script

sbatch --nodelist=sonicmem3 cluster

Queues:

For shared queues there is one standard queue which allows job to run up to a maximum of 10 days

As contributors to the cluster users from the School of Computer Science jobs are sent to the cs queue

Interactive Jobs 

If your jobs require user intervention then you need to run an interactive job which requests an bash shell on a worker node. The syntax for submitting an interactive job is 

srun --time=03:00:00 -c6 --pty bash

--time referes to the length of time you require the shell prompt

-c6 means the number of cores you require 

Graphical Interactive Jobs

To run a graphical interactive job you must forward your display to sonic and you must submit an interactive job to the graphical queue. To forward your display to sonic on linux use the following command ssh -X username@sonic.ucd.ie . For windows you need to run a service such as Xming and select "Enable X11 forwarding " on your putty session which is on the left hand side under SSH and X11. Once on sonic you run this command to request a shell session on a node that has graphical components installed . For 

srun --partition=graphical --time=01:00:00 --x11 --pty bash

Once you run this command the job will queue and will appear to "hang" after the message waiting on resources returns, this is normal typically you will wait about 30 seconds before getting a prompt. As these are interactive jobs if your session ends  the job will also end as well so please save output as you go along .

How many Jobs can I run at time (This information is subject to change)

Currently each standard shared user is allowed to use up to 48 cores at any one time. When the cluster is busy this number is affected by the amount of resources availble. Jobs with a smaller number of cores requested typically start earlier than those jobs wishing to reserve large amount of cores. When the cluster is quiet, a hard limit will allow users requesting more than 48 cores access of up to 260 cores, to utilise any free resources.

Contributory uses are allowed to utilise more cores at any one time that shared users, the amount of maximum. 

What Software is installed

The best way to find out what centrally installed software is availble is with the module avail command

Is there an easy way to load environment variables into my path

Yes, using the modules package you can load environmental for different packages into your path. More information on this can be found here

Access off Campus 

There is no direct access to the new cluster externally . If you have a @ucd.ie email address you can apply for a UCD VPN account using the VPN request form on VPN webpage  and access the cluster using this. Please ensure you are logged into UCD Connect to access the form 

If you have a @ucdconnect.ie email address you can apply for a bastion host account by emailing ithelpdesk@ucd.ie .

How do I use the Bastion Host ?

Windows User Guide for Bastion Host

User Guide for Bastion Host for Linux Clients

The following is the hardware spec of what was purchased for the upgrade of the Sonic Cluster. Interconnect has been upgraded to infiniband and a parallel file system (BeeGFS) has been implemented across 4 storage servers. New login and head nodes complete the infrastracuture environment . There are 20 new standard compute nodes which no longer have hyperthreading enabled and contain 384GB of RAM available and are equipped with 2 Intel Xeon Gold 6152 (22 cores each) CPU. There are 3 GPU servers each containing 2 Nvidia Tesla V100's. The school of Computer Science has contributed to the hardware purchase of the cluster and it's users have larger entitlements to that hardware. Home directories have quotas implemted at 50GB. The parallel storage is 180GB in size and is shared across the cluster. This storage is only for the computational use and not long term storage of data. In order for the cluster to remain onlines files only than 6 months will be removed off this storage 

Storage Nodes Manu Model CPU Spec Hard Drive Spec Memory/GB 
Storage 1 Dell R740XD  2 X Intel Xeon Gold 6136   24 X 2 TB 10K RPM SAS 12Gbp 384
Storage 2 HP R740XD  2 X Intel Xeon Gold 6136    24 X 2 TB 10K RPM SAS 12Gbp 384
Storage 3 Dell  R740XD 2 X Intel Xeon Gold 6246 24 X 2 TB 10K RPM SAS 12Gbp 384
MetaData 1 Dell R740XD   2 X Intel Xeon Gold 6136    24 X 2.4TB 10K RPM SAS 12Gbps + 2 X 800GB SSD Write Intensive 12GB  384
MetaData  2 Dell R740XD   2 X Intel Xeon Gold 6136   24 X 2.4TB 10K RPM SAS 12Gbps + 2 X 800GB SSD Write Intensive 12GB  384
Login Nodes Manu  Model  CPU Spec   Memory /GB
Login Node  Dell  R640  2 X Intel Xeon Gold 5118   256 
Head Node  Dell  R640  2 X Intel Xeon Gold 5118  256 
Compute Nodes Manu Model  CPU Spec Memory/GB 
Sonic 1 Dell C6220 V2 2 X Intel E5-2660 v2 (2.2Ghz, 10 cores ) 128
Sonic 2 Dell C6220 V2  2 X Intel E5-2660 v2 (2.2Ghz, 10 cores ) 128  
Sonic 3 Dell C6220 V2  2 X Intel E5-2660 v2 (2.2Ghz, 10 cores ) 128
Sonic 4 Dell C6220 V2  2 X Intel E5-2660 v2 (2.2Ghz, 10 cores ) 128
Sonic 5 Dell C6220 V2  2 X Intel E5-2660 v2 (2.2Ghz, 10 cores ) 128
Sonic 6 Dell C6220 V2  2 X Intel E5-2660 v2 (2.2Ghz, 10 cores ) 128
Sonic 7 Dell C6220 V2  2 X Intel E5-2660 v2 (2.2Ghz, 10 cores ) 128 
Sonic 8 Dell C6220 V2  2 X Intel E5-2660 v2 (2.2Ghz, 10 cores ) 128
Sonic 21 Dell C6220 V2 2 X Intel E5-2620 v2 (2.1Ghz, 6 Cores) 256 
Sonic 22 Dell C6220 V2 2 X Intel E5-2620 v2  (2.1Ghz, 6 Cores) 256
Sonic 23 Dell C6220 V2 2 X Intel E5-2620 v2  (2.1Ghz, 6 Cores) 256
Sonic 24 Dell C6220 V2   2 X Intel E5-2620 v2  (2.1Ghz, 6 Cores) 256
Sonic 25 Dell C6220 V2  2 X Intel E5-2660 v2 (2.2Ghz, 10 cores ) 128 
Sonic 26 Dell C6220 V2  2 X Intel E5-2660 v2 (2.2Ghz, 10 cores ) 128 
Sonic 27 Dell  C6220 V2  2 X Intel E5-2660 v2 (2.2Ghz, 10 cores ) 128 
Sonic 28 Dell C6220 V2  2 X Intel E5-2660 v2 (2.2Ghz, 10 cores ) 128
Sonic 29 Dell C6220 V2  2 X Intel E5-2660 v2 (2.2Ghz, 10 cores ) 128 
Sonic 30 Dell C6220 V2  2 X Intel E5-2660 v2 (2.2Ghz, 10 cores ) 128
Sonic 31 Dell C6220 V2  2 X Intel E5-2660 v2 (2.2Ghz, 10 cores ) 128 
Sonic 32 Dell C6220 V2  2 X Intel E5-2660 v2 (2.2Ghz, 10 cores ) 128 
Sonic 33 Dell C6220 V2  2 X Intel E5-2660 v2 (2.2Ghz, 10 cores ) 128
Sonic 34 Dell C6220 V2  2 X Intel E5-2660 v2 (2.2Ghz, 10 cores ) 128 
Sonic 35 Dell C6220 V2  2 X Intel E5-2660 v2 (2.2Ghz, 10 cores ) 128
Sonic 36 Dell C6220 V2  2 X Intel E5-2660 v2 (2.2Ghz, 10 cores ) 128
Sonic 37 Dell C6220 V2  2 X Intel E5-2660 v2 (2.2Ghz, 10 cores ) 128
Sonic 38 Dell C6220 V2  2 X Intel E5-2660 v2 (2.2Ghz, 10 cores ) 128 
Sonic 39 Dell  C6220 V2  2 X Intel E5-2660 v2 (2.2Ghz, 10 cores ) 128
Sonic 40 Dell  C6220 V2  2 X Intel E5-2660 v2 (2.2Ghz, 10 cores ) 128
Sonic 43 Dell  R640 2 X Intel Xeon Gold 6152 (2.1GHz, 22 cores) 384
Sonic 44 Dell  R640 2 X Intel Xeon Gold 6152 (2.1GHz, 22 cores) 384 
Sonic 45 Dell  R640 2 X Intel Xeon Gold 6152 (2.1GHz, 22 cores) 384 
Sonic 46 Dell  R640 2 X Intel Xeon Gold 6152 (2.1GHz, 22 cores) 384
Sonic 47 Dell  R640 2 X Intel Xeon Gold 6152 (2.1GHz, 22 cores) 384 
Sonic 48 Dell  R640 2 X Intel Xeon Gold 6152 (2.1GHz, 22 cores) 384 
Sonic 49 Dell  R640 2 X Intel Xeon Gold 6152 (2.1GHz, 22 cores) 384 
Sonic 50 Dell  R640 2 X Intel Xeon Gold 6152 (2.1GHz, 22 cores) 384 
Sonic 51 Dell  R640 2 X Intel Xeon Gold 6152 (2.1GHz, 22 cores) 384 
Sonic 52 Dell  R640 2 X Intel Xeon Gold 6152 (2.1GHz, 22 cores) 384 
Sonic 53 Dell  R640 2 X Intel Xeon Gold 6152 (2.1GHz, 22 cores) 384 
Sonic 54 Dell  R640 2 X Intel Xeon Gold 6152 (2.1GHz, 22 cores) 384 
Sonic 55 Dell  R640 2 X Intel Xeon Gold 6152 (2.1GHz, 22 cores) 384 
Sonic 56 Dell  R640 2 X Intel Xeon Gold 6152 (2.1GHz, 22 cores) 384 
Sonic 57 Dell  R640 2 X Intel Xeon Gold 6152 (2.1GHz, 22 cores) 384 
Sonic 58 Dell  R640 2 X Intel Xeon Gold 6152 (2.1GHz, 22 cores) 384 
Sonic 59 Dell  R640 2 X Intel Xeon Gold 6152 (2.1GHz, 22 cores) 384
Sonic 60 Dell  R640 2 X Intel Xeon Gold 6152 (2.1GHz, 22 cores) 384
Sonic 61 Dell R640 2 X Intel Xeon Gold 6152 (2.1GHz, 22 cores) 384 
Sonic 62 Dell  R640 2 X Intel Xeon Gold 6152 (2.1GHz, 22 cores) 384 
Sonic 63 Dell R640 2 X Intel Xeon Gold 6252 (2.1Ghz, 24 cores) 384
Sonic 64 Dell R640 2 X Intel Xeon Gold 6252 (2.1Ghz, 24 cores) 384
Sonic 65 Dell  R640 2 X Intel Xeon Gold 6252 (2.1Ghz, 24 cores) 384
Sonic 66 Dell R640 2 X Intel Xeon Gold 6252 (2.1Ghz, 24 cores) 384
Sonic 67 Dell R640 2 X Intel Xeon Gold 6252 (2.1Ghz, 24 cores) 384
GPU Node Manu  Model  CPU Spec  & GPU Spec Memory/GB
GPU 1 Dell R740XD 2 X Xeon 6140  (2.3Ghz, 18 Cores) & 2 Nvidia Tesla V100 (32GB) 256
GPU 2 Dell R740XD  2 X Xeon 6140  (2.3Ghz, 18 Cores) & 2 Nvidia Tesla V100 (32GB) 256 
GPU 3 Dell R740XD  2 X Xeon 6140  (2.3Ghz, 18 Cores) & 2 Nvidia Tesla V100 (32GB) 256 
GPU 4 Dell  R740 2 X Xeon 6240 (2.1Ghz, 18 Cores) & 2 Nvidia Tesla V100 (32GB) 384
GPU 5 Dell R740 2 X Xeon 6240 (2.1Ghz, 18 Cores) & 2 Nvidia Tesla V100 (32GB) 384
High Memory Node Dell  Model CPU Spec

Memory/GB
Mem2 Dell  R730 2 X Intel E5 - 2620 v3 (2.4Ghz, 6 Cores) 768
Mem3 Dell R640 2 X Intel Xeon 6140  (2.3Ghz, 18 Cores) 1536
 

Software on Sonic is installed from source and directed to  the /opt/software directory . Sonic uses "modules" to handle the environmental variables for software installations. Each piece of software installed has its own "module" file written for it .

To find out more details about what software is installed on sonic and how to load the environmental variables into your profile please use the below module commands

 

module list - shows a list of what modules are currently loaded into your session

module avail - shows a list of what modules(software) are available to be loaded

module load modulename - add a particular module to your session

module unload modulename - remove a particular modules from your environment

module purge - remove all loaded modules from your environment

If you do not specify a version of the module then the latest version will be selected automatically.