What is Parallel Computing

Overview

Teaching: 30 min
Exercises: 20 min

Questions

• How do we execute a task in parallel?

• What benefits arise from parallel execution?

• What are the limits of gains from execution in parallel?

• What is the difference between implicit and explicit parallelisation.

Objectives

• Prepare a job submission script for the parallel executable.

Methods of Parallel Computing

The two main types of parallel computing we will discuss are shared memory and distributed memory.

To understand the difference we first need to clarify some terms.

CPU: Unit that does the computations.

Task: One or more CPUs that share memory.

Node: The physical hardware. The upper limit on how many CPUs can be in a task.

Shared Memory: When multiple CPUs are used within a single task.

Distributed Memory: When multiple tasks are used.

Which methods are available to you is largely dependent on the nature of the problem and software being used.

Exercises in this episode will require a copy of the whothis.sh file from the workshop directory.

[yourUsername@mahuika ~]$  cp ../whothis.sh .

Shared-Memory (SMP)

Shared-memory multiproccessing divides work among CPUs or threads, all of these threads require access to the same memory.

Often called Multithreading.

This means that all CPUs must be on the same node, most Mahuika nodes have 72 CPUs.

Shared memory parallelism is what is used in our example script array_sum.r .

Number of threads to use is specified by the Slurm option --cpus-per-task.

Shared Memory Example

Create a new script called example_smp.sl

#!/bin/bash -e

#SBATCH --job-name        smp_job
#SBATCH --output          %x.out
#SBATCH --mem-per-cpu     500
#SBATCH --cpus-per-task   8

bash whothis.sh

then submit with

[yourUsername@mahuika ~]$ sbatch example_smp.sl

Solution

Checking the output should reveal

[yourUsername@mahuika ~]$ cat smp_job.out

I am task #0 running on node 'wbn224' with 8 CPUs

Distributed-Memory (MPI)

Distributed-memory multiproccessing divides work among tasks, a task may contain multiple CPUs (provided they all share memory, as discussed previously).

Message Passing Interface (MPI) is a communication standard for distributed-memory multiproccessing. While there are other standards, often ‘MPI’ is used synonymously with Distributed parallelism.

Each task has it’s own exclusive memory, tasks can be spread across multiple nodes, communicating via and interconnect. This allows MPI jobs to be much larger than shared memory jobs. It also means that memory requirements are more likely to increase proportionally with CPUs.

Distributed-Memory multiproccessing predates shared-memory multiproccessing, and is more common with classical high performance applications (older computers had one CPU per node).

Number of tasks to use is specified by the Slurm option --ntasks, because the number of tasks ending up on one node is variable you should use --mem-per-cpu rather than --mem to ensure each task has enough.

Tasks cannot share cores, this means in most circumstances leaving --cpus-per-task unspecified will get you 2.

Distributed Memory Example

Create a new script called example_mpi.sl

#!/bin/bash -e

#SBATCH --job-name            mpi_job
#SBATCH --output          %x.out
#SBATCH --mem-per-cpu     500
#SBATCH --ntasks          4

srun bash whothis.sh

then submit with

[yourUsername@mahuika ~]$ sbatch example_mpi.sl

Solution

[yourUsername@mahuika ~]$ cat mpi_job.out

I am task #1 running on node 'wbn012' with 2 CPUs
I am task #3 running on node 'wbn010' with 2 CPUs
I am task #0 running on node 'wbn009' with 2 CPUs
I am task #2 running on node 'wbn063' with 2 CPUs

Using a combination of Shared and Distributed memory is called Hybrid Parallel.

Hybrid Example

#!/bin/bash -e

#SBATCH --job-name        hybrid_job
#SBATCH --output          %x.out
#SBATCH --mem-per-cpu     500
#SBATCH --ntasks          2
#SBATCH --cpus-per-task   4

srun bash whothis.sh

[yourUsername@mahuika ~]$ sbatch example_hybrid.sl

Solution

[yourUsername@mahuika ~]$ cat hybrid_job.out

I am task #0 running on node 'wbn016' with 4 CPUs
I am task #1 running on node 'wbn022' with 4 CPUs

Job Array

Job arrays are not “multiproccessing” in the same way as the previous two methods. Ideal for embarrassingly parallel problems, where there are little to no dependencies between the different jobs.

Can be thought of less as running a single job in parallel and more about running multiple serial-jobs simultaneously. Often this will involve running the same process is run on multiple inputs.

Embarrassingly parallel jobs should be able scale without any loss of efficiency. If this type of parallelisation is an option, it will almost certainly be the best choice.

A job array can be specified using --array

If you are writing your own code, then this is something you will probably have to specify yourself.

Job Array Example

Create a new script called example_jobarray.sl

#!/bin/bash -e

#SBATCH --job-name        job_array
#SBATCH --output          %x_%a.out
#SBATCH --mem-per-cpu     500
#SBATCH --array           0-3

bash whothis.sh

then submit with

[yourUsername@mahuika ~]$ sbatch example_jobarray.sl

Solution

ls

job_array_0.out job_array_1.out job_array_2.out job_array_3.out

Each of which should contain,

 [yourUsername@mahuika ~]$ cat job_array_*.out

I am task #0 running on node 'wbn*' with 2 CPUs

How to Utilise Multiple CPUs

Requesting extra resources through Slurm only means that more resources will be available, it does not guarantee your program will be able to make use of them.

Generally speaking, Parallelism is either implicit where the software figures out everything behind the scenes, or explicit where the software requires extra direction from the user.

Scientific Software

The first step when looking to run particular software should always be to read the documentation. On one end of the scale, some software may claim to make use of multiple cores implicitly, but this should be verified as the methods used to determine available resources are not guaranteed to work.

Some software will require you to specify number of cores (e.g. -n 8 or -np 16), or even type of paralellisation (e.g. -dis or -mpi=intelmpi).

Occasionally your input files may require rewriting/regenerating for every new CPU combintation (e.g. domain based parallelism without automatic partitioning).

Writing Code

Occasionally requesting more CPUs in your Slurm job is all that is required and whatever program you are running will automagically take advantage of the additional resources. However, it’s more likely to require some amount of effort on your behalf.

It is important to determine this before you start requesting more resources through Slurm

If you are writing your own code, some programming languages will have functions that can make use of multiple CPUs without requiring you to changes your code. However, unless that function is where the majority of time is spent, this is unlikely to give you the performance you are looking for.

Python: Multiproccessing (not to be confused with threading which is not really parallel.)

MATLAB: Parpool

Key Points

• Parallel programming allows applications to take advantage of parallel hardware; serial code will not ‘just work.’

• There are multiple ways you can run