ARCHITECTURES FOR PARALLEL

COMPUTATION

FK Boachie.
 Why Parallel Computation
 Parallel Programs
 A Classification of Computer Architectures
 Performance of Parallel Architectures
 The Interconnection Network
 Array Processors
 Multiprocessors
 Multicomputers
 Vector Processors
1. The need for high performance!
 Two main factors contribute to high
performance of modern processors:
1. Fast circuit technology
2. Architectural features such as:
◦ large caches, multiple fast buses
◦ Accelerators ( Hardware accelerators – what is
that?) Supports the performance of normal CPUs.
Eg: a 3D graphics accelerator has its own
processor GPU (graphics processing unit) and
RAM
◦ Pipelining, superscalar architectures and VLIW
architectures (multiple funct. Units etc.)
 One solution is the need for high performance
architectures in which several CPUs are running in
order to solve a certain application.

 Such computers have been organized in very

different ways. Some key features:

◦ number and complexity of individual CPUs

◦ availability of common (shared memory)
◦ interconnection topology
◦ performance of interconnection network
◦ I/O devices
 Parallel Programs

Parallel sorting –don’t waste time on this

A possible program for parallel sorting:

var t: array[1..1000] of integer;

- - - - - - - - - - -
procedure sort(i,j:integer);
-sort elements between t[i] and t[j]-end sort;
procedure merge;
- - merge the four sub-arrays - - end merge;

- - - - - - - - - - -
begin
- - - - - - - -
cobegin sort(1,250)| sort(251,500)| sort(501,750)| sort(751,1000)
coend; merge;
- - - - - - - -
end;
2. Matrix addition:

a11 a21  am1 b 11 b 2 1  b m 1 c 1 1 c 2 1  c m 1

a12 a22  am2 b12 b22  bm2 c12 c22  cm2
a13 a23  am3 + b13 b23  bm3 = c13 c23  cm3
  
a1n a2n  amn b 1n b 2 n  b m n c 1 n c 2 n  c m n

var a: array[1..n,1..m] of integer;

b: array[1..n,1..m] of integer;
c: array[1..n,1..m] of integer; i:integer
- - - - - - - - - - -

begin
- - - - - - - -
for i:=1 to n do for j:= 1 to m do
c[i,j]:=a[i,j]+b[i,j]; end for
end for
- - - - - - - -
end;
Matrix addition - parallel version

var a: array[1..n,1..m] of integer;

b: array[1..n,1..m] of integer;
c: array[1..n,1..m] of integer; i:integer
- - - - - - - - - - -
procedure add_vector(n_ln:integer); var j:integer
begin
for j:=1 to m do c[n_ln,j]:=a[n_ln,j]+b[n_ln,j];
end for
end add_vector;

begin
- - - - - - - -
cobegin for i:=1 to n do add_vector(i);
coend;
- - - - - - - -
end;
So what is Parallel computing?

Parallel computing is the use of multiple computers or

processors working together on a common task.

• Each processor works on its section of the problem

• Processors are allowed to exchange information with

other processors
1.Flynn’s classification is based on the nature of the instruction flow
executed by the computer and that of the data flow on which the
instructions operate
2.Single Instruction stream, Single, Data stream (SISD)

CPU
Instr. Stream Processing
Control
Unit Unit

Data. Stream
Memory
2. Single Instruction stream, Multiple Data stream (SIMD)

SIMD with shared memory

DS1
Processing Unit_1

DS2
IS Processing Unit_2
Shared
Control Unit
Memory
DSn
Processing Unit_n
Important questions:
• How fast runs a parallel computer at its maximal potential?
• How fast execution can we expect from a parallel computer for
a concrete application?
• How do we measure the performance of a parallel computer
and the performance improvement we get by using such a
computer?
Have you read about both IBM and Google aiming to
commercialize quantum computers within the next few years?
Intel and Microsoft are also spending billions in their
development. Remember quantum computers work on qubits and
not on bits.
Their main purpose is to perform computations impossible for
any conventional computer – that works on bits. So no matter
how fast a computer is today, there is always some push for
further improvements. Like the need for driverless cars etc.
Single Bus
Remember that the wrong interconnection network of a group
of computer could jeopardize the output of a major project
such as some of the failed North Korea’s attempt for a space
shuttle test in the previous years. Recently as one can see,
they have mastered it and able to launch their missiles.
Completely connected network or meshed network of computers
Crossbar network