Parallel Processing (CS417)

Lecture 2

by
Nazia Shahzadi
nazia.shahzadi@giki.edu.pk
Lecturer (FCSE)
Ghulam Ishaq Khan Institute,
Topi, KPK, Pakistan
 Levels of Parallelism
• Depending upon the problem under consideration, parallelism in the solution of
the problem may be achieved at different levels and in different ways.
• Parallelism is a problem and its possible solutions may be exploited either
manually by the programmer or through automating compilers.
• We can have parallelism at four different levels:
1. Instruction level
2. Loop level
3. Procedure level
4. Program level
 Instruction Level Parallelism (ILP)
• It refers to the situation where different instructions of a program are executed by different processing
elements.
• Most of the processors have several execution units and can execute several instructions at the same
time. Some compilers can reorder the instructions to maximize the throughput.
for i=1, 100
Read V(i)

for i=1, 100

Read V(i) for i=91, 100
for i=1, 10 for i=11, 20 X(i) = V(i)*a
For i=1, 100 X(i) = V(i)*a X(i) = V(i)*a Y(i) = V(i)+b
X(i) = V(i)*a Y(i) = V(i)+b Y(i) = V(i)+b
Y(i) = V(i)+b ……… for i = 91, 100
for i = 1, 10 for i = 11, 20 Print X(i)
for I = 1, 100 Print X(i) Print X(i) Print Y(i)
Print X(i) Print Y(i) Print Y(i)
Print Y(i)
 Loop Level Parallelism (LLP)
• At this level, consecutive loop iterations are the candidates for parallel execution.
• For Example:
for(i=0; i<=n; i++)
A(i)=B(i)+C(i)
• Each of the instruction A(i)=B(i)+C(i) can be executed by different processing
elements provided there are at least n processing elements.
• However, data dependencies between subsequent iterations may restrict parallel
execution of instructions at loop level.
• For Example:
for(j=0; j<=n; j++)
A(j)=A(j-1)+B(j)
• Cannot be executed in parallel, as A(j) is data dependent on A(j-1).
 Procedure Level Parallelism (PLP)
• Here, the parallelism is available in the form of procedures. It also requires the good
programming skills to design such type of parallelism.
• For Example:
A procedure performs an operation on vector (V) multiply it by value (a) and sum it with
value (b)
Read vector (V)
Read V Multiply (V) * (a) = X
Sum (V) + b = Y
Print X, Y
V*a = X V+b = Y

Print X Print Y
Flynn’s Taxonomy
 Flynn’s Taxonomy
• It is based on the instruction stream and data stream in a computer system.
• SISD, SIMD, MISD, and MIMD
• M.J. Flynn offered a classification for a computer system’s organization based
on the number of instructions as well as data items.
• There are two types of a stream of information flow in a computer system:
• Instruction and data
• The instruction stream is defined as the sequence of instructions executed by
the processing data
• The data stream is defined as the sequence of data including inputs or the
results that are collected by instruction stream.
 Types of Flynn’s Taxonomy
• According to Flynn’s classification, either of the instruction streams can be single or multiple.
• The architecture of the computer can be classified into four categories:
1. SISD or Single Instruction stream-Single Data stream.
2. SIMD or Single Instruction stream-Multiple Data stream.
3. MISD or Multiple Instruction stream-Single Data stream.
4. MIMD or Multiple Instruction stream-Multiple Data stream.
 SISD
• In a SISD architecture, there is a single processor that executes a single instruction
stream and operates on a single data stream. This is the simplest type of computer
architecture and is used in most traditional computers.
• The computers based on Von Neumann Architecture, are classified as SISD
systems.

IS

IS DS
CU PU MM
 SIMD
• In a SIMD architecture, there is a single processor that executes the same
instruction on multiple data streams in parallel. This type of architecture is used in
applications.
• SIMD computer has a single control unit that issues one instruction at a time but it
has multiple ALUs or processing units to carry out on multiple data sets
simultaneously.
• Well suited for computing that involves lots of vector and matrix multiplication.

DS1
PU1 MM1

IS DS2
CU PU2 MM2

DSn
PUn MMn
IS
Example of SIMD
 MISD
• An MISD computing is a multiprocessor machine capable of executing different
instructions on processing elements but all of them operate on the same data
set.
• Multiple Instruction: Each processing unit operates on the data independently
via separate instruction streams.
• Single Data: A single data stream is fed into multiple processing units.
Example of MISD
 MIMD
• Its organization refers to a computer system capable of processing several
programs at the same time.
• Multiple Instruction: Every processor may be executing a different instruction
stream
• Multiple Data: Every processor may be working with a different data stream
Example of MIMD