PARALLEL AND DISTRIBUTED

COMPUTING
LECTURE#12
 Parallel programming models
• Parallel programming models exist as an abstraction above hardware
and memory architecture.
• Four types of programming models:
1. Shared memory model
2. Message passing model
3. Thread model
4. Data parallel model
1. Shared memory model
• In this programming model, processes or task share a common
address space, which they read and write to asynchronously.
• Various mechanism such as lock or semaphores are used to control
access to the share memory, resolve contentions and to prevent race
conditions and deadlocks.
• Data may be cache on the processors that works on it.
• Compiler translate user variables into “global” memory addresses.
• Simplest parallel programming model.
Shared memory model cont.…
Shared memory model cont.…
FOR EXAMPLE:
• The programmer views his program as collection of processes which use
common or share variables.
• The processor may not have a private program or data memory. A common
program and data are stored in the main memory. This is access by all
processors.
• Each processor is assigned a different part of the program and the data. The
main program creates separate process for each processor.
• The process is allocated to the processor along with the required data.
These process are execute independently on different processors.
Shared memory model cont.…
• After the execution, all the processors have to rejoin the main
program.
Shared memory model cont.…
Implementations
• Native compilers translates user program variables into addresses,
which are memory “global”.
Advantages
• Program development becomes simple. As all processes see and have
equal access to share memory.
• Data “ownership” is lacking, so there is no need to explicitly specify
the communication of data between process.
Shared memory model cont.…
Disadvantages
• It becomes more difficult to understand and manage data locality.
User will not understand where his variables use stored.
2. Message passing model
The message passing model (or distributed memory model) is defined
as:
• A set of processes or tasks use their own local memory
• Processes or task communicate by sending and receiving messages
• Data transfer requires cooperative operations to be performed by
each process : send() matched by received()
Programming with message passing is done by linking with and making
calls to libraries which manage the data exchange between processors.
Message passing model cont.…
Message passing model cont.…
Implementation
• The MPI (Message Passing Interface) is a message passing library standard.
Advantages
• The data can be stored anywhere
• Communication between processes is simple
• Many manage passing interface (MPIs) are available
Disadvantage
• Programmers should take care that there is a receive function for send function.
• If any of the process, quite, others also will stop working, as they are defendant.
3. Threads Model
• This programming model is a type of shared memory programming.
• In the threads model of parallel programming, a single process can
have multiple concurrent execution paths (called threads).
• Each threads has local data, but they also share common data
• The thread communicate through global memory. This require
synchronization construct to ensure that more than one thread is not
updating the same global address any time.
• Example: a subroutine within the main program (a.out). Any thread
can execute any subroutine at same time as other threads.
Threads Model cont..
Threads Model cont..
Different threads can be executed on same processor or on different
processor.
• If the threads are executed on same processor, then the processor
switches between the threads in a random fashion.
• If the thread are executed on different processors, they are executed
simultaneously.
Threads Model cont..
Threads Model cont..
Implementation
• OpenMP
Advantages
• Programmer need not have to worry about parallel processing.
Disadvantages
• Care must be taken that no two threads update the shared resources
simultaneously.
4. Data Parallel Model
• Focuses on parallel operations on a set (array) of data items i.e. Data
set. The data set is typically organized into a common structure, such
as an array.
• A set of task work collectively on the same data structure, however,
each task works on a different partition of the same data structure.
• Tasks perform the same operation on their partition of work.
• Suppose the task is to add two arrays of 100 elements store the result
in another array. If there are four processor then each processor can
do 25 additions.
Data Parallel Model cont..
THANKS