Message Passing Interface - An Overview

Ms. Anshu Kumari

Master of Computer Applications
Patna Women's College

Communicating Author: Ms. Anshu Kumari

Email: anshurk1@gmail.com

Abstract
The Message Passing Interface (MPI) is a communication library specification for
parallel computers as well as for the workstation network. This paper signifies
the role of MPI in parallel Computing. To give the general understanding of MPI
Specification, this paper also presents the detailed view of the primary subject
including “Evolution of MPI, Modes of MPI, MPI features, Communicator” etc.
This paper ideal for the learner, new to parallel programming with MPI.

Keyword
Message Passing Interface: MPI, MPI Modes, MPI Functions, Communicator

1.0 MPI
MPI is a portable standard library interface for coding parallel programs
using distributed -memory programming model. MPI is primarily focused
on message -passing in parallel programming model: data is transferred
from one process’s address space to another process’s address space
through cooperative operation on each process [1] . The goal of MPI is to
provide a universal standard for programs based on message passing.

1.1 History and Evaluation

The effort to design an MPI interface began in 1992 with the motivation to
provide a portable API’s. Application written to any of the API’s such as
IBM EUI [119], Intel NX [232], Thinking Machines CMMD [272];
either could not run on different machine or would not run efficiently. The
multiplicity of API’s design to solve this issue was hampering the
application development, thus arises the need of a single interface.

Electronic copy available at: https://ssrn.com/abstract=3467063

The initial version of MPI i.e. (MPI-1) was released in 1994 with the
defined feature of point to point communication, collective communication,
datatypes and nonblocking communication

Features of MPI-1
a. Point-to-point communication,
b. Collective communication,
c. Process groups and communication domains,
d. Virtual process topologies, and
e. Binding for Fortran and C.

The 2nd version of MPI i.e. (MPI-2) was released in 1997. The MPI-2
extend the basic message passing model with one-sided communication,
parallel I/O and dynamic processes.

Key functionality in MPI-2:

a. Dynamic Proceses -It provides routines to create new processes after
job startup.
b. One-Sided Communications -It provides routines for one directional
communication. It also includes shared memory operations (put/get)
and remote accumulate operations.
c. Extended Collective Operations -It allows for the application of
collective operations to inter-communicators
d. External Interfaces – It defines routines, allow developers to layer on
top of MPI, such as for debuggers and profilers.
e. Additional Language Bindings – It describes C++ bindings and other
Fortran-90 issues.
f. Parallel I/O – It define MPI support for parallel I/O.

The MPI-3 was released in 2012, includes nonblocking collectives,

neighborhood collectives, a tool information interface and significant
extensions to the one-sided communication interface.

Electronic copy available at: https://ssrn.com/abstract=3467063

 Features added in MPI-3
• Nonblocking Collective Operations – It permits tasks in a collective to
perform operations without blocking, possibly offering performance
improvements.
• Neighborhood Collectives - It extends the distributed graph and
Cartesian process topologies with additional communication power.
• Fortran 2008 Bindings - It expanded from Fortran90 bindings
• MPIT Tool Interface - It allows the MPI implementation to expose
certain internal variables, counters, and other states to the user[3] .
• Matched Probe - It fixes an old bug in MPI-2 where one could not probe
for messages in a multi-threaded environment.

1.2 Characteristic of MPI

An MPI program is a collection of a fixed number of processes, executing

their own code in their own address space. The process code doesn’t not
need to be identical. IPC requires calling MPI routine in another MPI
process. Thus, these processes communicate via call to MPI communication
primitive or subroutine.

Some of the characteristic of MPI are as follows -

1. It allows programmer to use MPI for writing parallel libraries. These

libraries can be safely used by other programmers in their MPI program
without any knowledge of the details of its implementation.

2. MPI support distributed program execution on heterogeneous

hardware. we may run a program that starts processes on multiple
computer system to work on the same problem.

3. MPI guarantees the reliability of message transmission. It also ensures

that the message sent from a particular task to another task arrive in the
order they were sent so no tedious book-keeping and checking are
required at the receiver end.

Electronic copy available at: https://ssrn.com/abstract=3467063

4. MPI have the concept of communicator. With communicator, it is
possible to isolate communication so that only those task that should
take part in the message -passing can do so.

5. MPI is extensible. With the MPI standard capabilities, it is possible to

achieve debugging, profiling and visualization of parallel program.

6. MPI supports the mapping of the application topologies into virtual

topologies and vice-versa. For example, a cartesian mesh topology is a
virtual topology. MPI provides routine for defining, examining and
manipulating this topology [2].

Communicator
The communication domain is a group of processes communicating with
each other. Information related to the communication domain are store in
MPI_Comm type variable, called “communicator”.
MPI define the default communicator MPI_COMM_WORLD, includes all
MPI processes.
A customized communicator or process filtering are required, if the
programmer needs to send a message within some of these processes

(Source : https://slideplayer.com/slide/5088918/) Communicator

Electronic copy available at: https://ssrn.com/abstract=3467063

1.3 Mode of MPI Connection

1.3.1 Point to point

Point to point communication takes place between pairs of processors. It can
be blocking or non-blocking, depending on the sender being waiting or not
for the receiver having acknowledged the reception of the message.
It works in two steps:
1. The sending process call MPI_Send to express its intent to transfer data
to receiving process.
2. The receiving process call MPI_Recv to express its intent to receive data
from a sending process.

The data transfer is not finished until both function s completed their
operation.

There are basically four communication modes for sending a message

a. Buffered mode: This mode allow user to control the space available
for buffering within a defined buffer model. Send can be initiated
whether or not matching receive has been initiated, also send may
complete before matching receive is initiated . The send primitive is
MPI_Send.

b. Synchronous mode: The sender is blocked until the receiver issues

the corresponding receive. At the expense of synchronization, message
can be transferred without intermediate copies and no buffer is
required .The send primitive is MPI_SSend.

c. Ready mode: It implement a simple protocol where the message is

sent “in hope” and dropped if there is no ready receive but use demand
special care. Send can only be initiated if matching receive has already
been initiated. The send primitive is MPI_RSend.

d. Standard mode: The sender rs blocked until the send buffer can be
reused without altering the message. It may behave like either
buffered mode or synchronous mode, depending on specific

Electronic copy available at: https://ssrn.com/abstract=3467063

 implementation of MPI and availability of memory for buffer space.
The send primitive is MPI_BSend.

A drawback to MPI_Send and MPI_Recv is that they force processes to wait

for other processes. An alternative solution to this issue is to use MPI_Isend
and MPI_Irecv , which performs send/receive operation without halting
either process.

1.3.2 Group communication

Group communication allows to send or receive message simultaneously to
all member of a set of processes. E.g. Broadcasting, scattering. Group
communication make use of barrier to introduce synchronization among
group members. Barrier synchronization block processes or makes a process
waiting for others having completed a task.
MPI provides a range of functions to implement the group communication
or collective message passing.Some of them are as follows

a. MPI_Bcast(msgaddr, count, datatype, rank, comm): This function is

used by a process ranked rank in group comm to broadcast the message to all
the members (including self) in the group.

b. MPI_Allreduce

c. MPI_Scatter(Sendaddr, Scount, Sdatatype, Receiveaddr, Rcount,

Rdatatype, Rank, Comm): Parallel Algorithms & Parallel Programming
Using this function process with rank rank in group comm sends personalized
message to all the processes (including self) and sorted message (according
to the rank of sending processes) are stored in the send buffer of the
process[5]. Sendaddr, Scount, Sdatatype parameters define buffer of sending
process and next three define buffer of receiving process.

d. MPI_Gather (Sendaddr, Scount, Sdatatype, Receiveaddr, Rcount,

Rdatatype,Rank, Comm): With MPI_Gather() process with rank rank in
group comm receives personalized message from all the processes (including
self) and sorted message (according to the rank of sending processes) are
stored in the receive buffer of the process[5].

Electronic copy available at: https://ssrn.com/abstract=3467063

 e. MPI_Alltoall(): Each process sends a personalized message to every other
process in the group.

f. MPI_Reduce (Sendaddr , Receiveaddr , count, datatype, op, rank,

comm): This function reduces the partial values stored in Sendaddr of each
process into a final result and stores it in Receiveaddr of the process with
rank rank. op specifies the reduction operator.

g. MPI_Scan (Sendaddr, Receiveaddr , count, datatype, op, comm): It

combines the partial values into p final results which are received into the
Receiveaddr of all p processes in the group comm.

h. MPI_Barrier(cmm): This function synchronizes all processes in the group

comm.

2.0 Functioning of MPI

The essential subroutine of MPI are define in mpi.h (C) and mpif.h (Fortran). Some
of the primary subroutine are as follows:

I. mpi_init()
It initiates an MPI based program, or computation. It initializes the library,
builds an MPI environment and creates an MPI jobs to store the information
about the process involved.It is defined as:

mpi_init(int *argc , char *argv)

Where argc – pointer to the number of arguments

argv – pointer to the array containing the arguments

II. mpi_comm_rank()
Each process in an MPI job is given a unique rank number, helps to
communicate with other processes. The mpi_comm_rank() determines the rank
of defined MPI process. It is defined as :

int mpi_comm_rank(mpi_comm comm, int *rank)

Electronic copy available at: https://ssrn.com/abstract=3467063

 Where comm – communicator
size – size of the processes in the communicator group
rank – process id in the communicator group

III. mpi_comm_size()
It is used to determine the total number of processes to be lunched and
monitored by the MPI program.It is defined as :

int mpi_comm_size(mpi_comm comm, int *size)

Where , comm – communicator, an Input parameter

rank – process id in the communicator group, an output
parameter

IV. mpi_send()
This function is used to send data to another process in the communicator
group. It is defined as :

int mpi_ send(void *buffer ,int count ,mpi_datatype datatype,int dest,

int tag,mpi_comm comm)

where buffer- pointer to the initial address of send buffer

count- numbers of elements to be send
datatype -datatype of each sendbuffer element
dest- process id of destination process
tag-message tag
comm-communicator

V. mpi_recv()
This function is used to receive data from another process in the
communicator group.It is defined as :

int mpi recv(void *buffer,int count ,mpi_datatype datatype,int

source, int tag,mpi_comm comm, MPI_Status *status)

where buffer- address where the data that is communicated is received

count- maximum number of items that can be received
datatype - descriptor for the data type of each item

Electronic copy available at: https://ssrn.com/abstract=3467063

 source- the rank of the source process, relative to the communicator
given in comm
tag-message tag that can be used to distinguish between multiple
messages that are sent to the process
comm- communicator that describes the group of processes involved
in the communication, of which the sending receiving processes are
part
status - an object that holds information about the received message

VI. mpi_barrier()
This subroutine is used to synchronize the execution of a group of
process specified within the communicator. when a process reaches
this operation, it has to wait until all other processes have reached
the mpi_barrier(). i.e. no process returns from mpi_barrier until all
the processes have called it.

mpi_barrier is a global operation that invokes all processes. A barrier

is a way of separation two phase of computation to ensure that
message generated in different phase do not interfere. It is defined as:

int mpi_ barrier (mpi_comm comm)

where comm define the communicator specify the group of

processes that are synchronized. A mpi_barrier call returns only if all
the processes in the group have called this function.

VII. mpi_finalize()
It is used to terminate an MPI process or shuts down the current
running task and inform the MPI library about the task completion.
A non-call to mpi_finalize() at the end of each task ,leaves network
communication pipes open causing system errors.It is defined as:

int mpi_finalize()

Electronic copy available at: https://ssrn.com/abstract=3467063

Example

#include < mpi.h>

#include < stdio.h>

int main(int argc, char **argv)

{

int myRank;
int partner;
int size, i,t;
char greeting[100];
MPI_Status stat;

MPI_Init(&argc, &argv); /*Start MPI */

MPI_Comm_rank(MPI_COMM_WORLD, &myRank);
/*Determine Rank Of This processor*/
MPI_Comm_size(MPI_COMM_WORLD, &size);
/*Determine Total Number Of Processors*/

sprintf(greeting, "Hello Everyone: processor %d of %d\n",

myRank, size);

if (myRank ==0)
{
fputs(greeting, stdout);
for (partner = 1; partner < size; partner++)
{

MPI_Recv(greeting, sizeof(greeting), MPI_BYTE,

partner, 1, MPI_COMM_WORLD, &stat);
fputs (greeting, stdout);

}
}
else
{
MPI_Send(greeting, strlen(greeting)+1, MPI_BYTE,
0,1,MPI_COMM_WORLD);
}

Electronic copy available at: https://ssrn.com/abstract=3467063

 if (myRank == 0)
printf("Task Completed\n");
MPI_Finalize(); /* Exit MPI */

Output

Hello Everyone: processor 0 of 4

Hello Everyone: processor 1 of 4
Hello Everyone: processor 2 of 4
Hello Everyone: processor 3 of 4
Task Completed

3.0 Limitation of MPI

1. MPI requires more programming changes to move from serial to
parallel version.
2. It can be harder to debug.
3. Performance is limited by the communication network between the
nodes.
4. MPI should not be used if,
a. One can achieve sufficient performance and portability using a
data parallel or shared-memory approach.
b. One can use a pre-exiting library of parallel routines.
c. Parallelism does not require at all.

4.0 Future Scope

MPI is widely used in large scale parallel applications in science and
engineering[4]. Most common MPI application area as follows,

a. Atmosphere, Earth, Environment

b. Physics - applied, nuclear, particle, condensed matter, high pressure,
fusion, photonics
c. Bioscience, Biotechnology, Genetics
d. Chemistry, Molecular Sciences
e. Geology, Seismology
f. Mechanical Engineering - from prosthetics to spacecraft
g. Electrical Engineering, Circuit Design, Microelectronics
h. Computer Science, Mathematics

Electronic copy available at: https://ssrn.com/abstract=3467063

Conclusion
Parallelism is critical and achieving performance improvement with the modern
hardware is crucial. MPI is a standard for writing message passing program. It
gives user explicit control on data management. Currently, MPI is widely used in
clusters and other message passing system due to its rich functionality. Thus, we
conclude that, MPI provides a sophisticated, portable and scalable parallel
programming environment.

References
[1].https://books.google.co.in/books?id=jQrpBQAAQBAJ&pg=PA45&dq=%22from+the+addres
s+space+of+one+process+to+the+address+space+of+another%22&hl=en&sa=X&ved=0ahUKEwj
23YXCzI_lAhUZcCsKHTcaBusQ6AEIKTAA#v=onepage&q=%22from%20the%20address%20space
%20of%20one%20process%20to%20the%20address%20space%20of%20another%22&f=false
[2]. https://books.google.co.in/books?id=o58GiMl-
_vkC&printsec=frontcover&dq=advanced+computer+architecture+,+RAJIV+CHOPRA&hl=e
n&sa=X&ved=0ahUKEwjGnr21wY3lAhVGfH0KHb3XDmgQ6AEIKTAA#v=snippet&q=M
PI&f=false
[3]. https://computing.llnl.gov/tutorials/dataheroes/mpi/
[4]. www.mcs.anl.gov › ~Balaji › permalinks › 2014-06-06-argonne-mpi-basic
[5]. https://issuu.com/imsf/docs/mcse-011

Electronic copy available at: https://ssrn.com/abstract=3467063