Message Passing

Distributed Computing Environment

• A distributed system consists of a set of processors connected by
a communication network.
• The communication network: Information exchange among
processors. The delay is finite but unpredictable.
• No common global memory among processors.
• Communicate solely by passing messages.
• There is no physical global clock in the system.
• The communication medium may deliver messages out of order,
messages may be lost, garbled, or duplicated due to timeout and
retransmission, processors may fail, and communication links
may go down.
• The system can be modeled as a directed graph in which vertices
represent the processes and edges represent unidirectional
communication channels.
 Distributed Computing Environment
• A distributed application runs as a collection of processes on a
distributed system.

• The system can be modeled as a directed graph in which vertices

represent the processes and edges represent unidirectional
communication channels.
 Distributed Program
• A set of n asynchronous processes
p1, p2,……. , pi, pn
that communicate using messages

• Assume that each process is running on a different processor.

• Let Cij denote the channel from process pi to process pj and let mij
denote a message sent by pi to pj.

• No global clock
• No global memory
• Communication delay finite but unpredictable
• Process execution and message transfer are asynchronous
– A process may execute an action spontaneously
– A process sending a message does not wait for the delivery of the message to
be complete
 Global state of Distributed
Computation
• Represented by states of the processes and the
communication channels.

• The state of a process is characterized by:

– the state of its local memory
– depends upon the context.

• The state of a channel is characterized by:

– the set of messages in transit in the channel.
 Distributed Executions
• The execution of a process consists of a
sequential execution of its actions.
• The actions are atomic .
• Actions of a process are modeled as three types
of events:
– internal events
– message send events
– message receive events.
 Distributed Executions

• For a message m, let send (m) and rec (m) denote its send and receive events.

• The occurrence of events changes:

– The states of respective processes and channels
– Thereby, causing transitions in the global system state.

• An internal event changes the state of the process at which it

occurs (affects only it)

• A send event (or a receive event) changes the state of the:

– process that sends (or receives) the message
– the state of the channel on which the message is sent (or received).
Distributed Executions
Distributed Executions
Distributed Executions
Distributed Executions: Casual precedence
relation
Distributed Executions: Casual Precedence
Distributed Executions
 Physical vs logical concurrency
• Two events are logically concurrent if and only if they do not
causally affect each other.
• Physical concurrency has a connotation that the events occur at
the same instant in physical time
• Two or more events may be logically concurrent even though they
do not occur at the same instant in physical time.
• Example:

• Whether a set of logically concurrent events coincide in the

physical time or in what order in the physical time they occur does
not change the outcome of the computation.
 Models of Communication Network
• Models of the service provided by communication networks :
– FIFO (first-in, first-out) : Each channel acts as a first-in first-out message
queue and thus, message ordering is preserved by a channel

– non-FIFO : A channel acts like a set in which the sender process adds
messages and the receiver process removes messages from it in a random
order.

– Causal ordering: Based on Lamport’s “happens before” relation.

A system that supports the causal ordering model satisfies the following
property

Causal ordering model is useful in developing distributed algorithms.

 Global State
• The global state of a distributed system is a collection of the local
states of the processes and the channels.
Global State
 Global State
• A meaningful global snapshot:
– the states of all the components of the distributed system must be recorded at the
same instant.
– Not possible

• Then atleast must be consistent:

– Effect cannot happen before cause.
– A message cannot be recorded as received if it has not been sent.

– Example
 Global State

• A global state is strongly consistent if it is transitless as well as

consistent
• Example:
 Cuts in a distributed environment
• A zigzag line joining one arbitrary point on each process line is termed a cut in
the computation.

• Such a line slices the space–time diagram

• Set of events in distributed computation are divided into
– PAST and a FUTURE.
– The PAST contains all the events to the left of the cut
– The FUTURE contains all the events to the right of the cut.
– For a cut C, let PAST(C) and FUTURE(C) denote the set of events in the PAST and
FUTURE of C.

• Every cut corresponds to a global state.

• Every global state can be graphically represented as a cut in the computation’s
space–time diagram
 Cuts in a distributed environment

• A consistent global state corresponds : Every message received in the PAST of

the cut was sent in the PAST of that cut. (Consistent Cut)

• All messages that cross the cut from the PAST to the FUTURE are in transit in
the corresponding consistent global state.

• A cut is inconsistent if a message crosses the cut from the FUTURE to the PAST.
• Example:
Cuts in a distributed environment
Past & Future cones
Past & Future cones
 Global state and snapshot
• Global state recordings must be consistent.
• Possible if the snapshots are taken at the same time in all processors.
• Infeasible due to lack of common clock.
• Even in presence of Common clock: Inconsistency will prevail due to delays in
read
 Global state and snapshot

• Condition C1 states the law of conservation of messages.

• Condition C2 states that in the collected global state, for every effect, its cause
must be present

• In a consistent snapshot:
– All the recorded local states of processes are concurrent.
– The recorded local state of no process casually affects the recorded local state of any other process.
 Issues in recording Global state
• I1: How to distinguish between the messages to be recorded in the snapshot
(either in channel state or a process state) from those not to be recorded.

• I2: How to determine the instant when a process takes its snapshot.

• Solutions:
• I1: Any message that is sent by a process before recording its snapshot, must
be recorded in the global snapshot (from C1). Any message that is sent by
a process after recording its snapshot, must not be recorded in the global
snapshot (from C2).

• I2: A process pj must record its snapshot before processing a message mij
that was sent by process pi after recording its snapshot (from C2).
 Snapshot algorithms for FIFO channels
• The Chandy-Lamport algorithm:
– It uses a control message, called a marker.
– After a site has recorded its snapshot, it sends a marker along all of its outgoing
channels before sending out any more messages.

– Addresses Issue-1: Since channels are FIFO, a marker separates the messages in
the channel into those to be included in the snapshot from those not to be
recorded in the snapshot.

– Addresses Issue-2 : Markers in a FIFO system act as delimiters for the messages in
the channels so that the channel state recorded by the process at the receiving end
of the channel satisfies the condition C2.
– Since all messages that follow a marker on channel Cij have been sent by process pi
after pi has taken its snapshot, process pj must record its snapshot no later than
when it receives a marker on channel Cij . In general, a process
 Snapshot algorithms for FIFO channels
• The Chandy-Lamport algorithm:
–
 Snapshot algorithms for FIFO channels
• The Chandy-Lamport algorithm:
• Marker Sending Rule:
– A process initiates snapshot collection by executing the marker sending
rule.
– It records its local state and sends a marker on each outgoing channel.

• Marker Receiving Rule:

– A process executes the marker receiving rule on receiving a marker.

• The algorithm can be initiated by any process by executing the

marker sending rule.
• The algorithm terminates after each process has received a
marker on all of its incoming channels.
 Snapshot algorithms for FIFO channels
• The Chandy-Lamport algorithm:
• Marker Sending Rule:
– A process initiates snapshot collection by executing the marker sending
rule.
– It records its local state and sends a marker on each outgoing channel.

• Marker Receiving Rule:

– A process executes the marker receiving rule on receiving a marker.

• The algorithm can be initiated by any process by executing the

marker sending rule.
• The algorithm terminates after each process has received a
marker on all of its incoming channels.
 Snapshot algorithms for FIFO channels
• The Chandy-Lamport algorithm:
• One policy is to have each process send its local snapshot to the
initiator of the algorithm.

• Another policy is to have each process send the information it

records along all outgoing channels

• Multiple processes can initiate the algorithm concurrently:

– Each initiation needs to be distinguished by using unique markers.
– Different initiations by a process are identified by a sequence number.
 Snapshot algorithms for FIFO channels
• The Chandy-Lamport algorithm: Proving Correctness
• A recorded snapshot must satisfy conditions C1 and C2.
• Since a process records its snapshot when it receives the first marker on any
incoming channel:
– No messages that follow markers on the channels incoming to it are recorded in
the process’s snapshot.
– Due to FIFO property of channels, no message sent after the marker on that
channel is recorded in the channel state.
– Thus, condition C2 is satisfied.

• When a process pj receives message mij that precedes the marker on channel
Cij,:
– if process pj has not taken its snapshot yet, then it includes mij in its recorded
snapshot.
– Otherwise, it records mij in the state of the channel Cij.
– Thus, condition C1 is satisfied.
 Snapshot algorithms for FIFO channels
• The Chandy-Lamport algorithm: Proving Correctness

– The recorded global state actually never occurred in the execution (It may happen)
 Snapshot algorithms for FIFO channels
• The Chandy-Lamport algorithm: Proving Correctness

– The recorded global state actually never occurred in the execution (It may happen).
– Rubber-band criterion
 Snapshot algorithms for non- FIFO channels
• A marker cannot be used to delineate messages into those to be
recorded in the global state from those not to be recorded in the global
state.

• In a non-FIFO system:
– Either some degree of inhibition
– Or piggybacking of control information on computation messages is used

• Example: Message inhibition: When a process receives a marker, it

immediately returns an acknowledgement.
– The sending process starts resending messages again only when it has received
acknowledgement
 Snapshot algorithms for non- FIFO channels
Lai-Yang Algorithm
• Makes use of Marker property to ensure condition C2
• Marker is implemented using coloring scheme here.

• Ensures that no message sent by a process after recording its local snapshot is
processed by the destination process before the destination records its local snapshot
• “marker” is piggybacked on computation messages using a coloring scheme.
Snapshot algorithms for non- FIFO channels
 Snapshot algorithms for non- FIFO channels
• Condition C1 holds :
– A white message mij is included in the snapshot of process pj if pj receives
mij before taking its snapshot. Otherwise, mij is included in the state of
channel Cij.

• Condition C2 holds :
– A red message is not included in the snapshot of the recipient process and
a channel state is the difference of two sets of white messages.

• Space requirements of the algorithm may be large

 Snapshot algorithms for non- FIFO channels
• Li et al’s Algorithm:
– A global snapshot algorithm in a non-FIFO system.

– Similar to the Lai–Yang algorithm.

– Markers are tagged so as to generalize the red/white colors to

accommodate repeated invocations of the algorithm and
multiple initiators.

– It is not concerned with the contents of computation

messages.

– State of a channel is computed as the number of messages in

transit in the channel.
 Snapshot algorithms for non- FIFO channels
• Li et al’s Algorithm:
– A process maintains two counters for each incident channel:
• Record the number of messages sent and received on the channel.
• Reports these counter values with its snapshot to the initiator.

– Channel states are computed using an incremental technique

that reduces the size of message histories to be stored and
transmitted.

– The initiator computes the state of Cij as:

• (the number of messages in Cij in the previous snapshot) +
(the number of messages sent on Cij since the last snapshot at process
pi) −
(the number of messages received on Cij since the last snapshot at
process pj).
 Snapshot algorithms for non- FIFO channels
• Li et al’s Algorithm: Process
– Snapshots initiated by an initiator are assigned a sequence number.

– All messages sent after a local snapshot recording are tagged by a tuple
< init_id, MKNO >,
• where init_id is the initiator’s identifier
• MKNO is the sequence number of the algorithm’s most recent invocation by initiator
init_id;

– To insure liveness, markers with tags similar to the above tags are explicitly
sent only on all outgoing channels on which no messages might be sent.

– The tuple < init_id MKNO > is a generalization of the red/white colors used
in Lai–Yang to accommodate repeated invocations of the algorithm and
multiple initiators.
 Snapshot algorithms for non- FIFO channels
• Li et al’s Algorithm: Process
– Local state recording is done similar to Lai Yang algorithm.
 Snapshot algorithms for non- FIFO channels
• Li et al’s Algorithm: Process
– Channel states are recorded as:
 Snapshot algorithms for non- FIFO channels
• Li et al’s Algorithm: Process

– If the initiator initiates a snapshot before the completion of the previous

snapshot, it is possible that some process may get a message with a lower
sequence number after participating in a snapshot initiated later.

– In this case, the algorithm uses the snapshot with the higher sequence
number to also create the snapshot for the lower sequence number.

– The algorithm works for multiple initiators if :

• Separate input/output counters are associated with each initiator
• Marker messages and the tag fields carry a vector of tuples, with one tuple for each
initiator.

– The local storage and size of tags on computation messages are of size O(n),
where n is the number of initiators
 Snapshot algorithms for casually ordered
channels
• The causal message delivery property CO provides a built-in message
synchronization to control and computation messages.

• Two global snapshot recording algorithms, namely, Acharya-Badrinath and

Alagar-Venkatesan exist that assume that the underlying system supports causal
message delivery.

• In both these algorithms recording of process state is identical and proceed as

follows :
– An initiator process broadcasts a token, denoted as token, to every process
including itself.
– Let the copy of the token received by process pi be denoted tokeni .
– A process pi records its local snapshot LSi when it receives tokeni and sends
the recorded snapshot to the initiator.
– The algorithm terminates when the initiator receives the snapshot recorded
by each process.
Snapshot algorithms for casually ordered
channels
 Snapshot algorithms for casually ordered
channels
Acharya Badrinath Algorithm
- Channel State Recording
 Snapshot algorithms for casually ordered
channels
Acharya Badrinath Algorithm
- Condition C1 and C2
 Snapshot algorithms for casually ordered
channels
Acharya Badrinath Algorithm
- Condition C1 and C2