Distributed Systems

Theoretical Foundations ♦ Opportunities

Performance through Parallelism
for

Resource Sharing (Cost Effectiveness)

Distributed Systems


Reliability and Availability

− Avoid single points of failure


Expandability
♦ Limitations
Manas Saksena


No Global Clock


University of Pittsburgh No Shared Memory

University of Pittsburgh Manas Saksena 1 University of Pittsburgh Manas Saksena 2

Distributed Algorithms Absence of a Global Clock

♦ Data may be scattered on different machines ♦ On a single machine


No global memory A clock can be used to determine ordering of

♦ Decisions based on local information events in concurrent processes


but, information can be exchanged ♦ Example Use



♦ Avoid Single Point of Failure make program



to improve reliability, availability ♦ Problem Scenario make

10 11 12 13
♦ Lack of a precise global time source P1

no common clock Foo.o created Foo.c modified

8 9 10 11
P2

University of Pittsburgh Manas Saksena 3 University of Pittsburgh Manas Saksena 4

Absence of Shared Memory Lamport’s Logical Clocks

♦ How much money is ♦ Clock
there in the two 

ordering by time (happened before)

$500 $200
accounts together? 

P1:A P2:B 

time interval
X = Query(P1:A) 

Y = Query(P2:B) absolute time

♦ Ordering by time


$50 Amount = X + Y
$450 $200
♦ What happens when


a counter is sufficient
P1:A P2:B money is in transit?
♦ Time Interval


clock must tick at the same rate as real-time

$450 $250
♦ Absolute Time
P1:A P2:B 

clock must be “in-sync” with absolute time

University of Pittsburgh Manas Saksena 5 University of Pittsburgh Manas Saksena 6

1
 Logical Clocks Logical Clocks
♦ Happened Before Relation → ♦ Associate a Clock Ci with each Process Pi
a → b, if a and b are in the same process, and a ♦ Clock


occurred before b 

associated a timestamp to each event

a → b, a is send event, b is corresponding receive


event ♦ How to implement clock to realize the



a → b and b → c then a → c (transitive)

happened before relation?
♦ Causally Related ♦ Two Conditions:


Ci(a) < Ci (b) if a occurs before b in Pi

Event a causally effects Event b if a → b


− IR1: Ci := Ci + d (d > 0)
♦ Concurrent 

Pi /a to Pj/b: tm = Ci(a)
if not (a → b) and not (b → a)


− IR2: Cj := max(Cj, tm + d) (d > 0)

University of Pittsburgh Manas Saksena 7 University of Pittsburgh Manas Saksena 8

Logical Clocks Vector Clocks

♦ Total Ordering from Lamport’s Clocks ♦ Clock:


Use process id to break ties Vector of length N

a ⇒ b, if
 !

Ci[i] = logical clock of Pi

− Ci(a) < Ci (b), or
"

Ci[j] = time of occurrence of last event at Pj which

− Ci(a) < Ci (b) and Pi < P j happened before the current time in Pi
♦ Limitation ♦ Implementation Rules:
#

if a → b then C(a) < C(b), but IR1: Ci[i] = Ci[i] + d (on local events)


− the reverse is not true

$

IR2: Ci[k] = max(Ci[k]+tm[k])

♦ How do we implement clocks for which this is
true?

University of Pittsburgh Manas Saksena 9 University of Pittsburgh Manas Saksena 10

Vector Clocks: Example Vector Clocks

♦ It is possible to compare clocks to see if
%

P1: 0,0,0 1,0,0 2,0,0 they are equal or not

− all are equal (one is not)
&

less than or equal (partial order)

P2: 0,0,0 1,1,0 2,2,1
− all are less than or equal
'

less than
P3: 0,0,0 2,0,1 − less than or equal and not equal
(

not less than

− not less than or equal and not equal
)

concurrent
− both not less than
University of Pittsburgh Manas Saksena 11 University of Pittsburgh Manas Saksena 12

2
 Vector Clocks Causal Message Ordering
♦ Causally Related ♦ What?
If send(m1) → send(m2) then
*

if ta < tb or tb < ta, otherwise concurrent

+

♦ a → b iff ta < tb − receive(m1) → receive(m2)

♦ Why?
,

Example: Replicas
− ensures replica consistency
send(m1)

send(m2)

University of Pittsburgh Manas Saksena 13 University of Pittsburgh Manas Saksena 14

Causal Broadcast Causal Ordering

♦ Message can be only delivered when all ♦ Message Reception: Delivered if
causally earlier messages have been
/

if message received from Pi then local sequence

delivered number for Pi must be 1 less than what’s in the
message
♦ Use Vector Clocks with Messages
-

− implies message is in sequence

Each process assigns a sequence number to the 0

messages it sends For all other processors, the local sequence

.

number must be no less than what’s in the

When a message is sent, the current vector at the
message
sending node is sent as the timestamp
− sender has not seen any message that the
receiver has missed
♦ Else: buffer for later delivery

University of Pittsburgh Manas Saksena 15 University of Pittsburgh Manas Saksena 16

Distributed Algorithms Model Distributed Algorithms Model

♦ A Set of Processes (or Processors) ♦ Communication
♦ Each process has local memory
4

One-Way Message Passing (no shared memory)

5

State = values of variables Point-to-Point Channels

6

♦ Each process can have local clock(s) Multicast Channels

2

− single sender, multiple receivers

Logical Clocks 7

− e.g., Timestamps, Counters, Sequence Nums Broadcast Channels

− No relationship to absolute time ♦ Channel Properties
8

Physical Clock (approximation of absolute time) Reliable

– clock synchronization − No Losses, No Duplicates, No errors, FIFO
– real-time systems
9

Unreliable

University of Pittsburgh Manas Saksena 17 University of Pittsburgh Manas Saksena 18

3
 Distributed Algorithms Model Distributed Algorithms Model
♦ Failures ♦ Lack of Precisely Synchronized Time
: =

Channels In some cases, assume synchronized within some

− Lost/Duplicate messages tolerance
− Message Errors − useful for real-time systems
− Crash (no messages after failure) ♦ Progress of Processes
; >

Processors Assume progress, but no assumptions on speed

− Crash ♦ Asynchronous
− Byzantine ♦ Causal
♦ Non-Instantaneous Communication ?

causal ordering within a process

<

What you see depends on where you are!!! @

causal ordering between message send & receive

University of Pittsburgh Manas Saksena 19 University of Pittsburgh Manas Saksena 20

Consistent Global States Global Consistent States

♦ Process State: ♦ A global state is inconsistent if
A E

encapsulates messages sent/received there is a message m, from Pi to Pj, such that

− sent(m) is not part of local state of Pi, but
B

all messages sent/received prior to recording state

are part of the state − recv(m) is part of local state of Pj
♦ Channel State: ♦ Recording Channel States
C

Suppose we have recorded states for Pi and Pj

F

D
done by receiver side of channel
Then, the channel state (for a channel from Pi to
G

Sender side sends a marker after recording its

Pj) should be: own state
− all messages m, such that sent(m) is in the H

Receiver side records channel state upon

recorded state of Pi, and recv(m) is not in receiving the marker
recorded state of Pj

University of Pittsburgh Manas Saksena 21 University of Pittsburgh Manas Saksena 22

Recording Channel State Example: System Model

♦ On receipt of marker on receiver side
♦ Case 1: if receiver state is not recorded C
P1 P1
I

record receiver state

J

mark channel state as empty C’

− all recorded sends on the sender side have
already been received by the receiver side
– NOTE: Assumes FIFO delivery
♦ Two Processes
L

P1 & P1
♦ Case 2: if receiver state is already recorded
K
♦ Two One-way Point-to-Point Channels
mark channel state as all messages received from M

c & c’
the time it recorded its own state until the receipt
of the marker
University of Pittsburgh Manas Saksena 23 University of Pittsburgh Manas Saksena 24

4
 Example: Possible System
Example: Process Behavior
Evolution
♦ Behavior of each process is represented using a finite state machine Event P1 P2 C C’
♦ Process States are represented abstractly (in this example) Initial A C - -
♦ Channel states are the messages in transit on the channel
e1: Send M B C M -
event state e2: Send M’ B D M M’
Send msg M on C Send msg M’ on C’ e3: Recv M’ A D M -
A B C D
♦ Three events: e1, e2, e3
N

recv msg M’ on C’ recv msg M on C assume that they occurred in this order
♦ Each row is a global state of the system
transition ♦ The table shows the evolution of the system
Process P1 Process P2 ♦ Note: we know that e2 must occur before e3 (e2→e3)

University of Pittsburgh Manas Saksena 25 University of Pittsburgh Manas Saksena 26

Example
Process P1 Process P2
♦ Record State (as A) ♦ Send M’ (e2)
♦ Send Marker on C ♦ Receive Marker on C
♦ Send M (e1) ♦ Record own state (as D)
♦ Receive M’ (e3) ♦ Record C state (as -)
♦ Receive Marker on C’ ♦ Send Marker on C’
♦ Record C’ state (as M’)
♦ Final Recorded State: P1 (A), P2 (D), C (-), C’ (M’)
♦ Note: this corresponds to event sequence: e2, e1, e3
− possible & consistent sequence, but may not have happened

University of Pittsburgh Manas Saksena 27