DISTRIBUTED SYSTEMS

Principles and Paradigms

Second Edition
ANDREW S. TANENBAUM
MAARTEN VAN STEEN

Chapter 7
Consistency And Replication (2)

Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5
 Plan
•  Motivation
•  Consistency models
–  Data-centric consistency
–  Client-centric consistency
•  Replica management
•  Consistency protocols

Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5
 Consistency Protocols
•  Describe algorithm for
achieving a given
consistency model
–  Data-centric
–  Client-Centric

Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5
 Continuous Consistency
•  Bounding deviations
so as to avoid
–  Numerical errors
•  E.g., that deviation of
replicas’ stock quote is
more than 1.00$
–  Staleness errors
•  E.g., that last stock
quote is more than 1
hour old
–  Order errors
•  E.g., that more than 3
updates to stock quote
are outstanding

Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5
 Bounding Numerical Deviation
•  Assume
–  N servers, S1 … SN
–  Single data item x
–  A write, W(x), has a weight, weight(W), that is the amount that x
changes
•  E.g., x = 1; x := x * 3 <- weight is …
•  2
–  weight(w) > 0
•  We have a limit on numerical deviation for the value of x, vi, at
server Si

W(x) E.g., multicast or

W(x)
epidemic protocol
x1 W(x) x2 x3
origin(W)

L1 L2 L3
Log of writes
performed
Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5
 Bounding Numerical Deviation
•  Define

•  I.e., sum of writes executed by Si

originating from Sj
–  Note, T[i,i] are all values submitted by Si

Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5
 Bounding Numerical Deviation
Current value of x at Si
Consistent value of x

•  Define

•  Goal
–  Ensure

Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5
 Bounding Numerical Deviation
•  Assume
–  Sk can learn about TW[i,k]
–  Sk maintains this as a view, TWk[i,k], of what Si has
seen from k
–  We have 0 ≤ TWk[i,k] ≤ TW[i,k] ≤ TW[j,k]
δi
•  If TW [k,k] − TW k [i,k] > then Sk forwards writes
N −1
from its log, Lk
–  Sk waits to commit new writes until these are
€ commited at Si
•  Guarantees that deviation is bounded as
wanted…
Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5
Bounding Numerical Deviation
N N
v(t) − v i = (v(0) + ∑ TW [k,k]) − (v(0) + ∑ TW [i,k])
k=1 k=1
N
= ∑ (TW [k,k] − TW [i,k])
k=1 TWk[i,k] ≤ TW[i,k]
N for all i, and j
≤ ∑ (TW [k,k] − TW k [i,k])
k=1
i−1
= ∑ (TW [k,k] − TW k [i,k]) + (TW [i,i] − TW i [i,i])
k=1
Si’s view of itself is up-to-date
N
+ ∑ (TW [k,k] − TW k [i,k])
k= i+1
TW[k,k]-TWk[i,k] ≤ ∂i/(N-1)
δi for all k
≤ (N −1)
N −1
= δi
Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5
 Bounding Staleness Deviations
•  Goal
–  Ensure that there are no W(x) older than TOi that Si
has not seen
•  ”Older than”?
–  Vector clocks?
–  If we know drift rate of clocks and communication
delay, we can synchronize within a bound

Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5
Bounding Staleness Deviations
•  In any case
–  Keep vector of time stamps
•  VTk[i] = Local time of last write from Si
–  If T(k) – VTk[i] > Limit, pull updates from Si
(and update VTk[i])

Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5
 Bounding Ordering Deviations
•  Check size of local queue of tentative updates
–  Updates may not have been committed due to
ordering constraints
•  If too large, try to agree on ordering of updates
before proceeding…
–  Ensuring consistent ordering

Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5
 Ensuring Consistent Ordering
•  Make a specific process responsible for
specific data item
–  Primary-Based Protocols
•  Enable multiple processes to carry out
write operations
–  Replicated-Write Protocols

Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5
 Remote-Write Protocols

W5

•  Figure 7-20. The principle of a primary-backup protocol.

Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5
 Local-Write Protocols

•  Figure 7-21. Primary-backup protocol in which the primary

migrates to the process wanting to perform an update.
Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5
Replicated-Write: Active Replication

•  Multicast updates to all replicas

–  Sequencer
–  Totally ordered multi-cast

•  May be inefficient

Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5
 Replicated-Write: Quorum
•  Multicast W(x), read x
–  Has the write been committed?
–  Is the value read up-to-date?

•  Static process group

–  Basic idea
•  Write (version(x), value(x))
•  When writing new version, get more than half of the
processes to approve
•  When reading, get response with latest version from more
than half of the processes
–  Avoid write/write conflicts
–  Avoid write/read conflicts
•  Can this be approved?
Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5
 Replicated-Write: Quorum
•  Process group of size N
•  Define
–  write quorum size, NW
•  At least NW processes need to agree to let us write
–  read quorum size, NR
•  At least NR processes need to agree to let us read

•  To make sure to read latest version

–  NW + NW > N No two processes can get
permission to write
–  NW + NR > N
At least one process
will have latest version

Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5
 Quorum-Based Protocols

•  Figure 7-22. Three examples of the voting algorithm. (a)

A correct choice of read and write set. (b) A choice that
may lead to write-write conflicts. (c) A correct choice,
known as ROWA (read one, write all).
Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5
 Client-Centric Consistency
•  Each client keeps track of
–  Read set: identifiers of the writes that are relevant to
the read operations performed by the client
–  Write set: identifiers of writes performed by the client

•  Monotonic-read consistency
–  Read performed
•  Client hands in read set to server
•  Server may need to
communicate with other
server to become up-to-date
•  Server returns updated read set
–  Write performed
•  (Nothing particular)
–  Also works when client moves
Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5
 Client-Centric Consistency
•  Monotonic-write consistency
–  Write
•  Client hands in write set to server
•  Server may need to communicate with other server to become up-to-date
•  Server returns updated write set
•  Read-your-writes consistency
–  Read
•  Client hands in write set to server
•  Server may need to communicate with other server to become up-to-date
•  Server returns updated write set
•  Writes-follow-reads consistency
•  Client hands in read set to server
•  Server may need to communicate
with other server to become
up-to-date with read set
•  Add identifiers from the read set to
the write set
•  Server returns updated write set

Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5
 Client-Centric Consistency
•  Efficiency?
•  Bound size of writes and
reads by grouping operations
into sessions
–  E.g., clear sets when a client
exits a specific application
•  Use a vector timestamp…
–  WVCi[j]: the timestamp of the
latest write operation from Sj that
Si has processed
–  SVCA[j]: the timestamp of the
latest write operation in the
session that originates from Sj
–  Si uses WVCi and SVCA to check
consistency

Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5
 Summary
•  Various protocols for ensuring consistency
•  Data-centric consistency
–  Primary-based protocols
–  Replicated-write
•  Primary-based protocols most widely used due
to simplicity
–  Exact selection of protocol type depends on rate at
which request are processed and on message delay
•  Client-centric consistency may be efficiently
implemented using vector timestamps

Tanenbaum & Van Steen, Distributed Systems: Principles and Paradigms, 2e, (c) 2007 Prentice-Hall, Inc. All rights reserved. 0-13-239227-5