JOURNALOF PARALLEL AND DISTRIBUTEDCOMPUTING 9,77-82 ( 1990)
Two Algorithms for Mutual Exclusion in Real-Time
Distributed Computer Systems
ANDRZEJ GOSCINSKI
Department ofcomputer Science, University College, The University ofNew South Wales,
Australian Defence Force Academy, Canberra, Australian Capital Territory, Australia
Two algorithms developed utilizing a priority-based event-or-
dering which manage mutual exclusion in distributed systems-
computer networks-are proposed. in these systems, processes
communicate only by messages and do not share memory. The
computer network functions either (a) in an environment re-
quiring priorities or (b) in a real-time environment. The algo-
rithms are based on broadcast requests and token passing service
approach, but the token need not be passed if no process wishes
to enter the critical section. These algorithms are fully distrih-
uted and are insensitive to the relative speeds of node computers
and communication links. They use only N messages per critical
section, where N is the number of nodes (processes). The algo-
rithms are optimal in the sense that a symmetrical, distributed
algorithm cannot use fewer messages if requests are processed
by each node computer concurrently. Both algorithms ensure
freedom from starvation. There are mechanisms to handle node
insertion and removal, node failure, the loss of the token, the
existence of more than one token, and delivery of messages out
of order. 0 1990 Academic Press, Inc.
1. INTRODUCTION
During the Second European SIGOPS Workshop
“Making Distributed Systems Work” the problem that
there seem to be very few real distributed applications was
again raised [9]. H. Kopetz remarked that real-time sys-
tems, where distribution is dictated by the application, and
which exhibit natural concurrency, are ignored. Indeed,
real-time systems are real concurrent systems, which gener-
ate new problems for the design of distributed operating sys-
tems. One of these is the synchronization of processes.
Some interesting distributed synchronization algorithms
[ 1,6, 11, 12 ] have been proposed during the last couple of
years. The most important features of these algorithms are
that all events are ordered on the basis of the times when
these events happened, and that critical sections are granted
in a first-come-first-served manner. Because of many prob-
77
lems associated with time stamps event-ordering algorithms
(generated by the lack of a common clock), it is suggested
that events be ordered on the basis of priorities of the pro-
cessesrather than on times when they happen. For real-time
systems and systems using priorities for scheduling, a prior-
ity-based approach is more natural and improves the per-
formance of the total system.
Two algorithms developed utilizing a priority-based
event-ordering which handle mutual exclusion in computer
networks are proposed. In these systems, processes commu-
nicate only by messages and do not share memory. The
computer network functions either (a) in an environment
requiring priorities or (b) in a real-time (also called hard
real-time) environment (where processes must meet their
deadlines). The following assumptions about the commu-
nication network were made: ( 1) it is a homogeneous or
heterogeneous network, and (2) it is an error-free network;
i.e., messages are not lost and delivered in the order sent.
However, some extensions to the proposed algorithms to
handle (i) the loss of token, (ii) the existence of more than
one token, and (iii) delivery of messages out of order are
also considered. In the most representative papers on algo-
rithms for mutual exclusion in computer networks pre-
sented by Ricart and Agrawala [ 111 and Suzuki and Kasami
[ 121 there is an assumption that transmission times may
vary and the communication delay is unpredictable. In this
paper we show that it is possible to develop an algorithm for
the case with the upper boundary of transmission delay.
The algorithms are based on (i) requests broadcasted to
all the other processes in the system by a process that wishes
to enter the critical section, and (ii) the token passing ser-
vice approach, but the token need not be passed if no pro-
cess wishes to enter the critical section. Possession of the
token entitles its holder to enter the critical section. These
algorithms are fully distributed and are insensitive to the
relative speeds of node computers and communication
links. They use only N message exchanges for one mutual
exclusion invocation, where N is the number of nodes
0743-73 15/90 $3.00
Copyright 0 1990 by Academic Press, Inc.
All rights of reproduction in any form reserved.
78 ANDRZEJ
(processes). The algorithms are optimal in the sense that a
symmetrical, distributed algorithm cannot use fewer mes-
sages if requests are processed by each node computer con-
currently and a network is error-free. It ensures that each
process requesting entry to its critical section will receive
the token within a finite time of requesting entry.
2. THE ALGORITHMS
Two algorithms developed utilizing a priority-based
event-ordering sharing the same concept were developed.
One is suitable for the environment requiring priorities, and
the other can be used in real-time environment. In the sys-
tem, each node of the computer network executes an identi-
cal algorithm, appropriate to the environment. The differ-
ences between these two algorithms are implied by the fea-
tures of environments as well as methods used to ensure
freedom from starvation. The general description of these
two algorithms is common; however, their important fea-
tures and specific differences will be pointed out. In the pre-
sentation of these algorithms we use the notation based on
an abstract definition of a queue [ 5 1. The following opera-
tions are used: Head-returns the head of a queue; De-
lete-deletes the head of a queue; IsNew-tests whether a
queue is empty; Append-concatenates two queues;
Merge-merges two priority queues; Add-inserts an item
to a queue. Due to this notation, the presentation is short
and facilitates the easy identification of the most important
features of both algorithms.
2.1. Basic Concepts
There are N processes located in N nodes of the network
indexed from 1 to N. Each process i is characterized: in the
case of a system with priorities (called here the P-system)
by its priority p(i), or in the case of a real-time system
(called here the RT-system) by remaining time T(i) to run
the process.
In the P-system, associated with the token is a priority
queueP=(i,,&,. . . , i,), where ij indicates a process which
wants to enter its critical section, i, , i2, . . . , i, is a permuta-
tion of the sequence of numbers 1,2, . . . , m, such that m
< N, and defined by relation ) in the following way: ik ) i,
ifandonlyifforeachk<t,p(&)>p(&).
In the RT-system, associated with the token is a priority
queueP=((i,, T(i,)),(&, T(&)),. . . ,(i,, T(i,))),where
ij indicates a process which wants to enter its critical section,
i, , i2, . . . , i, is a permutation of a sequence of numbers 1,
2 . . >m, such that m =GN, and defined by relation )) in
the following way: i,+)> i, if and only if for each k < t their
remaining times satisfy the condition T( ik) < T( it).
For both cases, if two processes have the same priority or
remaining times, the process with the smaller value of index
i is placed first.
GOSCINSKI
When a process i wants to enter its critical section it sends
the message request( i, p( i))-in the case of a P-system-
or request( i, T( i))-in the case of the RT-system-to all
other processes in the system. It is kept waiting until it re-
ceives the token. When the process j holding the token is
in its critical section, other processes wanting to enter their
critical sections can send request messages. An incoming
priority request queue Q is formed in the node computer
running process j. All processes in this queue are ordered
according to relation ), or relation ) ) (defined above). If
two processes have the same value of priority, the process
with a smaller value of index i is placed first.
A process j holding the token, after exiting the critical
section, examines the incoming request queue Q and the
queue P received with the token, if any. In the case of the
RT-system, the clock routine must perform decrementing
operations on appropriate fields of the queue P and request-
ing queue Q elements, i.e., T( i), to maintain real-time con-
sistency.
-If the queue P is empty and the queue Q is empty, the
process continues with its normal execution until it receives
a request message from some other process.
-If the queue P is not empty and the queue Q is empty,
the process j deletes (using function Delete) the process
with the highest priority (the greatest item), Delete(P) .
-If the queue P is empty and the queue Q is not empty,
the process j deletes (using function Delete) from the prior-
ity queue Q the process with the highest priority (the great-
est item), Delete(Q), and creates the new queue P by
appending (function Append)-P-system or merging
(function Merge)-RT-system the queue P and the
queue Q.
-If the queue P is not empty and the queue Q is not
empty, the construction of the queue P is different for both
cases.
For a P-system, the process j creates a new queue P by ap-
pending (function Append) the old queue P and the queue
Q. Processes from the second part (from Q) and each subse-
quent part (appended, if necessary, in the same way as the
first part) cannot be served while processes from the preced-
ing part are still in the queue. This requirement must be
fulfilled to avoid starvation of processes with smaller values
of priorities.
For a RT-system, the process j constructs (function
Merge) the new queue P by merging the old queue P and
the requesting queue Q .
The process finally sends the token with the new queue P
to the first process recorded in the queue P, Head(P) .
2.2. The Basic Algorithm
Figure 1 shows the algorithm, referred to as Algorithm,
for an error-free network for the case of a system with prior-
 TWO ALGORITHMS FOR MUTUAL EXCLUSION 79

const I : Integer; {the identifier of a given node}

var i, j : Integer;
P, 0 : queue of integers;
IsNew( IsNew : Boolean; {to test whether a queue is empty}
Requesting, HaveToken : Boolean;

procedure wants-to-enter;
begin
Requesting : = true;
if not HaveToken
then begin
foralljin{1,2,...,N}\{/}do
Send request(l, p) to node j;
Wait until token is received;
HaveToken := true
end;

Critical Section;

If IsNew and IsNew

then Wait until request is received
else begin
if not IsNew and IsNew(0)
then begin
Delete(P):
HaveToken : = false;
Send token, Token(Tail(P)), to Head(P)
end
else If IsNew and not IsNew
then begin
Delete(Q);
Append(P, Q) + P; {MergeP, 0) + P}
HaveToken : = false;
Send token, Token(P), to Head(P)
end
else begin
Delete(P);
HaveToken := false;
Append(P. 0) + P; { Mqe(P, Q) + P}
Send token, Token(P), to Head(P)
end
Requesting : = false
end;
end;

procedure request-arrives;
begin
if not HaveToken
then forj = {1,2, , N}\{/} do
discard request( j, p)
else begin
Delete(P);
forj={1,2,...,}\{I}do
Add( j, 0) + 0 ; {insert an additional request}
Append(P, 0) + P ; {Merge(P, Q) -. P)
Send token, Token(P), to Head(P)
end
end;

FIG. 1. The synchronization algorithm for an error-free network.

ities, P-system. The structure of the algorithm for a real- function Append with instances of function Merge. This is
time system, RT-system, is identical to the structure of this presented in comments to several lines of Algorithm. Proce-
algorithm. Moreover, algorithm for RT-system can be eas- dure wants-to-enter is called when a node attempts to en-
ily obtained from Algorithm by replacing all instances of ter the critical section and procedure request-arrives is exe-
80 ANDRZEJ GOSCINSKI
cuted when a request message arrives. In both cases the
same suitable algorithm is executed by each node.
3. ASSERTIONS
3.1. Mutual Exclusion
Mutual exclusion is achieved when no pair of processes
located in different nodes are ever simultaneously in their
critical sections [ 111. If process i is executing in its critical
section then no other process can be executing in its critical
section.
ASSERTION. Mutual exclusion is achieved.
Proof Only one process can be in its critical section at
a time, since there is only a single token.
3.2. Deadlock
Distributed processes are deadlocked if no process is in
its critical section, and no requesting process can ever pro-
ceed to its own critical section [ 1 I].
ASSERTION. Deadlock is impossible.
Proof: Suppose that two processes are deadlocked. This
means that two distributed processes are in their critical sec-
tions and wait for each other. This means that two nodes
hold tokens. This contradicts the token passing approach,
which asserts that there is one and only one token in the
system.
3.3. Starvation
Starvation occurs when one process must wait indefi-
nitely to enter its critical section while other processes are
entering and exiting their own critical sections [ 111.
ASSERTION. Freedom from starvation is ensured.
Proof: This assertion must be proved for both systems
separately, because of different approaches to the construc-
tion of the queue P in each case when the original queue
obtained with the token does contain elements as well as
the requesting queue Q is not empty.
P-system-Because (i) there is a finite number of distrib-
uted processes (N) ordered according to the priority values,
(ii) processes creating original queue P (received with the
token) have greater priority than processes from the re-
questing queue (ordered and attached to the original
sequence), and (iii) processes are served in the priority or-
der, the time of waiting for entering a critical section is fi-
nite. So, starvation is impossible.
RT-system-Because (i) all processes are ordered ac-
cording to the decreasing values of the remaining times to
run, (ii) the remaining times are finite, and (iii) remaining
time decrementing operations on appropriate fields of the
queue P and requesting queue Q elements, i.e., T(i), are
performed, a time of waiting for entering a critical section
is finite. Thus again, starvation is impossible.
4. MESSAGE TRAFFIC
Both algorithms proposed require one broadcast message
request sent by a process wanting to enter its critical section
and one token message passing, which makes it possible to
enter the critical section. The process in the node holding
the token does not need to send the request message to itself.
Since the network has N processes distributed among N
nodes, N messages have to be exchanged for one mutual
exclusion invocation.
The worst case delay involved in granting the critical sec-
tion resource is the period of time beginning with the re-
questing node asking for the critical section and ending
when that node enters its critical section for the process with
the longest remaining time. This delay can be assessed in
the following way. Let 7tp be the time of passing the token
from one node to another (the time of transferring the re-
quest message has approximately the same value) and T,,
be the length (in time) of the critical section. The process
with the longest remaining time occupies the (N - 1) th po-
sition in the ordered sequence. Assuming that the time, de-
noted by t, spent by procedures of an operating system to
perform operations such as inserting processes in the prior-
ity request queue Q, merging two priority queues, and re-
maining time decrementing operations is much smaller
than the time of token passing, i.e., t < 7tp, then the worst
case delay 6 is expressed by
i.e., the upper boundary can be assessed. This value depends
greatly on the length (in time) of the critical section. It is
likely that when the critical section includes exchange of
messages between producers and consumers, this time
could be very long. But, this is the problem of all synchroni-
zation algorithms.
5. PROBLEMS OF REAL NETWORKS
Three types of failures can happen in real networks: (i)
network node-oriented failures such as insertion of new
nodes, removal of nodes, or node failure; (ii) token-ori-
ented failures such as loss of the token, or the existence of
more than one token; and (iii) message delivery failures;
i.e., messages are delivered out of order. Because these algo-
rithms do not provide any detection algorithm before some-
thing extraordinary happens, these cases are treated uni-
formly. So, to prevent this situation from stopping the pro-
posed mutual exclusion algorithm, a time-out recovery
mechanism may be used.
 TWO ALGORITHMS FOR MUTUAL EXCLUSION 81

5. I. Node-Oriented Events var Time-out : integer;

HaveAck : Boolean;
Insertion of New Nodes. New processes located in new procedure send-token;
nodes may be added to the group participating in the mu- begin
tual exclusion algorithm-this operation can be accommo- HaveToken := false;
Send token, Token(P), to Head(Tail(P));
dated because the algorithms do not use and maintain any
Set Time-out;
logical ring. The only requirement is a unique name for the repeat
new process to be added. This problem is solved by a nam- if Time-out d 0
ing facility. Of course, the delay time will increase with each then begin
insertion, and this can cause a problem in a boundary case Delete(Tail(P));
Send token, Token(P), to Head(Tail(F’));
(real-time requirements).
Set Time-out
If the node was previously operational in the group (e.g., end
it failed and is now restoring), it should treat itself as a until HaveAck;
brand new process, sending its request message if it wants Clear buffer with queue P
to enter its critical section. If before failure this process held end;
the token, it should destroy this token and its associated FIG. 2. Procedure send_token.
queue P, to avoid the existence of two tokens in the system.
Removal of Nodes. A node wishing to leave the group
may do so if it presently does not hold the token by notify- Loss ofthe Token. If all processes wanting to enter their
ing all other nodes of its action. This is necessary to remove critical sections sent their request messages and did not re-
it from the queue P. The node holding the token must send ceive the token during the time 6 + 2~, they can assume that
an acknowledgment message (after removing the node). the token is lost. An appropriate action should be per-
Node Failure. It can happen that the node or process in formed. If the token is lost, an election is called and the
the node fails and will not behave as expected. The node elected process generates a new token [ 3, 7, 81. All pro-
failure can occur in two different situations: first, when the cesses which want to enter their critical sections have to
node holds the token, and second, when it sends a request send request messages again to reconstruct queue P. There
message wanting to enter its critical section and is recorded are other algorithms to solve these problems.
in the queue P. Because, as we said, these algorithms do not Existence of More than One Token. If more than one to-
provide any detection algorithm before the occurrence of ken is detected, the processes (located in two nodes) have
special events (e.g., long delays of holding the token), we to make an agreement to destroy one of the tokens they
use information associated with such events. We propose a hold. At the same time, if any two processes are deadlocked
time-out recovery mechanism to be used. (both of them are in their critical sections), recovery from
To implement such a mechanism, each process 4 sending this deadlock will be done.
the token to process, say i, , starts its time-out clock. If pro-
cess 4 did not receive from process il an acknowledgment 5.3. Messages Arrive Out of Order
message ack(j) before time-out, then it assumes that pro-
cess i, failed and sends the token with modified queue P to A real network can cause the loss of messages or their out
next node, say iz. If all processes which sent their request of order arrival. In the latter case, the situation is critical
messages do not receive the token during the token-rotation when requests from distinct sources arrive in the token des-
time, they can assume that the token is lost. Consequently, tination later than a token which was sent off earlier than
the appropriate action should be undertaken (see loss of the these requests; i.e., they arrive out of order. In the case of
token section). The appropriate procedure called send-to- the proposed algorithms such an out of order arrival can be
ken is illustrated in Fig. 2. treated as the loss of the request [ 41. The loss of a request
Thus, this recovery mechanism requires (i) sending the requires its retransmission. A process is allowed to retrans-
token and acknowledgment message (the acknowledgment mit after time-out. A time-out value should be set up to at
could be the token, in which case the sending process has to least the value of the worst case delay. The relevant part of
use group addressing), and (ii) performing operations on a procedure wants-to-enter is shown in Fig. 3.
the queue P and the requesting queue by two “neighbor”
(predecessor, successor) processes/nodes. 6. CONCLUSION

5.2. Token-Oriented Failures Two algorithms that implement mutual exclusion in

computer networks working in either an environment
Protocols providing for failure detection and recovery which requires priorities of processes or a real-time environ-
have to be provided to deal with these failures. ment are presented. Because of many problems associated
82 ANDRZEJ GOSCINSKI

procedure wants-to-enter;
begin
Requesting : = true;
if not HaveToken {modification starts}
then begin
repeat
foralljin{1,2,...,N}\{/}do
Send request(l, p. r-id, ret-id) to nodej;
Set up req.-time-out;
Wait until token is received;
until HaveToken;
HaveToken := true;
Send acknowledgment, Ack(Head(P));
Delete(f)
end
end;

FIG. 3. Retransmission of a request.

with time stamps event-ordering algorithms, priority-based
event-ordering of processes has been proposed. For real-
time systems and systems using priorities for scheduling, a
priority-based approach is more natural and improves the
performance of the total system. Both algorithms are con-
current and distributed, and use N messages per critical sec-
tion. In both cases freedom from starvation is ensured.
There are mechanisms to handle node insertion and re-
moval, node failure, the loss of the token, existence of more
than one token, and out of order arrival of messages.
ACKNOWLEDGMENTS
I thank my colleagues A. Quaine and C. Vance for reading drafts of this
paper and for their valuable comments. 1 am also grateful to the referees
for valuable comments on the paper.
REFERENCES
1. Chandy, K. M. A mutual exclusion algorithm for distributed systems.
Tech. Rep., University of Texas, 1982.
2. Fran&t, W. R., and Chlamtac, I. Local Networks. Lexington Books,
Lexington, MA, 1982.
3. Garcia-Molina, H. Elections in a distributed computing system. IEEE
Trans. Camp. C-31, I (Jan. 1982), 48-59.
4. Goscinski, A. Distributed Operating Systems. The Logical Design. Ad-
dison-Wesley, in press.
5. Hille, R. F. Data Abstraction and Program Development. Prentice-
Hall, New York, 1987.
6. Lamport, L. Time, clocks and the ordering of events in a distributed
system. Comm. ACMZI, 7 (July 1978), 558-565.
7. Le Lann, G. Distributed systems-Towards a formal approach. Proc.
IFIP Congress, Toronto. North-Holland, Amsterdam, Aug. 1977.
8. Le Lann, G. Algorithms for distributed data sharing systems which use
tickets. Proc. 3rd Berkeley Workshop, Aug. 1978.
9. Mullender, S. J. Report on the Second European SIGOPS Workshop
“Making Distributed Systems Work”, Amsterdam, Netherland, Sept.
1986, Oper. Syst. Rev. 21, 1 (Jan. 1987), 49-84.
10. Peterson, J. L., and Silbershatz, A. Operating Systems Concepts. Addi-
son-Wesley, Reading, MA, 1985.
I I. Ricart, G., and Agrawala, A. K. An optimal algorithm for mutual ex-
clusion in computer networks. Comm. ACM 24, 1 (Jan. I98 1), 9- 17.
12. Suzuki, I., and Kasami, T. A distributed mutual exclusion algorithm.
ACM Trans. Comput. Syst. 3,4 (Nov. 1985), 344-349.
ANDRZEJ GOSCINSKI received the M.Sc. degree in automatic control
in 1968, the Ph.D. degree in control engineering and computer science in
1973, and the D.Sc. degree in computer science in 1976 from the St. Staszic
University ofMining and Metallurgy, Krakow, Poland. From 1968 to 1984
he was with the Institute of Computer Science and Control Engineering, at
that University. In 1977 Dr. Goscinski was with the Department ofcontrol
Engineering, Computer Engineering and Information Sciences, Case
Western Reserve University, Cleveland, Ohio. In January 1985 he joined
the Department of Computing Science at the University of Wollongong,
and since January 1987 has been with the Department of Computer Sci-
ence, University College, University of New South Wales, Australian De-
fence Force Academy. He is currently a Senior Lecturer. His current re-
search activities are in distributed operating systems, applications of dis-
tributed systems, networks, and communication protocols. Dr. Goscinski
is a member of the IEEE Computer Society.

Received September 20, 1987; revised December 5, 1988