Chapter 2 - Architectures

Introduction

 how to organize the collection of software components

 logical organization and
 physical organization
 i.e., software architectures: how they are organized and how
they communicate
 we will discuss
 architectural styles
 system architectures: centralized vs decentralized ones

2
2.1 Architectural Styles
 the logical organization of distributed systems into
software components
 a component is a modular unit with well-defined required
and provided interfaces that is replaceable within its
environment
 a connector is a mechanism that mediates
communication, coordination, or cooperation among
components, e.g., facilities for RPC, message passing, or
streaming data
 there are various architectural styles
 Layered architectures
 Object-based architectures
 Data-centered architectures
 Event-based architectures

3
 Layered architectures
 components are organized in a layered fashion where a
component at layer Li is allowed to call components at
the underlying layer Li-1, but not the other way around;
e.g., network layers

the layered architectural style

4
 Object-based architectures
 each object corresponds to a component and these
components are connected through a remote procedure
call mechanism (client-server paradigm)

the object-based architectural style

5
 Data-centered architectures
 processes communicate through a common repository;
e.g., a shared distributed file system
 Event-based architectures
 processes communicate through the propagation of events
 publish/subscribe systems
 processes publish events and the middleware ensures that
only those processes that subscribed to those events will
receive them

the event-based architectural style 6

 shared data spaces
 event-based architectures combined with data-centered
architectures
 processes are decoupled in time

the shared data-space architectural style

7
2.2 System Architectures
 the logical organization of distributed systems into
software components or how are processes organized in a
system
2.2.1 Centralized Architectures
 thinking in terms of clients requesting services from
servers

general interaction between a client and a server

8
 communication between client and server can be
 by a connectionless protocol if the underlying network is
fairly reliable; efficient since there is no much overhead
 but assuring reliability is difficult
 when messages are lost or corrupted let the client
send the request again; applicable only for
idempotent operations
 an operation is idempotent if it can be repeated
multiple times without harm; e.g., reading a record
in a database
 see later in Chapter 8: Fault Tolerance
 by reliable connection-oriented protocol if the underlying
network is unreliable
 establishing and terminating connections is
expensive

9
 Application Layering
 no clear distinction between a client and a server; for
instance a server for a distributed database may act as a
client when it forwards requests to different file servers
 three levels exist
 the user-interface level: implemented by clients and
contains all that is required by a client; usually
through GUIs, but not necessarily
 the processing level: contains the applications
 the data level: contains the programs that maintain
the actual data dealt with

10
 e.g., the general organization of an Internet search engine into
three different layers

 Client-Server Architectures
 how to physically distribute a client-server application
across several machines
 Multitiered Architectures 11
 Two-tiered architecture: alternative client-server organizations
(a) put only terminal-dependent part of the user interface on the
client machine and let the applications remotely control the
presentation
(b) put the entire user-interface software on the client side
(c) move part of the application to the client, e.g. checking
correctness in filling forms
12
(d) and (e) are for powerful client machines (more popular)
 three tiered architecture: an example of a server acting as a client
 an example is the organization of Web sites

13
2.2.2 Decentralized Architectures
 vertical distribution: the ones discussed so far where the
different tiers correspond directly with the logical
organization of applications; place logically different
components on different machines
 horizontal distribution: physically split up the client or the
server into logically equivalent parts
 an example is a peer-to-peer system where processes
are equal and hence each process acts as a client and a
server at the same time (servent)
 read about the different approaches of peer-to-peer
architecture - pages 44 - 51 and about Architectures
versus Middleware - pages 54 - 66

14
 another example is the horizontal distribution of a Web
service

15
Chapter 3 - Processes

16
 Introduction

 communication takes place between processes

 a process is a program in execution
 from OS perspective, management and scheduling of
processes is important
 other important issues arise in distributed systems
 multithreading to enhance performance
 how are clients and servers organized
 process or code migration to achieve scalability and to
dynamically configure clients and servers

17
 3.1 Threads and their Implementation
 threads can be used in both distributed and nondistributed
systems
 Threads in Nondistributed Systems
 a process has an address space (containing program text
and data) and a single thread of control, as well as other
resources such as open files, child processes, accounting
information, etc.
Process 1 Process 2 Process 3

three processes each with one thread one process with three threads 18
 each thread has its own program counter, registers, stack, and
state; but all threads of a process share address space, global
variables and other resources such as open files, etc.

19
 Threads take turns in running
 Threads allow multiple executions to take place in the same
process environment, called multithreading
 Thread Usage – Why do we need threads?
 e.g., a wordprocessor has different parts; parts for
 interacting with the user
 formatting the page as soon as changes are made
 timed savings (for auto recovery)
 spelling and grammar checking, etc.
1. Simplifying the programming model: since many activities are
going on at once
2. They are easier to create and destroy than processes since
they do not have any resources attached to them
3. Performance improves by overlapping activities if there is too
much I/O; i.e., to avoid blocking when waiting for input or
doing calculations, say in a spreadsheet
4. Real parallelism is possible in a multiprocessor system 20
 having finer granularity in terms of multiple threads per process
rather than processes provides better performance and makes it
easier to build distributed applications
 in nondistributed systems, threads can be used with shared data
instead of processes to avoid context switching overhead in
interprocess communication (IPC)

context switching as the result of IPC

21
 Thread Implementation
 threads are usually provided in the form of a thread package
 the package contains operations to create and destroy a thread,
operations on synchronization variables such as mutexes and
condition variables
 two approaches of constructing a thread package
a. construct a thread library that is executed entirely in user
mode (the OS is not aware of threads)
 cheap to create and destroy threads; just allocate and free
memory
 context switching can be done using few instructions; store
and reload only CPU register values
 disadv: invocation of a blocking system call will block the
entire process to which the thread belongs and all other
threads in that process
b. implement them in the OS’s kernel
 let the kernel be aware of threads and schedule them
 expensive for thread operations such as creation and
deletion since each requires a system call
22
 solution: use a hybrid form of user-level and kernel-level threads,
called lightweight process (LWP)
 a LWP runs in the context of a single (heavy-weight) process, and
there can be several LWPs per process
 the system also offers a user-level thread package for some
operations such as creating and destroying threads, for thread
synchronization (mutexes and condition variables)
 the thread package can be shared by multiple LWPs

combining kernel-level lightweight processes and user-level threads

23
 Threads in Distributed Systems
 Multithreaded Clients
 consider a Web browser; fetching different parts of a page can

be implemented as a separate thread, each opening its own

TCP/IP connection to the server or to separate and replicated
servers
 each can display the results as it gets its part of the page

 Multithreaded Servers
 servers can be constructed in three ways

a. single-threaded process
 it gets a request, examines it, carries it out to completion

before getting the next request

 the server is idle while waiting for disk read, i.e., system calls

are blocking

24
b. threads
 threads are more important for implementing servers
 e.g., a file server
 the dispatcher thread reads incoming requests for a file
operation from clients and passes it to an idle worker thread
 the worker thread performs a blocking disk read; in which
case another thread may continue, say the dispatcher or
another worker thread

a multithreaded server organized in a dispatcher/worker model

25
 c. finite-state machine
 if threads are not available
 it gets a request, examines it, tries to fulfill the request from
cache, else sends a request to the file system; but instead of
blocking it records the state of the current request and
proceeds to the next request
 Summary

Model Characteristics
Single-threaded process No parallelism, blocking system calls
Parallelism, blocking system calls
Threads
(thread only)
Finite-state machine Parallelism, nonblocking system calls
three ways to construct a server

26
 3.2 Anatomy of Clients
 Two issues: user interfaces and client-side software for
distribution transparency
a. User Interfaces
 to create a convenient environment for the interaction of a

human user and a remote server; e.g. mobile phones with

simple displays and a set of keys
 GUIs are most commonly used

 The X Window System (or simply X)

 it has the X kernel: the part of the OS that controls the

terminal (monitor, keyboard, pointing device like a mouse)
and is hardware dependent
 contains all terminal-specific device drivers through the
library called xlib

27
the basic organization of the X Window System

28
b. Client-Side Software for Distribution Transparency
 in addition to the user interface, parts of the processing and

data level in a client-server application are executed at the

client side
 an example is embedded client software for ATMs, cash

registers, etc.
 moreover, client software can also include components to

achieve distribution transparency

 e.g., replication transparency

 assume a distributed system with replicated servers; the

client proxy can send requests to each replica and a client
side software can transparently collect all responses and
passes a single return value to the client application

29
 transparent replication of a server using a client-side solution

 access transparency and failure transparency can also be

achieved using client-side software

30
 3.3 Servers and design issues
3.3.1 General Design Issues
 How to organize servers?
 Where do clients contact a server?
 Whether and how a server can be interrupted
 Whether or not the server is stateless

a. Wow to organize servers?

 Iterative server
 the server itself handles the request and returns the result

 Concurrent server
 it passes a request to a separate process or thread and

waits for the next incoming request; e.g., a multithreaded

server; or by forking a new process as is done in Unix

31
b. Where do clients contact a server?
 using endpoints or ports at the machine where the server is
running where each server listens to a specific endpoint
 how do clients know the endpoint of a service?
 globally assign endpoints for well-known services; e.g. FTP

is on TCP port 21, HTTP is on TCP port 80

 for services that do not require preassigned endpoints, it

can be dynamically assigned by the local OS

 IANA (Internet Assigned Numbers Authority) Ranges

 IANA divided the port numbers into three ranges

 Well-known ports: assigned and controlled by IANA for

standard services, e.g., DNS uses port 53
32
 Registered ports: are not assigned and controlled by IANA; can
only be registered with IANA to prevent duplication e.g., MySQL
uses port 3306
 Dynamic ports or ephemeral ports : neither controlled nor
registered by IANA
 how can the client know this endpoint? two approaches
i. have a daemon running and listening to a well-known endpoint; it
keeps track of all endpoints of services on the collocated server
 the client will first contact the daemon which provides it with
the endpoint, and then the client contacts the specific server

33
 Client-to-server binding using a daemon
ii. use a superserver (as in UNIX) that listens to all endpoints and
then forks a process to take care of the request; this is instead of
having a lot of servers running simultaneously and most of them
idle

Client-to-Server binding using a superserver

34
c. Whether and how a server can be interrupted
 for instance, a user may want to interrupt a file transfer, may
be it was the wrong file
 let the client exit the client application; this will break the
connection to the server; the server will tear down the
connection assuming that the client had crashed
or
 let the client send out-of-bound data, data to be processed by
the server before any other data from the client; the server
may listen on a separate control endpoint; or send it on the
same connection as urgent data as is in TCP
d. Whether or not the server is stateless
 a stateless server does not keep information on the state of its
clients; for instance a Web server
 soft state: a server promises to maintain state for a limited
time; e.g., to keep a client informed about updates; after the
time expires, the client has to poll
35
  a stateful server maintains information about its clients; for
instance a file server that allows a client to keep a local copy
of a file and can make update operations

3.3.2 Server Clusters

 a server cluster is a collection of machines connected through a
network (normally a LAN with high bandwidth and low latency)
where each machine runs one or more servers
 it is logically organized into three tiers

36
the general organization of a three-tiered server cluster

37
 Distributed Servers
 the problem with a server cluster is when the logical switch
(single access point) fails making the cluster unavailable
 hence, several access points can be provided where the
addresses are publicly available leading to a distributed server
 e.g., the DNS can return several addresses for the same host
name

38
 3.4 Code Migration
 so far, communication was concerned on passing data
 we may pass programs, even while running and in
heterogeneous systems
 code migration also involves moving data as well: when a
program migrates while running, its status, pending signals, and
other environment variables such as the stack and the program
counter also have to be moved

39
 Reasons for Migrating Code
 to improve performance; move processes from heavily-loaded to
lightly-loaded machines (load balancing)
 to reduce communication: move a client application that performs
many database operations to a server if the database resides on
the server; then send only results to the client
 to exploit parallelism (for nonparallel programs): e.g., copies of a
mobile program (a crawler as is called in search engines) moving
from site to site searching the Web

40
 to have flexibility by dynamically configuring distributed systems:
instead of having a multitiered client-server application deciding
in advance which parts of a program are to be run where

the principle of dynamically configuring a client to communicate to a server; the

client first fetches the necessary software, and then invokes the server
41
 Models for Code Migration
 a process consists of three segments: code segment (set of
instructions), resource segment (references to external resources
such as files, printers, ...), and execution segment (to store the
current execution state of a process such as private data, the
stack, the program counter)
 Weak Mobility
 transfer only the code segment and may be some initialization
data; in this case a program always starts from its initial stage,
e.g. Java Applets
 execution can be by the target process (in its own address
space like in Java Applets) or by a separate process

42
 Strong Mobility
 transfer code and execution segments; helps to migrate a
process in execution
 can also be supported by remote cloning; having an exact
copy of the original process and running on a different
machine; executed in parallel to the original process; UNIX
does this by forking a child process
 migration can be
 sender-initiated: the machine where the code resides or is
currently running; e.g., uploading programs to a server; may
need authentication or that the client is a registered one
 receiver-initiated: by the target machine; e.g., Java Applets;
easier to implement

43
Summary of models of code migration

alternatives for code migration

44
 Migration and Local Resources
 how to migrate the resource segment
 not always possible to move a resource; e.g., a reference to TCP
port held by a process to communicate with other processes
 Types of Process-to-Resource Bindings
 Binding by identifier (the strongest): a resource is referred by its
identifier; e.g., a URL to refer to a Web page or an FTP server
referred by its Internet (IP) address
 Binding by value (weaker): when only the value of a resource is
needed; in this case another resource can provide the same
value; e.g., standard libraries of programming languages such as
C or Java which are normally locally available, but their location
in the file system may vary from site to site
 Binding by type (weakest): a process needs a resource of a
specific type; reference to local devices, such as monitors,
printers, ...

45
 in migrating code, the above bindings cannot change, but the
references to resources can
 how can a reference be changed? depends whether the resource
can be moved along with the code, i.e., resource-to-machine
binding
 Types of Resource-to-Machine Bindings
 Unattached Resources: can be easily moved with the migrating
program (such as data files associated with the program)
 Fastened Resources: such as local databases and complete
Web sites; moving or copying may be possible, but very costly
 Fixed Resources: intimately bound to a specific machine or
environment such as local devices and cannot be moved
 we have nine combinations to consider

46
 Resource-to machine binding
Unattached Fastened Fixed
By identifier MV (or GR) GR (or MV) GR
Process-to-
resource binding By value CP (or MV, GR) GR (or CP) GR
By type RB (or GR, CP) RB (or GR, CP) RB (or GR)

actions to be taken with respect to the references to local resources when

migrating code to another machine

 GR: Establish a global system wide reference

 MV: Move the resource
 CP: Copy the value of the resource
 RB: Rebind process to a locally available resource

47
 Migration in Heterogeneous Systems
 distributed systems are constructed on a heterogeneous
collection of platforms, each with its own OS and machine
architecture
 heterogeneity problems are similar to those of portability
 easier in some languages
 for scripting languages the source code is interpreted
 for Java an intermediary code is generated by the compiler for
a virtual machine
 in weak mobility
 since there is no runtime information, compile the source code
for each potential platform
 in strong mobility
 difficult to transfer the execution segment since there may be
platform-dependent information such as register values; Read
the book about possible solutions

48
Chapter 4 - Communication

 Introduction
 interprocess communication is at the heart of all distributed
systems
 communication in distributed systems is based on message
passing as offered by the underlying network as opposed to
using shared memory
 modern distributed systems consist of thousands of
processes scattered across an unreliable network such as
the Internet
 unless the primitive communication facilities of the network
are replaced by more advanced ones, development of large
scale Distributed Systems becomes extremely difficult

50
 Objectives of the Chapter
 review of how processes communicate in a network (the
rules or the protocols) and their structures
 introduce the four widely used communication models for
distributed systems:
 Remote Procedure Call (RPC)
 Remote Method Invocation (RMI)
 Message-Oriented Middleware (MOM)
 Streams

51
 A. Layered Protocols
 two computers, possibly from different manufacturers, must
be able to talk to each other
 for such a communication, there has to be a standard
 The ISO OSI (Open Systems Interconnection) Reference
Model is one of such standards - 7 layers
 TCP/IP protocol suite is the other; has 4 or 5 layers
 OSI
 Open – to connect open systems or systems that are open
for communication with other open systems using standard
rules that govern the format, contents, and meaning of the
messages sent and received
 these rules are called protocols
 two types of protocols: connection-oriented and
connectionless

52
layers, interfaces, and protocols in the OSI model
53
 Media (lower) Layers
 Physical: Physical characteristics of the media
 Data Link: Reliable data delivery across the link
 Network: Managing connections across the network
or routing
 Transport: End-to-end connection and reliability
(handles
lost packets); TCP (connection-oriented),
UDP (connectionless), etc.
 Session: Managing sessions between applications
(dialog control and synchronization); rarely
supported
 Presentation: Data presentation to applications; concerned
with the syntax and semantics of the
information transmitted
 Application: Network services to applications; contains
protocols that are commonly needed by
users; FTP, HTTP, SMTP, ...
Host (upper)
54
 Transport Protocols: Client-Server TCP

assuming no messages are lost,

 the client initiates a setup

connection using a three-way

handshake (1-3)
 the client sends its request (4)

 it then sends a message to

close the connection (5)

 the server acknowledges

receipt and informs the client

that the connection will be
closed down (6)
 then sends the answer (7)

followed by a request to close

the connection (8)
 the client responds with an ack

normal operation of TCP to finish conversation (9)

55
 much of the overhead in TCP is for managing the connection
 combine connection setup with
request and closing connection
with answer
 such protocol is called TCP for
Transactions (T/TCP)
 the client sends a single
message consisting of a setup
request, service request, and
information to the server that
the connection will be closed
down immediately after
receiving the answer (1)
 the server sends acceptance of
connection request, the
answer, and a connection
release (2)
 the client acknowledges tear
down of the connection (3) transactional TCP 56
 Application Protocols
 file transfer (FTP - File Transfer Protocol)

 HTTP - Hypertext Transfer Protocol for accessing data on the

WWW

 Middleware Protocols
 a middleware is an application that contains general-purpose

protocols to provide services

 example of middleware services

 authentication and authorization services - Chapter 8

 distributed transactions (commit protocols; locking
mechanisms) - Chapters 5 and 7
 middleware communication protocols (calling a procedure
or invoking an object remotely, synchronizing streams for
real-time data, multicast services) - see later in this Chapter
 hence an adapted reference model for networked
communications is required
57
 4.1 Remote Procedure Call
 the first distributed systems were based on explicit message
exchange between processes through the use of explicit
send and receive procedures; but do not allow access
transparency
 in 1984, Birrel and Nelson introduced a different way of
handling communication: RPC
 it allows a program to call a procedure located on another
machine
 simple and elegant, but there are implementation problems
 the calling and called procedures run in different address
spaces
 parameters and results have to be exchanged; what if the
machines are not identical?
 what happens if both machines crash?

58
 Conventional Procedure Call, i.e., on a single machine
 e.g. count = read (fd, buf, bytes); a C like statement, where
fd is an integer indicating a file
buf is an array of characters into which data are read
bytes is the number of bytes to be read
Stack
pointer

Stack
pointer

parameter passing in a local procedure

the stack while the called
call: the stack before the call to read procedure is active
 parameters can be call-by-value (fd and bytes) or call-by
reference (buf) or in some languages call-by-copy/restore 59
 Client and Server Stubs
 RPC would like to make a remote procedure call look the
same as a local one; it should be transparent, i.e., the calling
procedure should not know that the called procedure is
executing on a different machine or vice versa
 when a program is compiled, it uses different versions of
library functions called client stubs
 a server stub is the server-side equivalent of a client stub

principle of RPC between a client and server program 60

 Steps of a Remote Procedure Call
1. Client procedure calls client stub in the normal way
2. Client stub builds a message and calls the local OS
(packing parameters into a message is called parameter
marshaling)
3. Client's OS sends the message to the remote OS
4. Remote OS gives the message to the server stub
5. Server stub unpacks the parameters and calls the server
6. Server does the work and returns the result to the stub
7. Server stub packs it in a message and calls the local OS
8. Server's OS sends the message to the client's OS
9. Client's OS gives the message to the client stub
10. Stub unpacks the result and returns to client
 hence, for the client remote services are accessed by making
ordinary (local) procedure calls; not by calling send and
receive
 server machine vs server process; client machine vs client process
61
 Parameter Passing
1. Passing Value Parameters
 e.g., consider a remote procedure add(i, j), where i and j are
integer parameters

steps involved in doing remote computation through RPC

62
 the above discussion applies if the server and the client
machines are identical
 but that is not the case in large distributed systems
 the machines may differ in data representation (e.g., IBM
mainframes use EBCDIC whereas IBM PCs use ASCII)
 there are also differences in representing integers(1’s
complement or 2’s complement) and floating-point numbers
 byte numbering may be different (from right to left in Pentium
called little endian and left to right in SPARC, big endian)
 e.g.
 consider a procedure with two parameters, an integer and a
four-character string; each one 32-bit word (5, “JILL”)
 the sender is Intel and the receiver is SPARC

63
  original message on the Pentium
 (the numbers in boxes indicate the address of each
byte)

 the message after receipt on the SPARC; wrong integer (224+226 = 83886080),
but correct string

64
 one approach is to invert the bytes of each word after
receipt

 the message after being inverted (correct integer but wrong string)

 additional information is required to tell which is an

integer and which is a string

65
2. Passing Reference Parameters
 assume the parameter is a pointer to an array
 copy the array into the message and send it to the server
 the server stub can then call the server with a pointer to this
array
 the server then makes any changes to the array and sends it
back to the client stub which copies it to the client
 this is in effect call-by-copy/restore
 optimization of the method
 one of the copy operations can be eliminated if the stub
knows whether the parameter is input or output to the
server
 if it is an input to the server (e.g., in a call to write), it need
not be copied back
 if it is an output, it need not be sent over in the first place;
only send the size
 the above procedure can handle pointers to simple arrays
and structures, but difficult to generalize it to an arbitrary
data structure 66
 Parameter Specification and Stub Generation
 the caller and the callee need to use the same protocol

(format of messages) and the same steps; with such rules the
client and server stubs can assemble, communicate, and
interpret messages correctly
 consider the following example; the procedure foobar has 3

parameters: a character, a floating point number, and an array

of 5 integers

 assume a word is 4 bytes

 one possibility is to transmit the character
in the rightmost byte, a float as a whole
word, and an array as a group of words
equal to the array length preceded by a
word giving the length
 this way both client stub and server stub
can understand outgoing and incoming the corresponding message
67
messages
 other issues that need the agreement of the client and the
server
 how are simple data structures like integers (e.g. 2’s
complement), characters (e.g. 16-bit Unicode), Booleans, ...
represented?
 endianess
 which transport protocol to use - the connection-oriented
TCP or the unreliable connectionless UDP

68
 Extended RPC Models
 to solve some of the shortcomings of the original model

 no need of network communication if server and client are

on the same machine
 no need of blocking for the client in some cases

a. Doors
 the original RPC model assumes that the caller and the
callee can communicate only by means of passing
messages over a network; what if they are colocated on
the same machine?
 a door is a generic name for a procedure in the address
space of a server process that can be called by a process
colocated with the server
 support from the local OS is required

69
1. the server process registers a door before it can be called
(door_create) and a name is attached to it
2. a client calls a door by a system call (door_call) including
all parameters
3. results are returned by the system call door_return

the principle of using doors as IPC mechanism

70
 benefit: they allow the use of a single mechanism (procedure
calls) for communication
 disadv: application developers have to be aware of where a
procedure is located; is it
 local within the current process
 local to a different process on the same machine
 a remote process

71
b. Asynchronous RPC
 if there is no need to block the client until it gets a reply
 two cases
1. if there is no result to be returned
 e.g., adding entries in a database, ...
 the server immediately sends an ack promising that it
will carryout the request
 the client can now proceed without blocking

a) the interconnection between client and server in a traditional RPC

b) the interaction using asynchronous RPC 72
2. if the result can be collected later
 e.g., prefetching network addresses of a set of hosts, ...
 the server immediately sends an ack promising that it
will carryout the request
 the client can now proceed without blocking
 the server later sends the result

a client and server interacting through two asynchronous RPCs

73
  the above method combines two asynchronous RPCs
and is sometimes called deferred synchronous RPC
 variants of asynchronous RPC
 let the client continue without waiting even for an ack,
called one-way RPC
 problem: if reliability of communication is not
guaranteed

74
 DCE (Distributed Computing Environment) RPC
 a middleware and an example RPC system developed by
OSF (Open Software Foundation), now The Open Group
 it is designed to execute as a layer of abstraction between
existing OSs and distributed applications
 the Open Group sells the source code and vendors integrate
it into their systems
 it uses the client-server programming model and
communication is by means of RPCs
 services
 distributed file service: a worldwide file system that
provides a transparent way of accessing files
 directory service: to keep track of the location of all
resources in the system (machines, printers, data,
servers, ...); a process can ask for a resource without
knowing its location
 security service: for protecting resources; access is only
through authorization 75
  distributed time service: to maintain clocks on different
machines synchronized (clock synchronization is covered
in Chapter 5)
 Steps in writing a Client and a Server in DCE RPC
 the system consists of languages, libraries, daemons,
utility programs, ... for writing clients and servers
 IDL (Interface Definition Language) is the interface
language - the glue that holds everything together
 it contains type definitions, constant declarations
and what the procedures do (only their syntax)

76
 Uuidgen generates a prototype IDL file with a globally unique interface
identifier
 the IDL file is edited (filling the names of procedures and parameters) and
the IDL compiler is called to generate 3 files
 the application writer writes the client and server codes and are then
77
compiled and linked together with the stubs
 Binding a Client to a Server in DCE RPC
 for a client to call a server, the server must be registered (1
& 2)
 the registration allows the client to locate the server and
bind to it
 the DCE daemon maintains a table (server, endpoint) and the
protocols the server uses
 the directory server maintains the locations of all resources
in the system (machines, servers, data,, ...)
 two steps for server location
 locate the server’s machine (3)
 locate the server process on that machine (which has
what is called an endpoint or port) (4)

78
79
  4.2 Remote Object (Method) Invocation (RMI)
 resulted from object-based technology that has proven its
value in developing nondistributed applications
 it is an expansion of the RPC mechanisms
 it enhances distribution transparency as a consequence of
an object that hides its internal from the outside world by
means of a well-defined interface
 Distributed Objects
 an object encapsulates data, called the state, and the
operations on those data, called methods
 methods are made available through interfaces
 the state of an object can be manipulated only by invoking
methods
 this allows an interface to be placed on one machine while
the object itself resides on another machine; such an
organization is referred to as a distributed object
 the state of an object is not distributed, only the interfaces
are; such objects are also referred to as remote objects 80
 the implementation of an object’s interface is called a proxy
(analogous to a client stub in RPC systems)
 it is loaded into the client’s address space when a client
binds to a distributed object
 tasks: a proxy marshals method invocation into messages
and unmarshals reply messages to return the result of the
method invocation to the client
 a server stub, called a skeleton, unmarshals messages and
marshals replies

81
common organization of a remote object with client-side proxy

82
 Binding a Client to an Object
 a process must first bind to an object before invoking its

methods, which results in a proxy being placed in the

process’s address space
 binding can be implicit (directly invoke methods using

only a reference to an object) or explicit (by calling a

special function)
 an object reference could contain

 network address of the machine where the object

resides
 endpoint of the server
 an identification of which object
 the protocol used
 ...

83
Distr_object* obj_ref; //Declare a systemwide object reference
obj_ref = …; //Initialize the reference to a distributed object
obj_refdo_something(); //Implicitly bind and invoke a method
 (a)
Distr_object obj_ref; //Declare a systemwide object reference
Local_object* obj_ptr; //Declare a pointer to local objects
obj_ref = …; //Initialize the reference to a distributed object
obj_ptr = bind(obj_ref); //Explicitly bind and obtain a pointer to the local proxy
obj_ptrdo_something(); //Invoke a method on the local proxy
 (b)
a) an example with implicit binding using only global references
b) an example with explicit binding using global and local references

84
 Parameter Passing
 there are two situations when invoking a method with
object reference as parameter; is the object local or
remote to the client?
 remote object: copy and pass the reference of the object
as a value parameter; this means the object is passed by
reference
 local object: a copy of the object is passed; this means the
object is passed by value

85
 the situation when passing an object by reference or by value

 two examples:
 DCE Remote Objects

 Java RMI

86
 4.3 Message Oriented Communication

 RPCs and RMIs are not adequate for all distributed system
applications
 the provision of access transparency may be good but
they have semantics that is not adequate for all
applications
 example problems
 they assume that the receiving side is running at the
time of communication
 a client is blocked until its request has been processed

87
Persistence and Synchronicity in Communication
 assume the communication system is organized as a
computer network shown below

general organization of a communication system in which hosts are connected

through a network
88
 communication can be
 persistent or transient
 asynchronous or synchronous
 persistent: a message that has been submitted for
transmission is stored by the communication system as long
as it takes to deliver it to the receiver
 e.g., email delivery, snail mail delivery

persistent communication of letters back in the days of the Pony Express

89
 transient: a message that has been submitted for
transmission is stored by the communication system only as
long as the sending and receiving applications are executing
 asynchronous: a sender continues immediately after it has
submitted its message for transmission
 synchronous: the sender is blocked until its message is
stored in a local buffer at the receiving host or delivered to the
receiver
 the different types of communication can be combined
 persistent asynchronous: e.g., email
 transient asynchronous: e.g., UDP, asynchronous RPC
 in general there are six possibilities

Persistent Transient

Asynchronous  

Synchronous  message-oriented; three forms

90
persistent asynchronous communication  persistent synchronous
communication

91
 transient asynchronous receipt-based transient synchronous communication
communication

 weakest form; the sender is

blocked until the message is
stored in a local buffer at the
receiving host

92
 delivery-based transient synchronous
communication at message delivery
 the sender is blocked until the
message is delivered to the
receiver for further processing;
e.g., asynchronous RPC
response-based transient synchronous
communication
 strongest form; the sender is
blocked until it receives a reply
message from the receiver
93
Message-Oriented Transient Communication
 many applications are built on top of the simple message-
oriented model offered by the transport layer
 standardizing the interface of the transport layer by
providing a set of primitives allows programmers to use
messaging protocols
 they also allow porting applications
 Berkley Sockets
 an example is the socket interface as used in Berkley
UNIX
 a socket is a communication endpoint to which an
application can write data that are to be sent over the
network, and from which incoming data can be read

94
Primitive Meaning
Create a new communication endpoint; also executed by
Socket
reserve resources to send and receive messages both
Attach a local address to a socket; e.g., IP
Bind
address with a known port number
executed by
Announce willingness to accept connections; for
Listen servers
connection-oriented communication
Accept Block caller until a connection request arrives
Actively attempt to establish a connection; the executed by
Connect
client is blocked until connection is set up clients
Send Send some data over the connection
executed by
Receive Receive some data over the connection
both
Close Release the connection
socket primitives for TCP/IP

95
connection-oriented communication pattern using sockets

96
The Message-Passing Interface (MPI)
 sockets were designed to communicate across networks
using general-purpose protocol stacks such as TCP/IP
 they were not designed for proprietary protocols developed
for high-speed interconnection networks; of course
portability will suffer
 MPI is designed for parallel applications and tailored for
transient communication
 MPI assumes communication takes place within a known
group of processes, where each group is assigned an
identifier (groupID)
 each process within a group is also assigned an identifier
(processID)
 a (groupID, processID) identifies the source or destination of
a message, and is used instead of a transport-level address

97
Primitive Meaning
Append outgoing message to a local send buffer; to support
MPI_bsend
transient asynchronous communication
Send a message and wait until copied to local or remote
MPI_send buffer (to support receipt-based transient synchronous
communication)
Send a message and wait until receipt starts (to support
MPI_ssend
delivery-based transient synchronous communication)
Send a message and wait for reply (to support response-
MPI_sendrecv
based transient synchronous communication)
Pass reference to outgoing message, and continue (a
MPI_isend
variant of MPI_send)
Pass reference to outgoing message, and wait until receipt
MPI_issend
starts (a variant of MPI_ssend)
MPI_recv Receive a message; block if there are none
MPI_irecv Check if there is an incoming message, but do not block

some of the most intuitive message-passing primitives of MPI

98
Message-Oriented Persistent Communication
 there are message-oriented middleware services, called
message-queuing systems or Message-Oriented Middleware
(MOM)
 they support persistent asynchronous communication
 they have intermediate-term storage capacity for messages,
without requiring the sender or the receiver to be active
during message transmission
 unlike Berkley sockets and MPI, message transfer may take
minutes instead of seconds or milliseconds
 Message-Queuing Model
 applications communicate by inserting messages in
specific queues
 it permits loosely-coupled communication
 the sender may or may not be running; similarly the
receiver may or may not be running, giving four possible
combinations
99
four combinations for loosely-coupled communications using queues
100
Primitive Meaning
Append a message to a specified queue; by the sender
Put
and is nonblocking
Block until the specified queue is nonempty, and remove
Get
the first message
Check a specified queue for messages, and remove the
Poll
first. Never block
Install a handler to be called when a message is put into
Notify
the specified queue; usually a daemon
basic interface to a queue in a message-queuing system

101
 General Architecture of a Message-Queuing System
 messages can be put only into queues that are local to the
sender (same machine or on a nearby machine on a LAN)
 such a queue is called the source queue
 messages can also be read only from local queues
 a message put into a local queue must contain the
specification of the destination queue; hence a message-
queuing system must maintain a mapping of queues to network
locations; like in DNS

the relationship between queue-level addressing and network-level addressing 102

 messages are managed by queue managers
 they generally interact with the application that sends and
receives messages
 some also serve as routers or relays, i.e., they forward
incoming messages to other queue managers
 however, each queue manager needs a copy of the queue-
to-location mapping, leading to network management
problems for large-scale queuing systems
 the solution is to use a few routers that know about the
network topology

103
the general organization of a message-queuing system with routers
104
 Message Brokers
 how can applications understand the messages they receive
 each receiver can not be made to understand message formats
of new applications
 hence, in a message-queuing system conversations are
handled by message brokers
 a message broker converts incoming messages to a format
that can be understood by the destination application based on
a set of rules

the general organization of a message broker in a message-queuing system

105
 4.4 Stream Oriented Communication
 until now, we focused on exchanging independent and complete
units of information
 time has no effect on correctness; a system can be slow or fast
 however, there are communications where time has a critical role
 Multimedia
 media
 storage, transmission, interchange, presentation,
representation and perception of different data types:
 text, graphics, images, voice, audio, video, animation, ...
 movie: video + audio + …
 multimedia: handling of a variety of representation media
 end user pull
 information overload and starvation
 technology push
 emerging technology to integrate media

106
 The Challenge
 new applications
 multimedia will be pervasive in 10 years (as graphics)
 storage and transmission
 e.g., 2 hours uncompressed HDTV (1920×1080) movie:
1.1 TB (1920×1080x3x25x60x60x2)
 videos are extremely large, even compressed
 continuous delivery
 e.g., 30 frames/s (NTSC), 25 frames/s (PAL) for video
 guaranteed Quality of Service
 admission control
 search
 can we look at 100… videos to find the proper one?

107
 Types of Media
 two types
 discrete media: text, executable code, graphics, images;
temporal relationships between data items are not
fundamental to correctly interpret the data
 continuous media: video, audio, animation; temporal
relationships between data items are fundamental to
correctly interpret the data
 a data stream is a sequence of data units and can be applied
to discrete as well as continuous media
 stream-oriented communication provides facilities for the
exchange of time-dependent information (continuous media)
such as audio and video streams

108
 timing in transmission modes
 asynchronous transmission mode: data items are transmitted
one after the other, but no timing constraints; e.g. text transfer
 synchronous transmission mode: a maximum end-to-end
delay defined for each data unit; it is possible that data can be
transmitted faster than the maximum delay, but not slower
 isochronous transmission mode: maximum and minimum
end-to-end delay are defined; also called bounded delay jitter;
applicable for distributed multimedia systems
 a continuous data stream can be simple or complex
 simple stream: consists of a single sequence of data; e.g.,
mono audio, video only
 complex stream: consists of several related simple streams
that must be synchronized; e.g., stereo audio, video
consisting of audio and video (may also contain subtitles,
translation to other languages, ...)

109
movie as a set of simple streams

110
 a stream can be considered as a virtual connection between a
source and a sink
 the source or the sink could be a process or a device

setting up a stream between two processes across a network

setting up a stream directly between two devices 111

 the data stream can also be multicasted to several receivers
 if devices and the underlying networks have different
capabilities, the stream may be filtered, generally called
adaptation

an example of multicasting a stream to several receivers

112
 Quality of Service (QoS)
 QoS requirements describe what is needed from the
underlying distributed system and network to ensure
acceptable delivery; e.g. viewing experience of a user
 for continuous data, the concerns are
 timeliness: data must be delivered in time

 volume: the required throughput must be met

 reliability: a given level of loss of data must not be

exceeded
 quality of perception; highly subjective

113
 QoS Dimensions
 timeliness dimensions
 latency (maximum delay between consecutive frames)

 start-up latency (maximum delay before starting a

presentation)
 jitter (delay variance)

 volume dimensions
 throughput in frames/sec or bits/sec or bytes/sec

 reliability dimensions
 MTBF (Mean Time Between Failure) of disks

 MTTR (Mean Time To Repair)

 error rates on the telecommunication lines

114
 QoS Requirements
 deterministic
 precise values or ranges

 e.g., latency must be between 45 and 55 ms

 probabilistic
 probability of the required QoS

 e.g., latency should be < 50 ms for 95% of the frames

 stochastic distributions
 e.g., frame arrival should follow normal distribution with

mean interval-time of 40 ms and 5 ms variance

 classes
 e.g., guaranteed and best effort

115
 QoS Management
 can be static or dynamic
 Static QoS Management Functions
 specification

 e.g., deterministic range for timeliness, volume and

reliability categories
 negotiation

 the application may accept lower level of QoS for

lower cost
 admission control

 if this test is passed, the system has to guarantee the

promised QoS
 resource reservation

 may be necessary to provide guaranteed QoS

116
 Dynamic QoS Management Functions
 monitoring
 notices deviation from QoS level
 at a certain level of granularity (e.g., every 100 ms)
 policing
 detect participants not keeping themselves to the contract
 e.g., source sends faster than negotiated (e.g., 25 fps)
 maintenance
 sustaining the negotiated QoS
 e.g., the system requires more resources
 renegotiation
 client tries to adapt – may be can accept lower QoS

117
 QoS requirements can be specified using flow specification
containing bandwidth requirements, transmission rates,
delays, ...
 e.g. by Partridge (1992)
 it uses the token bucket algorithm which specifies how the

stream will shape its network traffic (in fact the leaky
bucket, as used in networking)
 the idea is to shape bursty traffic into fixed-rate traffic by

averaging the data rate

 packets may be dropped if the bucket is full

 the input rate may vary, but the output rate remains

constant

118
the principle of a token bucket algorithm

119
 Specifying QoS
Characteristics of the Input
 Maximum data unit size (bytes)
 Token bucket rate (bytes/sec)
 Token bucket size (bytes)
 Maximum transmission rate
(bytes/sec)
a flow specification
Service Required
 Loss sensitivity (bytes)
 Loss interval (sec)
 the above two specify the maximum
acceptable loss rate, e.g. 1 byte per
minute
 Burst loss sensitivity (data units);
how many consecutive data units
may be lost
 Minimum delay noticed (sec); how
long the network can delay the
delivery
 Maximum delay variation (sec);
maximum tolerated jitter
 Quality of guarantee; continue or
don’t establish a stream if the
communication system can not
provide the required service
120
  problem in flow specification
 an application may not know its requirements

 how can a user (human) specify quality using the various

parameters? usually very difficult

 may be provide defaults for various streams as high,

medium, low quality

 Setting up a Stream
 resources such as bandwidth, buffers, processing power
must be reserved once a flow specification is made
 on such protocol is RSVP - Resource reSerVation Protocol
 it is a transport-level protocol for enabling resource
reservation in network routers

121
the basic organization of RSVP for resource reservation in a distributed system

122
 Stream Synchronization
 how to maintain temporal relations between streams, e.g., lip
synchronization
 two approaches
1. explicitly by operating on the data units of simple
streams; the responsibility of the application

the principle of explicit synchronization on the level of data units

123
2. through a multimedia middleware that offers a collection of
interfaces for controlling audio and video streams as well as
devices such as monitors, cameras, microphones, ...

the principle of synchronization as supported by high-level interfaces

124