Communication

Contents
• Introduction to Distributed Systems
• Characterization of Distributed Systems
• Distributed Shared Memory Approaches
• Message Passing
• Programming Using the Message-
Passing Paradigm
– Group Communication
• Case Study (RPC and Java RMI)

SCOPE, VIT Chennai

Introduction to Distributed
Systems

SCOPE, VIT Chennai

 Examples of distributed
systems
• NFS
• The internet
• DNS
• Amazon AWS
• Google Cloud Platform

SCOPE, VIT Chennai

 Introduction
• Networks of computers are
everywhere!
– Mobile phone networks
– Corporate networks
– Factory networks
– Campus networks
– In-car networks
– Internet of Things (IoT)
– On board networks in
planes and trains

SCOPE, VIT Chennai

• Mobile - GSM/3G networks, cell towers connect to
backbones
• Corporate - LAN, consisting of PC’s, laptops, servers,
storage, printing
• Factory - Machines (drills, stamps, robots) communicate
over network
• Campus - Like CSSE Dept / Unimelb. Wired/wireless,
clients, servers
• Home - Laptops, PC’s, Xbox/Playstation, Set top boxes
• In-car - Network of sensors (monitoring engine, breaks,
tires etc)

SCOPE, VIT Chennai

Characteristics of
networked/distri
buted computing
systems and
applications

SCOPE, VIT Chennai

 What is Distributed System?
• Collection of entities (along with some
memory) to work towards a common goal
• Distribution is Transparent to the users and
appears as one local machine
A distributed system as one in which hardware or software
components located at networked computers communicate and
coordinate their actions only by message passing.-[Coulouris]

A distributed system is a collection of independent computers that

appear to the users of the system as a single computer.-[Tanenbaum]

SCOPE, VIT Chennai

 Generic Components of
Distributed System
• Computing system
– Mostly heterogeneous
– Multicore, general purpose / embedded
processing, SIMD or MIMD
• Communication
– LAN, Internet

SCOPE, VIT Chennai

 How is distributed system different from
Networked system?
• In networked systems, user is aware
of individual node, their location and
functionality
• Distributed system appears as a
single machine to the users

SCOPE, VIT Chennai

Networks Vs Distributed Systems
• Networks: A media for interconnecting local
and wide area computers and exchange
messages based on protocols. Network entities
are visible and they are explicitly addressed (IP
address).
• Distributed System: existence of multiple Distributed Systems
autonomous computers is transparent
• However, Computer Networks
– many problems (e.g., openness, reliability)
in common, but at different levels.
• Networks focuses on packets, routing, etc.,
whereas distributed systems focus on
applications.
• Every distributed system relies on services
provided by a computer network

SCOPE, VIT Chennai

Reasons for Distributed Systems
• Functional Separation:
– Existence of computers with different capabilities and purposes:
• Clients and Servers
• Data collection and data processing
• Inherent distribution:
– Information:
• Different information is created and maintained by
different people (e.g., Web pages)
– People
• Computer supported collaborative work (virtual teams,
engineering, virtual surgery)
– Retail store and inventory systems for supermarket chains
(e.g., Coles, Woolworths)

SCOPE, VIT Chennai

Reasons for Distributed Systems
• Power imbalance and load variation:
– Distribute computational load
among different computers.
• Reliability:
– Long term preservation and data
backup (replication) at different
locations.
• Economies:
– Sharing a printer by many users and
reduce the cost of ownership.
– Building a supercomputer out of a
network of computers.
SCOPE, VIT Chennai
Centralized System

SCOPE, VIT Chennai

Networked System

SCOPE, VIT Chennai

Distributed System

SCOPE, VIT Chennai

Consequences of Distributed
Systems
• Computers in distributed systems may be on separate
continents, in the same building, or the same room. DSs
have the following consequences:
– Concurrency – each system is autonomous.
• Carry out tasks independently
• Tasks coordinate their actions by exchanging messages.
– Heterogeneity
– No global clock
– Independent Failures

SCOPE, VIT Chennai

Characterization of
Distributed Systems

SCOPE, VIT Chennai

 Significant Characteristics
• Parallel activities
– Autonomous components executing concurrent tasks
• Communication via message passing
– No shared memory
• Resource sharing
– Printer, database, other services
• No global state
– No single process can have knowledge of the current global
state of the system
• No global clock
– Only limited precision for processes to synchronize their clocks

SCOPE, VIT Chennai

Goals of Distributed Systems
• Connecting Users and Resources
• Transparency
• Openness
• Scalability
• Enhanced Availability

SCOPE, VIT Chennai

• Distributed computing as a utility
– Cluster computing
– Grid computing
– Cloud computing

SCOPE, VIT Chennai

 Typical Distributed System
Design Goals
• Heterogeneity
• Availability
• Scalability
• Transparency
• Concurrency
• Efficiency
• Robustness

SCOPE, VIT Chennai

 Other Ideal Characteristics
• Efficient Resource Management: Middle ware should
manage efficiently heterogeneous systems
• Effective Synchronization: Issues in the absence of
global clock should be effectively managed
• Security
• Complexity
• Highly variable bandwidth and latency
• Openness – Distributed system should be extensible
• Transparency
• Quality of Service

SCOPE, VIT Chennai

 Examples of Distributed
Systems
• They (DS) are based on familiar and
widely used computer networks:
– Internet
– Intranets, and
– Wireless networks
• Example DS and its Applications:
– Web (and many of its applications like Online bookshop)
– Data Centers and Clouds
– Wide area storage systems
– Banking Systems
– User-level communication (Facebook, Zoom)

SCOPE, VIT Chennai

Selected application domains and
associated networked applications

SCOPE, VIT Chennai

 A typical portion of the Internet and its services:
Multimedia services providing access to music, radio, TV channels,
and video conferencing supporting several users

Intranet %
%
% ISP
%

backbone

satellite link
desktop computer:
server:
network link:

SCOPE, VIT Chennai

 A typical Intranet
A portion of Internet that is separately administered & supports internal sharing
of resources (file/storage systems and printers)

Desktop
computers
Print and other servers email server

Local area
Web server network

email server
print
File server
other servers
the rest of
the Internet
router/firewall
SCOPE, VIT Chennai
 Internet

WAP (Wireless Home Intranet

Host Intranet Wireless LAN App Protocol)
gateway

Printer
Mobile Supports continued access to
phone
Home intranet resources via
Camera Laptop Host site wireless and provision to utilise
resources (e.g., printers) that
are conveniently located
(location-aware computing).

SCOPE, VIT Chennai

• mobile computing
• location/contextaware computing
• ubiquitous computing
• spontaneous interoperation
• service discovery

SCOPE, VIT Chennai

Resource sharing and the Web: open protocols,
scalable servers, and pluggable browsers

http://www.google.com/search?q=Buyya
www.google.com

Web servers Browsers

www.cdk5.net Internet
Internet http://www.cdk5.net/

www.w3c.org

File system of http://www.w3c.org/Protocols/Activity.html

www.w3c.org Protocols User types in a URL into a web browser, and many
things happen:
1. The server that hosts the page is located
Activity.html 2. The protocol is identified (in this case, HTTP)
3. The relevant service is called upon (e.g. the
webserver) to satsify request
4. The particular page that is requested is located and
returned to user

SCOPE, VIT Chennai

Trends in Distributed System
• Emergence of pervasive technologies
– almost always connected
• Increased Multimedia support and
use
• Distributed systems as utility 
Cluster, Cloud, Grid

SCOPE, VIT Chennai

 Business Example and
Challenges
• Online bookstore (e.g. in World Wide
Web)
– Customers can connect their computer to
your computer (web server):
• Browse your inventory
• Place orders
•…

This example has been adapted from Torbin Weis, Berlin University of Technology

SCOPE, VIT Chennai

 Business Example – Challenges
• What if
– Your customer uses a completely different hardware? (PC, MAC,
iPad, Mobile…)
– … a different operating system? (Windows, Unix,…)
– … a different way of representing data? (ASCII, EBCDIC,…)
– Heterogeneity
• Or
– You want to move your business and computers to the
Caribbean (because of the weather or low tax)?
– Your client moves to the Caribbean (more likely)?
– Distribution transparency

SCOPE, VIT Chennai

 Business Example –
Challenges
• What if
– Two customers want to order the same item at
the same time?
– Concurrency
• Or
– The database with your inventory information
crashes?
– Your customer’s computer crashes in the
middle of an order?
– Fault tolerance

SCOPE, VIT Chennai

 Business Example – Challenges
• What if
– Someone tries to break into your
system to steal data?
– … sniffs for information?
– … your customer orders something and doesn’t
accept the delivery saying he didn’t?
– Security
• Or
– You are so successful that millions of people are
visiting your online store at the same time?
– Scalability

SCOPE, VIT Chennai

 Business Example –
Challenges
• When building the system…
– Do you want to write the whole software on
your own (network, database,…)?
– What about updates, new technologies?
– Reuse and Openness (Standards)

SCOPE, VIT Chennai

 Overview Challenges
• Heterogeneity
– Heterogeneous components must be able to interoperate
• Distribution transparency
– Distribution should be hidden from the user as much as possible
• Fault tolerance
– Failure of a component (partial failure) should not result in
failure of the whole system
• Scalability
– System should work efficiently with an increasing number of
users
– System performance should increase with inclusion of additional
resources

SCOPE, VIT Chennai

 Overview Challenges
• Concurrency
– Shared access to resources must be possible
• Openness
– Interfaces should be publicly available to
ease inclusion of new components
• Security
– The system should only be used in the way
intended

SCOPE, VIT Chennai

 Heterogeneity
• Heterogeneous components must be
able to interoperate across different:
– Operating systems
– Hardware architectures
– Communication architectures
– Programming languages
– Software interfaces
– Security measures
– Information representation

SCOPE, VIT Chennai

 Distribution Transparency
• ISO Reference Model for Open Distributed Processing
(ODP) identifies the following forms of transparencies:
• Access transparency
– Access to local or remote resources is identical
– E.g. Network File System / Dropbox
• Location transparency
– Access without knowledge of location
– E.g. separation of domain name from
machine address.
• Failure transparency
– Tasks can be completed despite failures
– E.g. message retransmission, failure of a Web server node
should not bring down the website
SCOPE, VIT Chennai
 Distribution Transparency
• Replication transparency
– Access to replicated resources as if there was just one. And
provide enhanced reliability and performance without
knowledge of the replicas by users or application programmers.
• Migration (mobility/relocation) transparency
– Allow the movement of resources and clients within a system
without affecting the operation of users or applications.
– E.g. switching from one name server to another at runtime;
migration of an agent/process from one node to another.

SCOPE, VIT Chennai

 Distribution Transparency
• Concurrency transparency
– A process should not notice that there are other sharing the same
resources
• Performance transparency:
– Allows the system to be reconfigured to improve performance as loads
vary
– E.g., dynamic addition/deletion of components, switching from linear
structures to hierarchical structures when the number of users increase
• Scaling transparency:
– Allows the system and applications to expand in scale without changes in
the system structure or the application algorithms.
• Application level transparencies:
– Persistence transparency
• Masks the deactivation and reactivation of an object
– Transaction transparency
• Hides the coordination required to satisfy the transactional properties of
operations

SCOPE, VIT Chennai

 Fault Tolerance
• Failure: an offered service no longer complies with its
specification (e.g., no longer available or very slow to
be usable)
• Fault: cause of a failure (e.g. crash of a component)
• Fault tolerance: no failure despite faults
i.e., programmed to handle failures
and hides them from users.

SCOPE, VIT Chennai

Fault Tolerance Mechanisms
• Fault detection
– Checksums, heartbeat, …
• Fault masking
– Retransmission of corrupted messages,
redundancy, …
• Fault toleration
– Exception handling, timeouts,…
• Fault recovery
– Rollback mechanisms,…

SCOPE, VIT Chennai

 Scalability
• System should work efficiently at
many different scales, ranging from a
small Intranet to the Internet
• Remains effective when there is a
significant increase in the number of
resources and the number of users

SCOPE, VIT Chennai

 Scalability
• Challenges of designing scalable distributed systems:
– Cost of physical resources
• Cost should linearly increase with system size
– Performance Loss
• For example, in hierarchically structure data, search
performance loss due to data growth should not be
beyond O(log n), where n is the size of data
– Preventing software resources running out:
– Avoiding performance bottlenecks:
• Use of decentralized algorithms

SCOPE, VIT Chennai

 Concurrency
• Provide and manage concurrent
access to shared resources:
– Fair scheduling
– Preserve dependencies (e.g. distributed
transactions -- buy a book using Credit card,
make sure user has sufficient funds prior to
finalizing order )
– Avoid deadlocks

SCOPE, VIT Chennai

 Openness and Interoperability
• An open system –key interfaces need to be
published!
• An open distributed system has:
– uniform communication mechanism
– published interfaces to shared resources
• Open DS - heterogeneous hardware and software,
possibly from different vendors, but conformance of
each component to published standard must be
tested and verified for the system to work correctly
• Open spec/standard developers - communities:
• ANSI, IETF, W3C, ISO, IEEE, OMG, Trade associations,...

SCOPE, VIT Chennai

 Security
• Resources are accessible to authorized users and used in
the way they are intended
• Confidentiality
– Protection against disclosure to unauthorized individual
information
– E.g. ACLs (access control lists) to provide authorized access to
information
• Integrity
– Protection against alteration or corruption
– E.g. changing the account number or amount value in a money
order

SCOPE, VIT Chennai

 Security
• Availability
– Protection against interference targeting
access to the resources.
– E.g. denial of service (DoS, DDoS) attacks
• Non-repudiation
– Proof of sending / receiving
an information
– E.g. digital signature

SCOPE, VIT Chennai

 Security Mechanisms
• Encryption
– E.g. Blowfish, RSA
• Authentication
– E.g. password, public key authentication
• Authorization
– E.g. access control lists

SCOPE, VIT Chennai

 Summary of DS
• Distributed Systems are everywhere
• Internet enables users throughout the world to access its
(application) services from anywhere
• Resource sharing is the main motivating factor for
constructing distributed systems
• Construction of DS produces many challenges:
– Heterogeneity, Openness, Security, Scalability, Failure handling,
Concurrency, and Transparency
• Distributed systems enable globalization:
– Community (Virtual teams, organizations, social networks)
– Science (e-Science)
– Business (..e-Banking..)
– Entertainment (YouTube, e-Friends)

SCOPE, VIT Chennai

 References
• https://mtjagtap.files.wordpress.com
/2018/08/slides10-2.pdf
• http://www.cloudbus.org/652/Lectu
reSlides.html
• Coulouris, G. (2012). Distributed
systems. Boston: Addison-Wesley

SCOPE, VIT Chennai

Distributed Shared Memory
(DSM) Approaches

SCOPE, VIT Chennai

 How to share data between
distributed processor nodes?
• Distributed Shared Memory
– It’s the abstraction to share data between
distributed computers
• It enables access across multiple
distributed physical memory to be
accessed as single shared memory

SCOPE, VIT Chennai

DSM Abstraction

SCOPE, VIT Chennai

 Working of DSM
• DSM enables easy access of individual
computer shared data items
• DSM run time support sends updates as
messages between computers
• Each computer has a local copy of
recently accessed data items stored in
DSM

SCOPE, VIT Chennai

 …Continued…
• Shared memory request for a non-local
piece of data is raised
• Single copy of data fetched and given to
the requested system
• If multiple machines access the data at
the same time, synchronization
primitive like semaphore is used to
handle the situation
SCOPE, VIT Chennai
 …Continued
• Read(shared variable)

• Write (data, shared variable)

SCOPE, VIT Chennai

Issues related to DSM Semantics
• Structure and granularity – data shared at the bit,
word or page level
• Consistency – If multiple requests for a single data
and each machine tries to update it, consistency
should be maintained. This involves cache coherence
like solution
• Heterogeneity – Accommodating different data
representations of different machines, languages and
OS
• Scalability – Bus latency, Increased broadcast
messages

SCOPE, VIT Chennai

DSM Versus Message Passing
• No marshaling of messages in DSM where as
messages are marshalled and unmarshalled in
message passing
• Synchronization in DSM is by constructs like
semaphore/locks where as in message passing it is
by message passing primitives
• DSM processes can communicate with non-
overlapping lifetimes where as in message passing
processes communicate at the same time
• Efficiency in DSM heavily depends on the pattern of
Data access by multiple machines where as the same
is not true for message passing

SCOPE, VIT Chennai

 DSM features
• Space- uncoupled
• Time – uncoupled
• State based service
• Used for parallel and distributed
computation
• Limited scalability
• Not associative

SCOPE, VIT Chennai

 Distributed Resource
Management:
Distributed Shared
Memory
Courtesy: CS-550: Distributed Shared
Memory [SiS ’94]
SCOPE, VIT Chennai
 DSM
• What
- The distributed shared memory (DSM)
implements the shared memory model in
distributed systems, which have no physical
shared memory
- The shared memory model provides a virtual
address space shared between all nodes
- To overcome the high cost of communication in
distributed systems, DSM systems move data to
the location of access

SCOPE, VIT Chennai

 DSM
• How:
- Data moves between main memory and secondary
memory (within a node) and between main
memories of different nodes
- Each data object is owned by a node
- Initial owner is the node that created object
- Ownership can change as object moves from node to
node
- When a process accesses data in the shared address
space, the mapping manager maps shared memory
address to physical memory (local or remote)
SCOPE, VIT Chennai
 DSM
NODE 1 NODE 2 NODE 3

Memory Memory Memory

Mapping Mapping Mapping

Manager Manager Manager

Shared Memory

SCOPE, VIT Chennai

 Advantages of distributed
shared memory (DSM)
• Data sharing is implicit, hiding data movement (as opposed to
‘Send’/‘Receive’ in message passing model)
• Passing data structures containing pointers is easier (in message passing
model data moves between different address spaces)
• Moving entire object to user takes advantage of locality difference
• Less expensive to build than tightly coupled multiprocessor system: off-the-
shelf hardware, no expensive interface to shared physical memory
• Very large total physical memory for all nodes: Large programs can run more
efficiently
• No serial access to common bus for shared physical memory like in
multiprocessor systems
• Programs written for shared memory multiprocessors can be run on DSM
systems with minimum changes

SCOPE, VIT Chennai

Algorithms for implementing
DSM
• Issues
- How to keep track of the location of
remote data
- How to minimize communication
overhead when accessing remote data
- How to access concurrently remote
data at several nodes

SCOPE, VIT Chennai

The Central Server Algorithm
• Central server maintains all shared data
– Read request: returns data item
– Write request: updates data and returns acknowledgement message
• Implementation
– A timeout is used to resend a request if acknowledgment fails
– Associated sequence numbers can be used to detect duplicate write
requests
– If an application’s request to access shared data fails repeatedly, a
failure condition is sent to the application
• Issues: performance and reliability
• Possible solutions
– Partition shared data between several servers
– Use a mapping function to distribute/locate data

SCOPE, VIT Chennai

The Central Server Algorithm

SCOPE, VIT Chennai

 The Migration Algorithm
• Operation
– Ship (migrate) entire data object (page, block) containing data
item to requesting location
– Allow only one node to access a shared data at a time
• Advantages
– Takes advantage of the locality of reference
– DSM can be integrated with VM at each node

SCOPE, VIT Chennai

 Migration Algorithm
• To locate a remote data object:
– Use a location server
– Maintain hints at each node
– Broadcast query
• Issues
– Only one node can access a data object at a
time
– Thrashing can occur: to minimize it, set
minimum time data object resides at a node

SCOPE, VIT Chennai

The Migration Algorithm

SCOPE, VIT Chennai

 The Read-Replication
Algorithm
• Replicates data objects to multiple nodes
• DSM keeps track of location of data objects
• Multiple nodes can have read access or one node write
access (multiple readers-one writer protocol)
• After a write, all copies are invalidated or updated
• DSM has to keep track of locations of all copies of data
objects. Examples of implementations:
– IVY: owner node of data object knows all nodes that have copies
– PLUS: distributed linked-list tracks all nodes that have copies
• Advantage
– The read-replication can lead to substantial performance
improvements if the ratio of reads to writes is large

SCOPE, VIT Chennai

The Read-Replication
Algorithm

SCOPE, VIT Chennai

 The Full–Replication Algorithm

• Extension of read-replication algorithm: multiple

nodes can read and multiple nodes can write
(multiple-readers, multiple-writers protocol)
• Issue: consistency of data for multiple writers
• Solution: use of gap-free sequencer
– All writes sent to sequencer
– Sequencer assigns sequence number and sends write request to
all sites that have copies
– Each node performs writes according to sequence numbers
– A gap in sequence numbers indicates a missing write request:
node asks for retransmission of missing write requests

SCOPE, VIT Chennai

The Full–Replication Algorithm

SCOPE, VIT Chennai

 Memory coherence
• DSM are based on
- Replicated shared data objects
- Concurrent access of data objects at many nodes
• Coherent memory: when value returned by read operation is the
expected value (e.g., value of most recent write)
• Mechanism that control/synchronizes accesses is needed to
maintain memory coherence
• Sequential consistency: A system is sequentially consistent if
- The result of any execution of operations of all processors is the same as if
they were executed in sequential order, and
- The operations of each processor appear in this sequence in the order
specified by its program
• General consistency:
- All copies of a memory location (replicas) eventually contain same data
when all writes issued by every processor have completed

SCOPE, VIT Chennai

 Memory coherence (Cont.)
• Processor consistency:
- Operations issued by a processor are performed in the
order they are issued
- Operations issued by several processors may not be
performed in the same order (e.g. simultaneous reads of
same location by different processors may yields different
results)
• Weak consistency:
- Memory is consistent only (immediately) after a
synchronization operation
- A regular data access can be performed only after all
previous synchronization accesses have completed

SCOPE, VIT Chennai

 Memory coherence (Cont.)
• Release consistency:
- Synchronization operations must be consistent
which each other only within a processor
- Synchronization operations: Acquire (i.e. lock),
Release (i.e. unlock)
- Sequence: Acquire
Regular access
Release

SCOPE, VIT Chennai

 Coherence Protocols
• Issues
- How do we ensure that all replicas have the
same information
- How do we ensure that nodes do not access
stale data

SCOPE, VIT Chennai

 Write-invalidate protocol
• A write to shared data invalidates all copies except one
before write executes
• Invalidated copies are no longer accessible
• Advantage: good performance for
– Many updates between reads
– Per node locality of reference
• Disadvantage
– Invalidations sent to all nodes that have copies
– Inefficient if many nodes access same object
• Examples: most DSM systems: IVY, Clouds, Dash,
Memnet, Mermaid, and Mirage

SCOPE, VIT Chennai

 Write-update protocol

• A write to shared data causes all

copies to be updated (new value
sent, instead of invalidation)
• More difficult to implement

SCOPE, VIT Chennai

 Design issues
• Granularity: size of shared memory unit
- If DSM page size is a multiple of the local virtual memory (VM)
management page size (supported by hardware), then DSM can be
integrated with VM, i.e. use the VM page handling
- Advantages vs. disadvantages of using a large page size:
- (+) Exploit locality of reference
- (+) Less overhead in page transport
- (-) More contention for page by many processes
- Advantages vs. disadvantages of using a small page size
- (+) Less contention
- (+) Less false sharing (page contains two items, not shared but needed by two
processes)
- (-) More page traffic
- Examples
• PLUS: page size 4 Kbytes, unit of memory access is 32-bit word
• Clouds, Munin: object is unit of shared data structure

SCOPE, VIT Chennai

 Design issues (cont.)
• Page replacement
- Replacement algorithm (e.g. LRU) must take into account page access
modes: shared, private, read-only, writable
- Example: LRU with access modes
• Private (local) pages to be replaced before shared ones
• Private pages swapped to disk
• Shared pages sent over network to owner
• Read-only pages may be discarded (owners have a copy)

SCOPE, VIT Chennai