CS-423 Parallel and Distributed Computing

Lecture 1
Introduction to Parallel and Distributed Computing
Agenda
• What is Computing?
• Parallel Computing
• Types of Parallelism
• Applications, Advantages, Limitations, and Future of Parallelism
• Distributed Computing
• Distributed Computing Models
• Types of Transparency
• Advantages, Disadvantages of Distributed Computing
• Difference between Parallel and Distributed Computing
• Which is Better?
What is Computing?
• Computing is the process to complete a given goal-oriented task by
using computer technology.
• Computing may encompass the design and development of software
and hardware systems for a broad range of purposes - often
structuring, processing and managing any kind of information.
• Computing has also been defined as a branch of engineering science
that deals with the systematic study of algorithmic processes, which
are used to describe and transform information.
• Serial Computing
• Computer software was written conventionally for serial computing.
This meant that to solve a problem, an algorithm divides the problem
into smaller instructions. These discrete instructions are then
executed on the Central Processing Unit of a computer one by one.
Only after one instruction is finished, next one starts.
1. In this, a problem statement is broken into discrete instructions. 
2. Then the instructions are executed one by one. 
3. Only one instruction is executed at any moment of time.
Parallel Computing
• Parallel computing is the process of performing computational tasks
 across multiple processors at once to improve computing speed and
efficiency. It divides tasks into sub-tasks and executes them
simultaneously through different processors.
• It is the use of multiple processing elements simultaneously for
solving any problem. Problems are broken down into instructions and
are solved concurrently as each resource that has been applied to work
is working at the same time. 
Types of Parallelism
1. Bit Level Parallelism
• It is the form of parallel computing which is based on the increasing
processor’s word size. It reduces the number of instructions that the
system must execute in order to perform a task on large-sized data. 
• Example: 
• Consider a scenario where an 8-bit processor must compute the sum of two
16-bit integers. It must first sum up the 8 lower-order bits, then add the 8
higher-order bits, thus requiring two instructions to perform the operation. A
16-bit processor can perform the operation with just one instruction.
Types of Parallelism • Multiple Functional Units
• Instructions issued in parallel
2. Instruction Level Parallelism • Parallel execution of instructions
• Multiple functional units within the
• A processor can only address less CPU are required
than one instruction for each clock
cycle phase. These instructions can
be re-ordered and grouped which are
later on executed concurrently
without affecting the result of the
program. This is called instruction-
level parallelism.
• Employs a stream of instructions to
allow processors to execute more
than one instruction per clock cycle
Types of Parallelism
3. Task Level Parallelism
• Task parallelism employs the decomposition of a
task into subtasks and then allocating each of the
subtasks for execution. The processors perform
the execution of sub-tasks concurrently.
Types of Parallelism
4. Data Level Parallelism
• Instructions from a single stream operate concurrently on several data – Limited by
non-regular data manipulation patterns and by memory bandwidth.
• Performing same operation on different pieces of data
Advantages of Parallelism
• It saves time and money as many resources working together will
reduce the time and cut potential costs. 
• It can be impractical to solve larger problems on Serial Computing. 
• It can take advantage of non-local resources when the local resources
are finite. 
• Serial Computing ‘wastes’ the potential computing power, thus
Parallel Computing makes better work of the hardware. 
Limitations of Parallelism
• It addresses such as communication and synchronization between
multiple sub-tasks and processes which is difficult to achieve.
• The algorithms must be managed in such a way that they can be
handled in a parallel mechanism.
• The algorithms or programs must have low coupling and high
cohesion. But it’s difficult to create such programs.
• More technically skilled and expert programmers can code a
parallelism-based program well.
Applications of Parallelism
• Databases and Data mining.
• Real-time simulation of systems.
• Science and Engineering.
• Advanced graphics, augmented reality, and virtual reality.
Future of Parallelism
•  The computational graph has undergone a great transition from serial
computing to parallel computing. Tech giant such as Intel has already
taken a step towards parallel computing by employing multicore
processors. Parallel computation will revolutionize the way computers
work in the future, for the better good. With all the world connecting
to each other even more than before, Parallel Computing does a better
role in helping us stay that way. With faster networks, distributed
systems, and multi-processor computers, it becomes even more
necessary.
Distributed Computing
• Distributed computing is the process of connecting multiple computers via
a local network or wide area network so that they can act together as a
single ultra-powerful computer capable of performing computations that no
single computer within the network would be able to perform on its own.
• A distributed system contains multiple nodes that are physically separate
but linked together using the network. All the nodes in this system
communicate with each other and handle processes in tandem.
• In distributed computing we have multiple autonomous computers which
seems to the user as single system. In distributed systems there is no
shared memory and computers communicate with each other through
message passing. In distributed computing a single task is divided among
different computers.
Distributed Computing Models
1. Cluster Computing
• Cluster computing is a collection of tightly coupled connected
computers that work together so that they act as a single entity. The
connected computers execute operations all together thus creating
the idea of a single system.
• The clusters are generally connected through fast 
local area networks (LANs)
• All nodes have same types of hardware and software, particularly the
operating system.
• Made up of computers of homogenous hardware configurations.
Distributed Computing Models
2. Grid Computing
• Grid Computing refers to distributed computing, in which a group of
computers from multiple locations are connected with each other to
achieve a common objective.
• These computer resources are heterogeneous and geographically
dispersed.
• Grid Computing breaks complex task into smaller pieces, which are
distributed to CPUs that reside within the grid.
• Made up of computers of heterogeneous hardware configurations.
• Grid Computing is Application oriented Architecture.
• User does not have to pay for any usage.
Distributed Computing Models
3. Cloud Computing
• Cloud computing is the delivery of computing services—
including servers, storage, databases, networking, software,
analytics, and intelligence—over the Internet (“the cloud”) to
offer faster innovation, flexible resources, and economies of
scale.
• More flexible.
• The users pay for what they use.
• Highly scalable.
• Cloud Computing is Service Oriented Architecture.
Types of Transparency
• Access
• Hides the way in which resources are accessed and the differences in data
platform.
• Location
• Hides where resources are located.
• Technology
• Hides different technologies such as programming language and OS from user.
• Migration / Relocation
• Hide resources that may be moved to another location which are in use.
Types of Transparency
• Replication
• Hide resources that may be copied at several location.
• Concurrency
• Hide resources that may be shared with other users.
• Failure
• Hides failure and recovery of resources from user.
• Persistence
• Hides whether a resource ( software ) is in memory or disk.
Advantages of Distributed Computing
• Resource sharing
• Sharing of hardware and software resources.
• Openness
• Flexibility of using hardware and software of different vendors.
• Concurrency
• Concurrent processing to enhance performance.
• Scalability
• Increased throughput by adding new resources.
• Fault tolerance
• The ability to continue in operation after a fault has occurred.
Disadvantages of Distributed Computing
• Complexity
• They are more complex than centralized systems.
• Security
• More susceptible to external attack.
• Manageability
• More effort required for system management.
• Unpredictability
• Unpredictable responses depending on the system organization and network
load.
 Parallel Computing Distributed Computing
Number of Computers Typically requires one computer with multiple Involves several autonomous (and often
Required processors. geographically separate and/or distant) computer
systems working on divided tasks.

Scalability Less scalable than distributed computing A distributed computing system can always scale
systems because the memory of a single with additional computers.
computer can only handle so many
processors at once.
Memory All processors share the same memory, and Have their own memory and processors.
the processors communicate with each other
with the help of this shared memory.

Synchronization All processors share a single master clock for Use synchronization algorithms.
synchronization

Usage Parallel computing is used to increase Distributed computing is used to share resources
computer performance and for scientific and improve scalability.
computing.
Parallel Computing
• Many operations are performed
simultaneously
• Single computer is required
• Multiple processors perform
multiple operations
• It may have shared or distributed
memory
• Processors communicate with each
other through bus
• Improves the system performance
Distributed Computing
• System components are located at
different locations
• Uses multiple computers
• Multiple computers perform multiple
operations
• It have only distributed memory
• Computer communicate with each
other through message passing.
• Improves system scalability, fault
tolerance and resource sharing
capabilities
Which is Better?
• It’s hard to say which is “better”—parallel or distributed computing—
because it depends on the use case (see section above). If you need
pure computational power and work in a scientific or other type of
highly analytics-based field, then you’re probably better off with
parallel computing. If you need scalability and resilience and can
afford to support and maintain a computer network, then you’re
probably better off with distributed computing.