Parallel Computing Landscape

(CS 526)

Muhammad Yasir,

Department of Computer Science,

The University of Lahore,
 Outlin
• Introduction of
e
parallel and distributed computing
• Moore’s Law
• Multi-cores
• Moore’ Law in the era of multi-cores
• Flynn’s Taxonomy: SISD, SIMD, MISD, MIMD
• Hardware Architectures:
– Multi-core processors (Heterogeneous multi-cores i.e., GPUs)
– Multi-processors System (shared Memory)
– Multi-processor System (Distributed Memory)
• Parallel Architectures: Classification of MIMD (NUMA, UMA,
COMA, CC-NUMA)
– Cluster
– Grid Computing
– Cloud and Fog Computing
 Parallel computing?
• What is parallel computing?

– Use of multiple processors or computers working

together on a common task
– Each processor works on its section of
the problem
– Processors may exchange information
Parallel computing?
 Parallel
Computing?
• Parallel computing: using multiple processors in
parallel to solve problems more quickly than with
a single processor

• Examples of parallel machines:

– A cluster computer that contains multiple PCs
combined together with a high speed network
– A shared memory multiprocessor (Symmetric Multi-
Processor) by connecting multiple processors to a
single memory system
– A Chip Multi-Processor (CMP) contains multiple
processors (called cores) on a single chip
 Why do parallel computing?
1. Limits of single CPU computing:
• Performance
• Available memory, etc.

2. To solve large problems:

• Problems that don’t fit on a single CPU
• Problems that can’t be solved in a
reasonable time (weather forecast, etc.)
 Why do parallel computing?
3. Faster Results
• Time-critical problems, can be solved faster

4. ALL Computers are parallel these days:

• Desktops
• Laptops
• Even smart-phones has multiple cores
• Etc.
 Single Processor Systems
– From mid-1980s until 2004 computers got
faster:
• More number of transistors were
packed
every year.
• Size of transistors continued to get
smaller
• Resulted in faster processors (frequency
scaling method  more GHzs means more
performance)
 Moore’s
Law
• Increased density of components on a chip
• Gordon Moore – co-founder of Intel
The number of transistors on a chip will double every
year

• Since the 1970’s development has slowed a little:

– Number of transistors doubled every 18 months
• Higher packing density means shorter
electrical
paths, giving higher performance