Lecture 3: Computer Architectures

Basic Computer Architecture

Memory

instruction data

Von Neumann Architecture

Input unit

ALU Processor CU Reg.

Output unit

Levels of Parallelism

Bit level parallelism

Within arithmetic logic circuits Multiple instructions execute per clock cycle Overlap of memory operations with computation More than one processor Multiple jobs run in parallel on SMP

Instruction level parallelism

Memory system parallelism

Operating system parallelism

Loop level Procedure level

Levels of Parallelism
Bit Level Parallelism
Within arithmetic logic circuits

Levels of Parallelism
Instruction Level Parallelism (ILP)
Multiple instructions execute per clock cycle

Pipelining (instruction - data) Multiple Issue (VLIW)

Levels of Parallelism
Memory System Parallelism
Overlap of memory operations with computation

Levels of Parallelism
Operating System Parallelism

There are more than one processor Multiple jobs run in parallel on SMP

Loop level Procedure level

Flynns Taxonomy

Single Instruction stream - Single Data stream (SISD) Single Instruction stream - Multiple Data stream (SIMD) Multiple Instruction stream - Single Data stream (MISD) Multiple Instruction stream - Multiple Data stream (MIMD)

Single Instruction stream Single Data stream (SISD)

Memory

instruction data

Von Neumann Architecture

CU

ALU

Processor

Flynns Taxonomy

Single Instruction stream - Single Data stream (SISD) Single Instruction stream - Multiple Data stream (SIMD) Multiple Instruction stream - Single Data stream (MISD) Multiple Instruction stream - Multiple Data stream (MIMD)

Single Instruction stream Multiple Data stream (SIMD)

PE

data

Instructions of the program are broadcast to more than one processor Each processor executes the same instruction synchronously, but using different data

Memory

instruction CU

PE

data

PE

data

PE

data

instruction

Used for applications that operate upon arrays of data

Flynns Taxonomy

Single Instruction stream - Single Data stream (SISD) Single Instruction stream - Multiple Data stream (SIMD) Multiple Instruction stream - Single Data stream (MISD) Multiple Instruction stream - Multiple Data stream (MIMD)

Multiple Instruction stream Multiple Data stream (MIMD)

Each processor has a separate program

An instruction stream is generated for each program on each processor Each instruction operates upon different data

Multiple Instruction stream Multiple Data stream (MIMD)

Shared memory

Distributed memory

Shared vs Distributed Memory

M
P

M
P

M
P

M
P

Distributed memory

Network

Each processor has its own local memory Message-passing is used to exchange data between processors

Shared memory

P Bus

Memory

Single address space All processes have access to the pool of shared memory

Distributed Memory
M P NI M P NI M P NI M P NI

Processors cannot directly access another processors memory Each node has a network interface (NI) for communication and synchronization

Network

Distributed Memory
M
instr data

CU

PE

data

data

instr data

CU

PE Network

instr data

data

CU

PE

Each processor executes different instructions asynchronously, using different data

instr data

data

CU

PE

Shared Memory
CU PE
data

data

data

CU

PE
data

CU

PE

instruction

Memory

CU

PE

Each processor executes different instructions asynchronously, using different data

Shared Memory
P P
Bus Memory

Uniform memory access (UMA)

Each processor has uniform access to memory (symmetric multiprocessor - SMP)

P Bus

P Bus

Non-uniform memory access (NUMA)

Memory Network

Memory

Time for memory access depends on the location of data Local access is faster than nonlocal access Easier to scale than SMPs

Distributed Shared Memory

Making the main memory of a cluster of computers look as if it is a single memory with a single address space Shared memory programming techniques can be used

Distributed Systems

Clusters

Individual computers, that are tightly coupled by

software, in a local environment, to work together on single problems or on related problems

Grid

Many individual systems, that are geographically

distributed, are tightly coupled by software, to work together on single problems or on related problems