Professional Documents
Culture Documents
Distributed Systems
Lecture 1 Introduction
1
Goal of the Course
We study architectures, algorithms, and programming
paradigm for parallel and distributed systems.
• Distributed transactions
5
Administrative Issues
• Timing:
– Class: TTH 5:00-6:15PM
– Office Hours: TTH 10:30AM-12:00PM at CU320 or by
appointment
• Course website
– http://www.mscs.mu.edu/~rge/mscs6060
• Textbook:
– An Introduction to Parallel Computing (Ed. 2) by Ananth
Grama et al
– Designing and Building Parallel Programs (Online) by Ian
Foster.
• Exams
– Midterm: Oct 18
– Take home final: 5/13/2010, Tuesday
6
Administrative Issues II
• Grading
– Reading -- 10%
– Programming assignment -- 25%
– Term projects -- 25%
– Midterm exam I -- 20%
– Final exam -- 20%
• Late assignment
– each student is allowed to submit one of his
assignments/projects within 48-hours of due time. All
other assignments and projects must be submitted in
time.
• Course policies
7
About Myself
• Name: Rong Ge
• PhD from Virginia Tech
• Research areas:
– Parallel and distributed systems
– High performance computing
– Energy-efficient computing
– Performance analysis and evaluation
8
Today’s Lecture
• Big picture
– Why do we build parallel and distributed systems?
9
Non Parallel Distributed Systems
terminal
terminal
terminal
Server
10
Free Lunch is Over
11
Parallel System Are Ubiquitous Today
12
Multicore Processors
13
A Multiprocessor, Tightly-Coupled Parallel System
14
A Loosely-Coupled Computer Cluster
15
A Distributed System with Multiple Servers
Server
Server
Server
Server Server
Traditional Uses
• Numerical simulations of complex systems and
"Grand Challenge Problems" such as:
– weather and climate
– chemical and nuclear reactions
– biological, human genome
– geological, seismic activity
– mechanical devices - from prosthetics to spacecraft
– electronic circuits
– manufacturing processes
18
Emerging use
Amazon
Google
Airline reservation systems
Finance: e-transactions, e-banking, stock-
exchange
Military
Advanced graphics and Virtual reality
Networked video and multi-media technologies
19
Why Parallel and Distributed Systems
• Computational speedup
– Some tasks too large for even the fastest single
computer
• Real time weather/climate modeling, human
genome project, fluid turbulence modeling, etc
• Resource sharing
– Distributed systems offer access to specialized
resources of many systems
• Some nodes may have special databases/info
• Some nodes may have access to special hardware
• Reliability and availability
– Redundancy, fail over
20
Why not distributed systems?
23
Multiprogramming and
Multithreading
24
5 components of any Computer
Computer Keyboard,
Processor Memory Devices
Mouse
(active) (passive)
Input Disk
Control
(“brain”) (where (where
programs, programs,
data data
Datapath live when live when
(“brawn”) Output
running) not running)
Display,
Printer
Levels of Abstraction for a Computer
Applications
Operating
Compiler System
Software Assembler
Instruction Set
Hardware Processor Memory I/O system Architecture
OS manages resources
OS manages resources and schedule processes
26
History of Computing Systems
• Phase 1: hardware is expensive, humans are cheap
– User at console: single-user systems
– Batching systems (1955-1965)
– Multiprogramming systems (1965-1980)
• Phase 2: hardware is cheap, humans are expensive
– Time sharing: users use cheap terminal and share
servers
• Phase 3: hardware is very cheap, humans are very
expensive
– Personal computing: one system per user
– Distributed computing: lots of systems per user
27
Multiprogramming
• Multiple sequential processes in a run state can be
swapped in and out by the OS
– Run state: the executable file and data are held in memory
– Better utilization of CPU
– High throughput
– Short timespan for a single program
• IO interrupt
– While I/O is in progress for program P1, the computer will execute
several thousand instructions of program P2 and return to process the
data obtained for P1 when it is ready or when P2 is waiting on I/O
• Time sharing
– The execution of program P1 will be interrupted by time too
Process
• A process is an abstraction that supports running
programs
• A process is the basic unit of execution in an operating
system
• Different processes may run several instances of the
same program
• At a minimum, process execution requires
– Memory to contain the program file and data
– A set of CPU registers to support execution
29
Anatomy of a Process
• Executable file
– Header
– Code
– Initialized data
• Process’s address space
• Process control block
30
CPU Switch From Process to Process
31
Costs of Concurrency Transparency
• Independent processes cannot maliciously or
inadvertently affect the correctness of each other’s
behaviors.
• Cost of concurrency transparency
– Creation: OS must create a complete independent address space
• Initialize memory segments
• Copy the associated program into a text segment
• Set up a stack for temporary data
– Switching:
• Save the CPU context (register values, program counter,
stack pointer, …)
• Modify registers of the MMU
• Invalidate address translation caches
• Swap processes between main memory and disk
32
Single and Multithreaded Processes
33
Threads vs. Processes
Thread Processes
34
Thread Implementation
• Process defines address space
• Threads share address space
TCB for thread1 Process’s address space
$pc
• Process Control Block (PCB) $sp Reserved
contains process-specific info State
DLL’s
Registers
– PID, owner, heap pointer, active … Stack – thread 1
threads and pointers to thread info …
CODE
35
Benefits of Threading
• Responsiveness
– When one thread is blocked, your browser still responds
– E.g. download images while allowing your interaction
• Resource Sharing
– Share the same address space
– Reduce overhead (e.g. memory)
• Economy
– Creating a new process costs memory and resources
– E.g. in Solaris, 30 times slower in creating process than thread
• Utilization of MP Architectures
– Threads can be executed in parallel on multiple processors
– Increase concurrency and throughput
36
Examples of Threads
• A web browser
– One thread displays images
– One thread retrieves data from network
• A word processor
– One thread displays graphics
– One thread reads keystrokes
– One thread performs spell checking in the background
• A web server
– One thread accepts requests
– When a request comes in, separate thread is created to service
– Many threads to support thousands of client requests
37
Once Again....
A PRO CESS A THREAD
P ro c e s s ID T h re a d ID
P ro g ra m C o u n te r P ro g ra m C o u n te r
S ig n a l D is p a t c h S ig n a l D is p a t c h
T a b le T a b le
R e g is t e r s R e g is t e r s
P r o c e s s P r io r ity T h r e a d P r io r it y
S t a c k P o in te r & S t a c k P o in t e r &
S ta c k S ta c k
Heap A ll th r e a d s s h a r e
th e s a m e
m e m o ry , h e a p ,
M e m o ry M a p a n d file h a n d le s
(a n d o ffs e ts )
F ile D e s c r ip t o r T a b le
What does the developer have
to do?
• Decide how to decompose the computation into
parallel parts.
• Create and destroy threads to support the
decomposition
• Add synchronization to make sure dependences
are covered.
What’s POSIX Got To Do With It?
43
Pthreads: POSIX Threads
• Types: pthread[_object]_t
• Functions: pthread[_object]_action
• Constants/Macros: PTHREAD_PURPOSE
• Examples:
– pthread_t: the type of a thread
– pthread_create(): creates a thread
– pthread_mutex_t: the type of a mutex lock
– pthread_mutex_lock(): lock a mutex
pthread_create()
struct arguments {
double *array;
int size;
double *sum;
}
pthread_create() example
if (pthread_create(&worker_thread, NULL,
do_work, (void *)arg)) {
fprintf(stderr,”Error while creating thread\n”);
exit(1);
}
...
}
pthread_create() example
size = argument->size;
array = argument->array;
sum = argument->sum;
*sum = 0;
for (i=0;i<size;i++)
*sum += array[i];
return NULL;
}
Comments about the example