You are on page 1of 8

Note: this document is based on the tutorial developed

by Blaise Barney, Livermore Computing. The
complete document is accessable at :

What is Parallel Computing?
• Traditionally, software has been written for serial computation:
o To be executed by a single computer having a single Central Processing
Unit (CPU);
o Problems are solved by a series of instructions, executed one after the
other by the CPU. Only one instruction may be executed at any moment in
• In the simplest sense, parallel computing is the simultaneous use of multiple
compute resources to solve a computational problem.
• The compute resources can include:
o A single computer with multiple processors;
o An arbitrary number of computers connected by a network;
o A combination of both.
• The computational problem usually demonstrates characteristics such as the
ability to be:
o Broken apart into discrete pieces of work that can be solved
o Execute multiple program instructions at any moment in time;
o Solved in less time with multiple compute resources than with a single
compute resource.
• Parallel computing is an evolution of serial computing that attempts to emulate
what has always been the state of affairs in the natural world: many complex,
interrelated events happening at the same time, yet within a sequence. Some
o Planetary and galactic orbits
o Weather and ocean patterns
o Tectonic plate drift
o Rush hour traffic in LA
o Automobile assembly line
o Daily operations within a business
o Building a shopping mall
o Ordering a hamburger at the drive through.
• Traditionally, parallel computing has been considered to be "the high end of
computing" and has been motivated by numerical simulations of complex systems
and "Grand Challenge Problems" such as:

Absolute limits are the speed of light (30 cm/nanosecond) and the transmission limit of copper . parallel computing is an attempt to maximize the infinite but seemingly scarce commodity called time.from prosthetics to spacecraft o electronic circuits o manufacturing processes • Today. o Overcoming memory constraints . For large problems. human genome o geological. o weather and climate o chemical and nuclear reactions o biological. Example applications include: o parallel databases.the speed of a serial computer is directly dependent upon how fast data can move through hardware. Overview Why Use Parallel Computing? • There are two primary reasons for using parallel computing: o Save time .using available compute resources on a wide area network.wall clock time o Solve larger problems • Other reasons might include: o Taking advantage of non-local resources . web based business services o computer-aided diagnosis in medicine o management of national and multi-national corporations o advanced graphics and virtual reality. These applications require the processing of large amounts of data in sophisticated ways. data mining o oil exploration o web search engines. particularly in the entertainment industry o networked video and multi-media technologies o collaborative work environments • Ultimately. using the memories of multiple computers may overcome this obstacle.both physical and practical reasons pose significant constraints to simply building ever faster serial computers: o Transmission speeds .using multiple "cheap" computing resources instead of paying for time on a supercomputer. o Cost savings .single computers have very finite memory resources. • Limits to serial computing . commercial applications are providing an equal or greater driving force in the development of faster computers. seismic activity o mechanical devices . or even the Internet when local compute resources are scarce.

The CPU executes a stored program that specifies a sequence of read and write operations on the memory. decodes the instructions and then sequentially performs them. even with molecular or atomic-level components. virtually all computers have followed a common machine model known as the von Neumann computer. • The future: during the past 10 years. distributed is increasingly expensive to make a single processor faster. a limit will be reached on how small components can be. • A von Neumann computer uses the stored-program concept. wire (9 cm/nanosecond).processor technology is allowing an increasing number of transistors to be placed on a chip. Concepts and Terminology von Neumann Architecture • For over 40 years. • Basic design: o Memory is used to store both program and data instructions o Program instructions are coded data which tell the computer to do something o Data is simply information to be used by the program o A central processing unit (CPU) gets instructions and/or data from memory. the trends indicated by ever faster networks. o Limits to miniaturization . o Economic limitations . Using a larger number of moderately fast commodity processors to achieve the same (or better) performance is less expensive. Named after the Hungarian mathematician John von Neumann. and multi-processor computer architectures (even at the desktop level) suggest that parallelism is the future of computing. Increasing speeds necessitate increasing proximity of processing elements. . However.

single CPU workstations and mainframes o Multiple Instruction. Multiple Data MISD MIMD Multiple Instruction. Single Data (SISD): • A serial (non-parallel) computer • Single instruction: only one instruction stream is being acted on by the CPU during any one clock cycle • Single data: only one data stream is being used as input during any one clock cycle • Deterministic execution • This is the oldest and until recently. • . • The matrix below defines the 4 possible classifications according to Flynn. the most prevalent form of computer • Examples: most PCs.Flynn's Classical Taxonomy • There are different ways to classify parallel computers. One of the more widely used classifications. SISD SIMD Single Instruction. Single Data Single Instruction. Single Data (MISD): • Few actual examples of this class of parallel computer have ever existed • Some conceivable examples might be: o multiple frequency filters operating on a single signal stream o multiple cryptography algorithms attempting to crack a single coded message. Single Data Multiple Instruction. is called Flynn's Taxonomy. • Flynn's taxonomy distinguishes multi-processor computer architectures according to how they can be classified along the two independent dimensions of Instruction and Data. in use since 1966. Multiple Data Single Instruction. Each of these dimensions can have only one of two possible states: Single or Multiple.

Parallel Execution Execution of a program by more than one task. with each task being able to execute the same or different statement at the same moment in time. Task A logically discrete section of computational work. In the simplest sense. Synchronization The coordination of parallel tasks in real time. . tasks can only logically "see" local machine memory and must use communications to access memory on other machines where other tasks are executing. A task is typically a program or program-like set of instructions that is executed by a processor. However. Often implemented by establishing a synchronization point within an application where a task may not proceed further until another task(s) reaches the same or logically equivalent point. Some of the more commonly used terms associated with parallel computing are listed below. Communications Parallel tasks typically need to exchange data. refers to network based memory access for physical memory that is not common. Distributed Memory In hardware. As a programming model. There are several ways this can be accomplished. such as through a shared memory bus or over a network. Shared Memory From a strictly hardware point of view. it describes a model where parallel tasks all have the same "picture" of memory and can directly address and access the same logical memory locations regardless of where the physical memory actually exists. In a programming sense. this is what happens on a one processor machine. parallel computing has its own "jargon". virtually all parallel tasks will have sections of a parallel program that must be executed serially. Parallel Task A task that can be executed by multiple processors safely (yields correct results) Serial Execution Execution of a program sequentially. one statement at a time.Concepts and Terminology Some General Parallel Terminology Like everything else. very often associated with communications. however the actual event of data exchange is commonly referred to as communications regardless of the method employed. describes a computer architecture where all processors have direct (usually bus based) access to common physical memory.

as opposed to doing useful work. The meaning of many keeps increasing. and can therefore cause a parallel application's wall clock execution time to increase. etc. granularity is a qualitative measure of the ratio of computation to communication. Synchronization usually involves waiting by at least one task. • Coarse: relatively large amounts of computational work are done between communication events • Fine: relatively small amounts of computational work are done between communication events Observed Speedup Observed speedup of a code which has been parallelized. Factors that contribute to scalability include: • Hardware . libraries. operating system.particularly memory-cpu bandwidths and network communications • Application algorithm PI Calculation . but currently means more than 1000. Scalability Refers to a parallel system's (hardware and/or software) ability to demonstrate a proportionate increase in parallel speedup with the addition of more processors. • Task termination time Massively Parallel Refers to the hardware that comprises a given parallel system . Parallel Overhead The amount of time required to coordinate parallel tasks. Parallel overhead can include factors such as: • Task start-up time • Synchronizations • Data communications • Software overhead imposed by parallel compilers. Granularity In parallel computing.having many processors. defined as: wall-clock time of serial execution wall-clock time of parallel execution One of the simplest and most widely used indicators for a parallel program's performance. tools.

Randomly generate points in the square 3.npoints generate 2 random numbers between 0 and 1 xcoordinate = random1 . the better the approximation • Serial pseudo code for this procedure: npoints = 10000 circle_count = 0 do j = 1. PI ~ 4 r 6. Let r be the number of points in the circle divided by the number of points in the square 5. Determine the number of points in the square that are also in the circle 4. • The value of PI can be calculated in a number of ways.0*circle_count/npoints  Note that most of the time in running this program would be spent executing the loop . Inscribe a circle in a square 2. ycoordinate = random2 if (xcoordinate. ycoordinate) inside circle then circle_count = circle_count + 1 end do PI = 4. Consider the following method of approximating PI 1. Note that the more points generated.

One task acts as master and collects the results.num generate 2 random numbers between 0 and 1 xcoordinate = random1 . ycoordinate = random2 if (xcoordinate. npoints = 10000 circle_count = 0 p = number of tasks num = npoints/p find out if I am MASTER or WORKER do j = 1. Leads to an embarrassingly parallel solution • Computationally intensive • Minimal communication • Minimal I/O PI Calculation Parallel Solution • Parallel strategy: break the loop into portions that can be executed by the tasks. o Uses the SPMD model. ycoordinate) inside circle then circle_count = circle_count + 1 end do if I am MASTER receive from WORKERS their circle_counts compute PI (use MASTER and WORKER calculations) else if I am WORKER send to MASTER circle_count endif . • Pseudo code solution: red highlights changes for parallelism. o Each task can do its work without requiring any information from the other tasks (there are no data dependencies). • For the task of approximating PI: o Each task executes its portion of the loop a number of times.