CS10 Concurrency & Parallelism, 10 mi up…

The Beauty and Joy of Computing Intra-computer Inter-computer

Lecture #8 : Concurrency   Today’s lecture   Week 12’s lectures
  Multiple computing   Multiple computing
UC Berkeley EECS 2010-09-29 “helpers” are cores “helpers” are different
Lecturer SOE
Dan Garcia within one machine machines
  Aka “multi-core”   Aka “distributed
Nvidia CEO Jen-Hsun Huang said:   Although GPU parallism computing”
“whatever capability you think you have is also “intra-computer”   Grid & cluster computing
today, it’s nothing compared to what you’re
going to have in a couple of years … due
to supercomputers in the cloud”. Now,
cloud computing does what you could do
on your PC. Imagine 40,000 times that!
Garcia, Fall 2010
www.theregister.co.uk/2010/09/24/huang_muses_at_gtc/
UC Berkeley CS10 “The Beauty and Joy of Computing” : Concurrency (2)

Anatomy: 5 components of any Computer But what is INSIDE a Processor?

John von Neumann Processor •  Primarily Crystalline Silicon
invented this Control •  1 mm – 25 mm on a side
(“brain”)
architecture •  2009 “feature size” (aka process)
Datapath ~ 45 nm = 45 x 10-9 m
Computer (“brawn”) (then 32, 22, and 16 [by yr 2013])
Processor •  100 - 1000M transistors
Devices
Control
•  3 - 10 conductive layers
(“brain”) Memory Input •  “CMOS” (complementary metal oxide
Bare Processor Die semiconductor) - most common
Datapath Output
(“brawn”) •  Package provides:
a)  Control
b)  Datapath •  spreading of chip-level signal paths to
c)  Memory board-level
d)  Input What causes the most headaches •  heat dissipation.
e)  Output for SW and HW designers with •  Ceramic or plastic with gold wires.
multi-core computing? Garcia, Fall 2010 Chip in Package Garcia, Fall 2010

UC Berkeley CS10 “The Beauty and Joy of Computing” : Concurrency (4) UC Berkeley CS10 “The Beauty and Joy of Computing” : Concurrency (6)

en.wikipedia.org/wiki/Moore's_law
Moore’s Law Moore’s Law and related curves
Predicts: 2X Transistors / chip every 2 years
# of transistors on an
integrated circuit (IC)

What is this “curve”?

Gordon Moore
a)  Constant
b)  Linear Intel Cofounder
c)  Quadratic B.S. Cal 1950!
d)  Cubic
e)  Exponential

Year
Garcia, Fall 2010 Garcia, Fall 2010

UC Berkeley CS10 “The Beauty and Joy of Computing” : Concurrency (7) UC Berkeley CS10 “The Beauty and Joy of Computing” : Concurrency (8)
Moore’s Law and related curves Power Density Prediction circa 2000
10000"

Power Density (W/cm2)"

Sunʼs Surface"
Rocket Nozzle"
1000"

Nuclear Reactor"
100"
Core 2 "

10" 4004"
8086" Hot Plate" P6"
8008" Pentium® proc"
286" 386"
486"
8080" 8085"
1"
1970" 1980" 1990" 2000" 2010"
Year"

Garcia, Fall 2010 Source:  S.  Borkar  (Intel)   Garcia, Fall 2010

UC Berkeley CS10 “The Beauty and Joy of Computing” : Concurrency (9) UC Berkeley CS10 “The Beauty and Joy of Computing” : Concurrency (10)

Going Multi-core Helps Energy Efficiency Energy & Power Considerations

  Power of typical integrated circuit ~ C V2 f
  C = Capacitance, how well it “stores” a charge
  V = Voltage
  f = frequency. I.e., how fast clock is (e.g., 3 GHz)

Activity Monitor
(on the lab Macs)
shows how active
your cores are

Garcia, Fall 2010 Garcia, Fall 2010

William Holt, HOT Chips 2005" Courtesy: Chris Batten"
UC Berkeley CS10 “The Beauty and Joy of Computing” : Concurrency (11) UC Berkeley CS10 “The Beauty and Joy of Computing” : Concurrency (12)

Parallelism again? What’s different this time? Background: Threads

“This shift toward increasing parallelism is not a   A Thread stands for “thread of execution”, is a
triumphant stride forward based on breakthroughs in single stream of instructions
novel software and architectures for parallelism;   A program / process can split, or fork itself into separate
instead, this plunge into parallelism is actually a threads, which can (in theory) execute simultaneously.
retreat from even greater challenges that thwart   An easy way to describe/think about parallelism
efficient silicon implementation of traditional   A single CPU can execute many threads by
uniprocessor architectures.” Time Division Multipexing
Thread0
– Berkeley View, December 2006 CPU
Thread1
  HW/SW Industry bet its future that breakthroughs will Time Thread2
appear before it’s too late   Multithreading is running multiple threads
view.eecs.berkeley.edu through the same hardware
Garcia, Fall 2010 Garcia, Fall 2010

UC Berkeley CS10 “The Beauty and Joy of Computing” : Concurrency (13) UC Berkeley CS10 “The Beauty and Joy of Computing” : Concurrency (14)
 en.wikipedia.org/wiki/Amdahl's_law
Speedup Issues : Amdahl’s Law Speedup Issues : Overhead
•  Applications can almost never be completely parallelized; some serial
code remains   Even assuming no sequential portion, there’s…
Time
  Time to think how to divide the problem up
Parallel portion
  Time to hand out small “work units” to workers
Serial portion   All workers may not work equally fast
1 2 3 4 5   Some workers may fail
Number of Cores
  There may be contention for shared resources
•  s is serial fraction of program, P is # of cores (was processors)
  Workers could overwriting each others’ answers
•  Amdahl’s law:
  You may have to wait until the last worker returns to
Speedup(P) = Time(1) / Time(P)
proceed (the slowest / weakest link problem)
≤ 1 / ( s + [ (1-s) / P) ], and as P  ∞
  There’s time to put the data back together in a way
≤1/s
that looks as if it were done by one
•  Even if the parallel portion of your application speeds up perfectly,
your performance may be limited by the sequential portion Garcia, Fall 2010 Garcia, Fall 2010

UC Berkeley CS10 “The Beauty and Joy of Computing” : Concurrency (15) UC Berkeley CS10 “The Beauty and Joy of Computing” : Concurrency (16)

en.wikipedia.org/wiki/Concurrent_computing
Life in a multi-core world… But Concurrent programming is hard!
  This “sea change” to multi-   What if two people
core parallelism means were calling withdraw
that the computing at the same time?
community has to rethink:   E.g., balance=100 and
a)  Languages two withdraw 75 each
b)  Architectures   Can anyone see what
c)  Algorithms the problem could be?
d)  Data Structures   This is a race condition

e)  All of the above   In most languages,

this is a problem.
  In Scratch, the system
doesn’t let two of these
Garcia, Fall 2010 run at once. Garcia, Fall 2010

UC Berkeley CS10 “The Beauty and Joy of Computing” : Concurrency (17) UC Berkeley CS10 “The Beauty and Joy of Computing” : Concurrency (18)

Summary
  “Sea change” of
computing because
of inability to cool
CPUs means we’re
now in multi-core
world
  This brave new world
offers lots of potential
for innovation by
computing
professionals, but also
challenges
Garcia, Fall 2010

UC Berkeley CS10 “The Beauty and Joy of Computing” : Concurrency (19)