Professional Documents
Culture Documents
SPEC Requirements
f1 f2 Problem
i
Focus
p
Algorithms f5 f3 s q
fp
j
f2() {
f3(s2, &j, &i);
f3
*s2->p = 10; f4
}
i = *s2->q + i;
Prog. Lang./OS
i1: ld r1, b <p1>
i2: ld r2, c <p1>
ISA i3:
i4:
ld r5, z
mul r6, r5, 3
<p3>
<p3>
i5: add r3, r1, r2 <p1>
uArch
Performance
Circuit
Gate
Focus
Device
Source Drain
Appendix A - Pipelining 1
Instruction Set Architecture
Application
Intel Pentium X
AMD K6, Athlon, Opteron
Implementation Transmeta Crusoe TM5x00
Appendix A - Pipelining 2
Instruction Set Architecture
• Strong influence on cost/performance
Appendix A - Pipelining 3
Traditional Issues
• Strongly constrained by the number of
bits available to instruction encoding
• Opcodes/operands
• Registers/memory
• Addressing modes
• Orthogonality
• 0, 1, 2, 3 address machines
• Instruction formats
• Decoding uniformity
Appendix A - Pipelining 4
Introduction
A.1 What is Pipelining?
A.2 The Major Hurdle of Pipelining-Structural Hazards
– Data Hazards
– Control Hazards
Appendix A - Pipelining 5
What Is Pipelining
• Laundry Example
• Ann, Brian, Cathy, Dave A B C D
each have one load of clothes
to wash, dry, and fold
• Washer takes 30 minutes
Appendix A - Pipelining 6
What Is Pipelining
6 PM 7 8 9 10 11 Midnight
Time
30 40 20 30 40 20 30 40 20 30 40 20
T
a A
s
k
B
O
r
d C
e
r
D
Sequential laundry takes 6 hours for 4 loads
If they learned pipelining, how long would laundry take?
Appendix A - Pipelining 7
What Is Pipelining
Start work ASAP
6 PM 7 8 9 10 11 Midnight
Time
30 40 40 40 40 20
T
a A
s • Pipelined laundry takes
k 3.5 hours for 4 loads
B
O
r
d C
e
r
D
Appendix A - Pipelining 8
What Is Pipelining
Lessons
Pipelining
• Pipelining doesn’t help
6 PM 7 8 9 latency of single task, it helps
throughput of entire workload
Time
• Pipeline rate limited by
T slowest pipeline stage
a 30 40 40 40 40 20
• Multiple tasks operating
s
simultaneously
k A
• Potential speedup = Number
O pipe stages
r B • Unbalanced lengths of pipe
d stages reduces speedup
e • Time to “fill” pipeline and time
r C to “drain” it reduces speedup
Appendix A - Pipelining 9
What Is MIPS Without
Pipelining Pipelining
Instruction Instr. Decode Execute Memory Write
Fetch Reg. Fetch Addr. Calc Access Back
IR
L
M
D
Appendix A - Pipelining 10
What Is
Pipelining MIPS Functions
Instruction Instr. Decode Execute Memory Write
Fetch Reg. Fetch Addr. Access Back
Calc Passed To Next Stage
IR <- Mem[PC]
NPC <- PC + 4
IR L
M
D
Appendix A - Pipelining 11
What Is
Pipelining MIPS Functions
Instruction Instr. Decode Execute Memory Write
Fetch Reg. Fetch Addr. Access Back
Calc Passed To Next Stage
A <- Regs[IR6..IR10];
B <- Regs[IR10..IR15];
IR L
Imm <- ((IR16) ##IR16-31
M
D
Appendix A - Pipelining 12
What Is
Pipelining MIPS Functions
Instruction Instr. Decode Execute Memory Write
Fetch Reg. Fetch Addr. Access Back
Calc Passed To Next Stage
A <- A func. B
cond = 0;
IR L
M
D
Appendix A - Pipelining 14
What Is
Pipelining MIPS Functions
Instruction Instr. Decode Execute Memory Write
Fetch Reg. Fetch Addr. Access Back
Calc Passed To Next Stage
Regs <- A, B;
IR L
M
D
Appendix A - Pipelining 15
The Basic Pipeline For MIPS
Latches between
each stage provide
pipelining.
Appendix A - Pipelining 16
The Basic Pipeline For MIPS
Cycle 1 Cycle 2 Cycle 3 Cycle 4 Cycle 5 Cycle 6 Cycle 7
ALU
Ifetch Reg DMem Reg
I
n
s
ALU
Reg
t Ifetch Reg DMem
r.
ALU
O Ifetch Reg DMem Reg
r
d
ALU
e Ifetch Reg DMem Reg
Figure 3.3
Appendix A - Pipelining 17
Pipeline Hurdles
A.1 What is Pipelining?
A.2 The Major Hurdle of Pipelining-
Structural Hazards
-- Structural Hazards Limits to pipelining: Hazards prevent next
– Data Hazards instruction from executing during its designated
clock cycle
– Control Hazards
– Structural hazards: HW cannot support this
A.3 How is Pipelining Implemented
combination of instructions (single person to fold
A.4 What Makes Pipelining Hard to and put clothes away)
Implement?
– Data hazards: Instruction depends on result of
A.5 Extending the MIPS Pipeline to prior instruction still in the pipeline (missing
Handle Multi-cycle Operations sock)
– Control hazards: Pipelining of branches &
other instructions that change the PC
– Common solution is to stall the pipeline until the
hazard is resolved, inserting one or more
“bubbles” in the pipeline
Appendix A - Pipelining 18
Pipeline Hurdles
Definition
• conditions that lead to incorrect behavior if not fixed
• Structural hazard
– two different instructions use same h/w in same cycle
• Data hazard
– two different instructions use same storage
– must appear as if the instructions execute in correct order
• Control hazard
– one instruction affects which instruction is next
Resolution
• Pipeline interlock logic detects hazards and fixes them
• simple solution: stall
• increases CPI, decreases performance
• better solution: partial stall
• some instruction stall, others proceed better to stall early than late
Appendix A - Pipelining 19
Structural Hazards
When two or
Time (clock cycles) more different
instructions want
Cycle 1 Cycle 2 Cycle 3 Cycle 4 Cycle 5 Cycle 6 Cycle 7 to use same
hardware
I Load Ifetch
ALU
Reg DMem Reg resource in same
n cycle
s Instr 1
ALU
Ifetch Reg DMem Reg
e.g., MEM uses
t the same memory
r. port as IF as
ALU
Ifetch Reg DMem Reg
Instr 2 shown in this
slide.
O
ALU
r Instr 3 Ifetch Reg DMem Reg
d
e
ALU
Reg
Instr 4 Ifetch Reg DMem
Figure 3.6
Appendix A - Pipelining 20
Structural Hazards
Time (clock cycles)
Cycle 1 Cycle 2 Cycle 3 Cycle 4 Cycle 5 Cycle 6 Cycle 7 This is another
way of looking
at the effect of
I Load Ifetch
ALU
Reg DMem Reg
a stall.
n
s
ALU
t
Instr 1 Ifetch Reg DMem Reg
r.
ALU
Reg
Instr 2 Ifetch Reg DMem
O
r
Stall Bubble Bubble Bubble Bubble Bubble
d
e
r
ALU
Instr 3 Ifetch Reg DMem Reg
Figure 3.7
Appendix A - Pipelining 21
Structural Hazards
Appendix A - Pipelining 22
Structural Hazards
Dealing with Structural Hazards
Stall
• low cost, simple
• Increases CPI
• use for rare case since stalling has performance effect
Pipeline hardware resource
• useful for multi-cycle resources
• good performance
• sometimes complex e.g., RAM
Replicate resource
• good performance
• increases cost (+ maybe interconnect delay)
• useful for cheap or divisible resources
Appendix A - Pipelining 23
Structural Hazards
Structural hazards are reduced with these rules:
• Each instruction uses a resource at most once
• Always use the resource in the same pipeline stage
• Use the resource for one cycle only
Many RISC ISA’a designed with this in mind
Sometimes very complex to do this. For example, memory of
necessity is used in the IF and MEM stages.
Appendix A - Pipelining 24
Structural Hazards
This is the example on Page 144.
Appendix A - Pipelining 25
Data Hazards
A.1 What is Pipelining?
A.2 The Major Hurdle of Pipelining-
Structural Hazards
-- Structural Hazards
These occur when at any time, there are
– Data Hazards instructions active that need to access the
– Control Hazards same data (memory or register) locations.
A.3 How is Pipelining Implemented
Where there’s real trouble is when we have:
A.4 What Makes Pipelining Hard to
Implement?
instruction A
A.5 Extending the MIPS Pipeline to
Handle Multi-cycle Operations instruction B
Appendix A - Pipelining 26
Data Hazards
Execution Order is:
Read After Write (RAW)
InstrJ tries to read operand before InstrI writes it
InstrI
InstrJ
I: add r1,r2,r3
J: sub r4,r1,r3
• Caused by a “Dependence” (in compiler nomenclature).
This hazard results from an actual need for
communication.
Appendix A - Pipelining 27
Data Hazards
Execution Order is:
Write After Read (WAR)
InstrJ tries to write operand before InstrI reads i
InstrI
InstrJ – Gets wrong operand
I: sub r4,r1,r3
J: add r1,r2,r3
K: mul r6,r1,r7
Appendix A - Pipelining 28
Data Hazards
Execution Order is:
Write After Write (WAW)
InstrJ tries to write operand before InstrI writes it
InstrI
InstrJ – Leaves wrong result ( InstrI not InstrJ )
I: sub r1,r4,r3
J: add r1,r2,r3
K: mul r6,r1,r7
Appendix A - Pipelining 29
Data Hazards
Simple Solution to RAW
Appendix A - Pipelining 30
Data Hazards
Time (clock cycles)
IF ID/RF EX MEM WB
I
ALU
add r1,r2,r3Ifetch Reg DMem Reg
n
s
ALU
Ifetch Reg DMem Reg
t sub r4,r1,r3
r.
ALU
Ifetch Reg DMem Reg
and r6,r1,r7
O
r
ALU
Ifetch Reg DMem Reg
d
or r8,r1,r9
e
ALU
Reg
r xor r10,r1,r11 Ifetch Reg DMem
The use of the result of the ADD instruction in the next three instructions causes a
hazard, since the register is not written until after those instructions read it.
Figure 3.9
Appendix A - Pipelining 31
Forwarding is the concept of making data
available to the input of the ALU for
Data Hazards subsequent instructions, even though the
generating instruction hasn’t gotten to WB
Forwarding To Avoid in order to write the memory or registers.
Data Hazard
Time (clock cycles)
I
n add r1,r2,r3 Ifetch
ALU
Reg DMem Reg
s
t
ALU
r. sub r4,r1,r3 Ifetch Reg DMem Reg
ALU
Ifetch Reg DMem Reg
r and r6,r1,r7
d
e
ALU
Ifetch Reg DMem Reg
r or r8,r1,r9
ALU
Ifetch Reg DMem Reg
xor r10,r1,r11
Figure 3.10
Appendix A - Pipelining 32
The data isn’t loaded until after
Data Hazards the MEM stage.
Time (clock cycles)
ALU
Reg DMem Reg
n
s
t
ALU
sub r4,r1,r6 Ifetch Reg DMem Reg
r.
ALU
Ifetch Reg DMem Reg
and r6,r1,r7
r
d
e
ALU
Ifetch Reg DMem Reg
r
or r8,r1,r9
There are some instances where hazards occur, even with forwarding.
Figure 3.12
Appendix A - Pipelining 33
The stall is necessary as shown
Data Hazards here.
Time (clock cycles)
I
n
ALU
Reg
s lw r1, 0(r2) Ifetch Reg DMem
t
r.
ALU
sub r4,r1,r6 Ifetch Reg Bubble DMem Reg
O
r
d Bubble
ALU
Ifetch Reg DMem Reg
e and r6,r1,r7
r
Bubble
ALU
Ifetch Reg DMem
or r8,r1,r9
There are some instances where hazards occur, even with forwarding.
Figure 3.13
Appendix A - Pipelining 34
This is another
representation
Data Hazards of the stall.
Appendix A - Pipelining 35
Data Hazards Pipeline Scheduling
lw Rb, b
lw Rc, c
lw Re, e
Add Ra, Rb, Rc
lw Rf, f
sw a, Ra
sub Rd, Re, Rf
sw d, Rd
Appendix A - Pipelining 36
Pipeline Scheduling
Data Hazards
scheduled unscheduled
54%
gcc 31%
42%
spice 14%
tex 65%
25%
Appendix A - Pipelining 37
Control Hazards
A.1 What is Pipelining?
A.2 The Major Hurdle of Pipelining-
Structural Hazards
-- Structural Hazards
– Data Hazards A control hazard is when we
– Control Hazards need to find the destination
A.3 How is Pipelining Implemented of a branch, and can’t fetch
A.4 What Makes Pipelining Hard to any new instructions until
Implement? we know that destination.
A.5 Extending the MIPS Pipeline to
Handle Multi-cycle Operations
Appendix A - Pipelining 38
Control Hazard on
Control Hazards Branches
Three Stage Stall
ALU
10: beq r1,r3,36 Ifetch Reg DMem Reg
ALU
Ifetch Reg DMem Reg
14: and r2,r3,r5
ALU
Reg
18: or r6,r1,r7 Ifetch Reg DMem
ALU
Ifetch Reg DMem Reg
22: add r8,r1,r9
ALU
36: xor r10,r1,r11 Ifetch Reg DMem Reg
Appendix A - Pipelining 39
Control Hazards Branch Stall Impact
• If CPI = 1, 30% branch, Stall 3 cycles => new CPI = 1.9!
(Whoa! How did we get that 1.9???)
• Two part solution to this dramatic increase:
– Determine branch taken or not sooner, AND
– Compute taken branch address earlier
• MIPS Solution:
– Move Zero test to ID/RF stage
– Adder to calculate new PC in ID/RF stage
• must be fast
• can't afford to subtract
• compares with 0 are simple
• Greater-than, Less-than test signbit, but not-equal must OR all bits
• more general compares need ALU
– 1 clock cycle penalty for branch versus 3
In the next chapter, we’ll look at ways to avoid the branch all together.
Appendix A - Pipelining 40
Control Hazards Five Branch Hazard
Alternatives
#1: Stall until branch direction is clear
Appendix A - Pipelining 41
Control Hazards Five Branch Hazard
Alternatives
#4: Execute Both Paths
branch instruction
sequential successor1
sequential successor2
........
sequential successorn Branch delay of length n
branch target if taken
Appendix A - Pipelining 42
Control Hazards Delayed Branch
Appendix A - Pipelining 43
Control Hazards Evaluating Branch
Alternatives
Appendix A - Pipelining 44
Control Hazards Pipelining Introduction
Summary
Appendix A - Pipelining 45
Control Hazards Compiler “Static”
Prediction of
The compiler can program what it thinks
the branch direction will be. Here are Taken/Untaken Branches
the results when it does so.
70% 14%
60% 12%
Frequency of Misprediction
50% 10%
Misprediction Rate
40% 8%
30% 6%
20% 4%
10% 2%
0% 0%
doduc
gcc
doduc
espresso
gcc
ora
compress
tomcatv
espresso
hydro2d
ora
compress
tomcatv
hydro2d
alvinn
swm256
alvinn
mdljsp2
swm256
mdljsp2
Always taken Taken backwards
Not Taken Forwards
Appendix A - Pipelining 46
Control Hazards Compiler “Static”
Prediction of
Taken/Untaken Branches
• Two strategies
– Backward branch predict taken, forward branch not taken
– Profile-based prediction: record branch behavior, predict branch
based on prior run
Appendix A - Pipelining 47
Evaluating Static
Control Hazards Branch Prediction
Strategies
100000
• Misprediction ignores
• “Instructions between
mispredicted branches” 1000
is a better metric
100
10
doduc
gcc
espresso
ora
compress
tomcatv
hydro2d
alvinn
swm256
mdljsp2
Profile-based Direction-based
Appendix A - Pipelining 48
What Makes Pipelining Hard?
A.1 What is Pipelining?
A.2 The Major Hurdle of Pipelining-
Structural Hazards
– Data Hazards
– Control Hazards
A.3 How is Pipelining Implemented
A.4 What Makes Pipelining Hard to
Implement?
A.5 Extending the MIPS Pipeline to
Handle Multi-cycle Operations
Appendix A - Pipelining 49
What Makes Interrupts cause
great havoc!
Pipelining Hard?
Examples of interrupts: There are 5 instructions executing
• Power failing, in 5 stage pipeline when an
interrupt occurs:
• Arithmetic overflow,
• How to stop the pipeline?
• I/O device request,
• How to restart the pipeline?
• OS call, • Who caused the interrupt?
• Page fault
Appendix A - Pipelining 50
What Makes Interrupts cause
Appendix A - Pipelining 51
What Makes Interrupts cause
great havoc!
Pipelining Hard?
• Simultaneous exceptions in more than one pipeline stage, e.g.,
– Load with data page fault in MEM stage
– Add with instruction page fault in IF stage
– Add fault will happen BEFORE load fault
• Solution #1
– Interrupt status vector per instruction
– Defer check until last stage, kill state update if exception
• Solution #2
– Interrupt ASAP
– Restart everything that is incomplete
Another advantage for state update late in pipeline!
Appendix A - Pipelining 52
What Makes Interrupts cause
great havoc!
Pipelining Hard?
Here’s what happens on a data page fault.
1 2 3 4 5 6 7 8 9
i F D X M W
i+1 F D X M W < page fault
i+2 F D X M W < squash
i+3 F D X M W < squash
i+4 F D X M W < squash
i+5 trap > F D X M W
i+6 trap handler > F D X M W
Appendix A - Pipelining 53
What Makes Complex
Instructions
Pipelining Hard?
Complex Addressing Modes and Instructions
• Address modes: Autoincrement causes register change
during instruction execution
– Interrupts? Need to restore register state
– Adds WAR and WAW hazards since writes are no longer the
last stage.
• Condition Codes
Appendix A - Pipelining 54
Handling Multi-cycle Operations
A.1 What is Pipelining?
A.2 The Major Hurdle of Pipelining-
Structural Hazards
– Data Hazards Multi-cycle instructions also
– Control Hazards lead to pipeline complexity.
A.3 How is Pipelining Implemented
A.4 What Makes Pipelining Hard to
Implement? A very lengthy instruction
A.5 Extending the MIPS Pipeline to
causes everything else in
Handle Multi-cycle Operations the pipeline to wait for it.
Appendix A - Pipelining 55
Multi-Cycle Floating Point
Operations
Floating point gives long execution time.
This causes a stall of the pipeline.
It’s possible to pipeline the FP execution unit so it can initiate new instructions
without waiting full latency. Can also have multiple FP units.
Appendix A - Pipelining 56
Multi-Cycle Floating Point
Operations
Divide, Square Root take 10X to 30X longer than Add
– Interrupts?
– Adds WAR and WAW hazards since pipelines are
no longer same length
1 2 3 4 5 6 7 8 9 10 11
i IF ID EX MEM WB
I+1 IF ID EX EX EX EX MEM WB
I+2 IF ID EX MEM WB
I+3 IF ID EX EX EX EX MEM WB
I+4 IF ID EX MEM WB
I+5 IF ID -- -- EX EX
I+6 IF -- -- ID EX
Notes:
I + 2: no WAW, but this complicates an interrupt
I + 4: no WB conflict
I + 5: stall forced by structural hazard
I + 6: stall forced by in-order issue
Appendix A - Pipelining 57
Summary of Pipelining Basics
• Hazards limit performance
– Structural: need more HW resources
– Data: need forwarding, compiler scheduling
– Control: early evaluation & PC, delayed branch, prediction
• Increasing length of pipe increases impact of hazards; pipelining
helps instruction bandwidth, not latency
• Interrupts, Instruction Set, FP makes pipelining harder
• Compilers reduce cost of data and control hazards
– Load delay slots
– Branch delay slots
– Branch prediction
Appendix A - Pipelining 58
Credits
I have not written these notes by myself. There’s a great deal of fancy
artwork here that takes considerable time to prepare.
Appendix A - Pipelining 59
Summary
A.1 What is Pipelining?
A.2 The Major Hurdle of Pipelining-Structural Hazards
– Data Hazards
– Control Hazards
Appendix A - Pipelining 60