PCIe Basics

PCI Express Basics
Richard Solomon LSI Corporation
Copyright 2010, PCI-SIG, All Rights Reserved
Acknowledgements
I would like to acknowledge the contributions of Ravi Budruk, Mindshare, Inc.
PCIe Technical Seminar
PCI Express Introduction

PCI Express architecture is a high performance, IO interconnect for peripherals in computing/communication platforms Evolved from PCITM and PCI-XTM architectures
Yet PCI Express architecture is significantly different from its predecessors PCI and PCI-X
PCI Express is a serial point-to-point interconnect between two devices Implements packet based protocol for information transfer Scalable performance based on number of signal Lanes implemented on the PCI Express interconnect
PCI Express Terminology

PCI Express Device A
Signal
Link Lane
Wire
PCI Express Device B
PCI Express Throughput

Link Width x1 PCIe 1.x BW (GB/s) PCIe 2.x BW (GB/s) PCIe 3.0 BW (GB/s) 0.5 1 2 x2 1 2 4 x4 2 4 8 x8 4 8 16 x12 6 12 24 x16 8 16 32 x32 16 32 64
Derivation of these numbers: 2.5 GT/s (PCIe 1.x), 5.0 GT/s (PCIe 2.x), or 8GT/s (PCIe 3.0) signaling in each direction 20% overhead due to 8b/10b encoding in 1.x and 2.x Aggregate bandwidth, implying traffic in both directions
PCIe Technical Seminar Copyright 2010, PCI-SIG, All Rights Reserved 5
PCI Express Features

Point-to-point connection Serial bus means fewer pins Scaleable: x1, x2, x4, x8, x12, x16, x32 Dual Simplex connection 2.5, 5.0 and 8.0 GT/s transfer/direction/s Packet based transaction protocol
PCIe Device A
Packet
Link (x1, x2, x4, x8, x12, x16 or x32)

Packet
PCIe Device B
6
Differential Signaling
Electrical characteristics of PCI Express signal
Differential signaling
Transmitter Differential Peak voltage = 0.4 - 0.6 V Transmitter Common mode voltage = 0 - 3.6 V
DVcm D+ V Diffp
Two devices at opposite ends of a Link may support different DC common mode voltages
Additional Features
Switches used to interconnect multiple devices Packet based protocol Bandwidth and clocking Same memory, IO and configuration address space as PCI
Similar transaction types as PCI with additional message transaction
PCI Express Transactions include:

memory read/write, memory read lock, IO read/write, configuration read/write, message requests
Split transaction model for non-posted

Posted: Memory writes and messages Non Posted: Memory reads, config read/write, IO read /weite Completions: Read completions, write completions
http://zone.ni.com/devzone/cda/tut/p/id/3767
Additional Features
Data Integrity and Error Handling
RAS capable (Reliable, Available, Serviceable) Data integrity at: 1) Link level, 2) end-to-end
Virtual channels (VCs) and traffic classes (TCs) to support differentiated traffic or Quality of Service (QoS)
The ability to define levels of performance for packets of different TCs 8 TCs and 8 VCs available
Additional Features
Flow Control
No retry as in PCI
MSI style interrupt handling

Also supports legacy PCI interrupt handling in-band
Advanced power management

Active State PM PCI compatible PM
Message Signaled Interrupt (PCIe 2.2) Allows 1,2,4,8,16,32 interrrupts Message Signaled Interrupt X (PCIe 3.0) Allows up to 2048 interrrupts http://en.wikipedia.org/wiki/Message_Signaled_Interrupts
10
Additional Features
Hot Plug and Hot Swap support
Native No sideband signals
PCI compatible software model

PCI configuration and enumeration software can be used to enumerate PCI Express hardware PCI Express system will boot existing OS PCI Express supports existing device drivers New additional configuration address space requires OS and driver update
11
PCI Express Topology

CPU
Root Complex
Bus 0 (Internal) PCIe 1 PCIe 6 PCIe 7
Memory
PCIe PCIe 3 Switch Endpoint

PCIe 4 PCIe 5
PCIe Endpoint
Switch
PCIe Bridge To
PCI/PCI-X
Virtual PCI Bridge
Virtual PCI Bridge
Bus 2
Virtual PCI Bridge
Virtual PCI Bridge
PCIe Endpoint
Legacy Endpoint
PCI/PCI-X Bus 8
Legend PCI Express Device Downstream Port PCI Express Device Upstream Port
PCI Express System

Processor FSB PCI Express GFX GFX
Root Complex
DDR SDRAM
Slots
HDD
Serial ATA USB 2.0 LPC
PCI Express PCI IO Controller Hub (ICH) IEEE 1394

Slot
S IO
COM1 COM2
GB Ethernet
Add-In Add-In Add-In PCI Express
PCI Express Link

13
Transaction Types, Address Spaces

Request are translated to one of four transaction types by the Transaction Layer:
1. Memory Read or Memory Write. Used to transfer data from or to a memory mapped location The protocol also supports a locked memory read transaction variant. I/O Read or I/O Write. Used to transfer data from or to an I/O location These transactions are restricted to supporting legacy endpoint devices. Configuration Read or Configuration Write. Used to discover device capabilities, program features, and check status in the 4KB PCI Express configuration space. Messages. Handled like posted writes. Used for event signaling and general purpose messaging.
2.
3.
4.
14
PCI Express TLP Types

Description
Memory Read Request Memory Read Request Locked Access Memory Write Request IO Read Request IO Write Request Configuration Read Request Type 0 and Type 1 Configuration Write Request Type 0 and Type 1 Message Request without Data Payload Message Request with Data Payload Completion without Data (used for IO, configuration write completions and read completion with error completion status) Completion with Data (used for memory, IO and configuration read completions)
Abbreviated Name
MRd MRdLk MWr IORd IOWr CfgRd0, CfgRd1 CfgWr0, CfgWr1 Msg MsgD Cpl CplD
Completion for Locked Memory Read without Data (used for error status)
Completion for Locked Memory Read with Data
CplLk
CplDLk
15
Three Methods For Packet Routing

Each request or completion header is tagged as to its type, and each of the packet types is routed based on one of three schemes:
Address Routing ID Routing Implicit Routing
Memory and IO requests use address routing. Completions and Configuration cycles use ID routing. Message requests have selectable routing based on a 3-bit code in the message routing sub-field of the header type field.
16
Programmed I/O Transaction

Processor MRd Requester: -Step 1: Root Complex (requester) initiates Memory Read Request (MRd) -Step 4: Root Complex receives CplD MRd Switch A MRd CplD Switch B MRd Endpoint Endpoint Endpoint CplD Completer: -Step 2: Endpoint (completer) receives MRd -Step 3: Endpoint returns Completion with data (CplD)
17
Processor FSB
Root Complex CplD Switch C
DDR SDRAM
Endpoint
Endpoint
DMA Transaction
Processor Processor
FSB
Completer: -Step 2: Root Complex (completer) receives MRd -Step 3: Root Complex returns Completion with data (CplD) CplD Switch A CplD MRd
Root Complex MRd Switch C
DDR SDRAM
Switch B
CplD Endpoint
Endpoint MRd
Endpoint
Endpoint
Endpoint
Requester: -Step 1: Endpoint (requester) initiates Memory Read Request (MRd) -Step 4: Endpoint receives CplD
18
Peer-to-Peer Transaction
Processor FSB Processor
Root Complex
DDR SDRAM
CplD
Switch A CplD MRd Switch B CplD
MRd
MRd
Switch C MRd Endpoint
CplD
CplD Endpoint Completer: -Step 2: Endpoint (completer) receives MRd -Step 3: Endpoint returns Completion with data (CplD)
Endpoint MRd
Endpoint
Endpoint
Requester: -Step 1: Endpoint (requester) initiates Memory Read Request (MRd) -Step 4: Endpoint receives CplD
19
PCI Express Device Layers

PCI Express Device A Device Core PCI Express Core Logic Interface
TX RX TX
PCI Express Device B Device Core PCI Express Core Logic Interface
RX
Transaction Layer Data Link Layer Physical Layer

Link
20
TLP Origin and Destination

TX
TLP Transmitted
TX RX
TLP Received
RX

Link
TLP: Transaction Layer Packet

TLP Structure
Information in core section of TLP comes from Software Layer / Device Core Bit transmit direction
Start Sequence Header

1B 2B 3-4 DW
Data Payload
0-1024 DW
ECRC LCRC End

1DW 1DW 1B
Created by Transaction Layer
Appended by Data Link Layer
Appended by Physical Layer

DW: Double Word (32 bits)
DLLP Origin and Destination

TX RX TX
RX
Transaction Layer
DLLP Transmitted

Link DLLP Received
Data Link Layer Physical Layer
23
DLLP Structure
Bit transmit direction
Start
1B
DLLP
4B
CRC
2B
End
1B
Data Link Layer
Appended by Physical Layer
ACK / NAK Packets Flow Control Packets Power Management Packets Vendor Defined Packets
24
Ordered-Set Origin and Destination

TX RX TX
RX
Transaction Layer Data Link Layer

Ordered-Set Transmitted

Link Ordered-Set
Physical Layer
Received
25
Ordered-Set Structure
COM Identifier Identifier Identifier
Training Sequence One (TS1)

16 character set: 1 COM, 15 TS1 data characters
Training Sequence Two (TS2)

16 character set: 1 COM, 15 TS2 data characters
SKIP
4 character set: 1 COM followed by 3 SKP identifiers
Fast Training Sequence (FTS)

4 characters: 1 COM followed by 3 FTS identifiers
Electrical Idle (IDLE)

4 characters: 1 COM followed by 3 IDL identifiers
Electrical Idle Exit (EIEOS) (new to 2.0 spec)

16 characters
26
Quality of Service
Processor PCI Express GFX GFX Endpoint Processor
Root Complex
DDR SDRAM
InfiniBand Switch Out-of-Box
InfiniBand
Endpoint
Switch
10Gb Ethernet Endpoint
Switch
Fiber Channel
RAID Disk array
Add-In
Slot
Switch
10Gb Ethernet Endpoint
PCI Express to-PCI Endpoint
SCSI Endpoint PCI
PCI Express Link

Video Camera
SCSI
S IO
COM1 COM2
Endpoint
IEEE 1394
Slots
27
PCI Express Flow Control

Credit-based flow control is point-to-point based, not end-to-end
Buffer space available TLP VC Buffer
Transmitter
Receiver
Flow Control DLLP (FCx) Receiver sends Flow Control Packets (FCP) which are a type of DLLP (Data Link Layer Packet) to provide the transmitter with credits so that it can transmit packets to the receiver
28
ACK/NAK Protocol Overview

Transmit Device A
From Transaction Layer Tx
Receiver Device B
To Transaction Layer Rx
Data Link Layer

TLP
Sequence TLP LCRC
Data Link Layer

DLLP
ACK / NAK
DLLP
ACK / NAK
TLP
Sequence De-mux TLP LCRC
Replay Buffer Mux
De-mux Mux Error Check
Tx
Rx
DLLP
ACK / NAK
Tx
Rx
Link
TLP Sequence PCIe Technical Seminar TLP LCRC 29
ACK/NAK Protocol: Point-to-Point

1a. Request 4b. ACK Requester 1b. ACK 4a. Completion Switch 2b. ACK 3a. Completion 2a. Request 3b. ACK Completer
ACK returned for good reception of Request or Completion NAK returned for error reception of Request or Completion
30
Interrupt Model: Three Methods

PCI Express supports three interrupt reporting mechanisms:
1. Message Signaled Interrupts (MSI)
Legacy endpoints are required to support MSI (or MSI-X) with 32- or 64-bit MSI capability register implementation
Native PCI Express endpoints are required to support MSI with 64-bit MSI capability register implementation
2. 3.
Message Signaled Interrupts - X (MSI-X)

Legacy and native endpoints are required to support MSI-X (or MSI) and implement the associated MSI-X capability register Native and Legacy endpoints are required to support Legacy INTx Emulation PCI Express defines in-band messages which emulate the four physical interrupt signals (INTA-INTD) routed between PCI devices and the system interrupt controller Forwarding support required by switches
Copyright 2010, PCI-SIG, All Rights Reserved 31
INTx Emulation.

Native and Legacy Interrupts
PCIe -
PCIe
PCIe
32
PCI Express Configuration Space
33
Thank you for attending the PCIe Technical Seminar
For more information please go to www.pcisig.com


PCIe Basics

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

PCIe Basics

Uploaded by

Copyright:

Available Formats

PCI Express Basics

Richard Solomon LSI Corporation

Copyright 2010, PCI-SIG, All Rights Reserved

PCIe Technical Seminar

Copyright 2010, PCI-SIG, All Rights Reserved

PCI Express Introduction

PCIe Technical Seminar

Copyright 2010, PCI-SIG, All Rights Reserved

PCI Express Terminology

PCI Express Device B

PCIe Technical Seminar

Copyright 2010, PCI-SIG, All Rights Reserved

PCI Express Throughput

PCI Express Features

Link (x1, x2, x4, x8, x12, x16 or x32)

PCIe Technical Seminar

Copyright 2010, PCI-SIG, All Rights Reserved

PCI Express Transactions include:

Split transaction model for non-posted

PCIe Technical Seminar

Copyright 2010, PCI-SIG, All Rights Reserved

MSI style interrupt handling

Advanced power management

PCIe Technical Seminar

Copyright 2010, PCI-SIG, All Rights Reserved

PCI compatible software model

PCIe Technical Seminar

Copyright 2010, PCI-SIG, All Rights Reserved

PCI Express Topology

PCIe PCIe 3 Switch Endpoint

Virtual PCI Bridge

Virtual PCI Bridge

PCI Express System

Serial ATA USB 2.0 LPC

PCI Express PCI IO Controller Hub (ICH) IEEE 1394

Add-In Add-In Add-In PCI Express

PCI Express Link

Copyright 2010, PCI-SIG, All Rights Reserved

Transaction Types, Address Spaces

PCIe Technical Seminar

Copyright 2010, PCI-SIG, All Rights Reserved

PCI Express TLP Types

PCIe Technical Seminar

Copyright 2010, PCI-SIG, All Rights Reserved

Three Methods For Packet Routing

PCIe Technical Seminar

Copyright 2010, PCI-SIG, All Rights Reserved

Programmed I/O Transaction

Root Complex CplD Switch C

PCIe Technical Seminar

Copyright 2010, PCI-SIG, All Rights Reserved

Root Complex MRd Switch C

PCIe Technical Seminar

Copyright 2010, PCI-SIG, All Rights Reserved

PCIe Technical Seminar

PCI Express Device Layers

Transaction Layer Data Link Layer Physical Layer

Transaction Layer Data Link Layer Physical Layer

PCIe Technical Seminar

Copyright 2010, PCI-SIG, All Rights Reserved

TLP Origin and Destination

Transaction Layer Data Link Layer Physical Layer

Transaction Layer Data Link Layer Physical Layer

TLP: Transaction Layer Packet