Transmit Flow basics

Page 1
 TX flow:
Memory 1. The Host creates a descriptor
Host Descriptor ring and configure one of Odem
transmit queues with the
address location, length, head
Descriptor Head and tail pointers of the ring
Odem
2. On packet arrival from the
Tail TDT=2 application, Host initializes
descriptor (s) that point to the
data buffer(s). The host place
the descriptor in the correct
location at the appropriate Tx
ring
3. The Host updates the
appropriate queue tail pointer
Packet
0x123456 TDT
4. Odem reads the descriptor
5. Using the address in the
descriptor, Odem reads packet
6. Odem send packet to network
7. Odem writes back the descriptor
and interrupts the driver it is
Packet done

0x123456
DD bit Length

Page 2 Intel Confidential

 Legacy Transmit Descriptor Format

 Buffer address field (64) – Physical address of a data buffer in

host memory that contains a portion of a transmit packet
 Length field - (TDESC.LENGTH) specifies the length in bytes
to be fetched from the buffer address provided.
– For the first phase the value will be 64 bytes
 CSS, STA, CSO and VLAN fields should be constrained to ‘0’

Page 3 Intel Confidential

 Transmit Descriptor Ring Structure

Transmit Descriptor Head register

(TDH) - This register holds a value
which is an offset from the base,
and indicates the in-progress
Descriptor descriptor. Updated by HW
Transmit Descriptor Base Address
While SW can read it.
register (TDBAL/H) This register Descriptor
indicates the start address of the Descriptor
descriptor ring buffer in the Descriptor
host memory
Descriptor
Descriptor
Descriptor Transmit Descriptor Tail register
Transmit Descriptor Length register
Descriptor (TDT) This register holds a value,
(TDLEN) - This register determines
which is an offset from the base.
the number of bytes allocated to the Descriptor This is the location where software
circular buffer. This value must be Descriptor writes the first new descriptor.
0 modulo 128 Updated by SW, while HW can
read it

Page 4 Intel Confidential

 Odem Transmit Data Flow
Step Description
The host creates a descriptor ring and configures one of Sageville’s transmit queues with the address location, length, head, and tail
1
pointers of the ring (one of 128 available Tx queues).

2 The host is requested by the TCP/IP stack to transmit a packet, it gets the packet data within one or more data buffers.

The host initializes the descriptor(s) that point to the data buffer(s) and have additional control parameters that describes the needed
3
hardware functionality. The host places that descriptor in the correct location at the appropriate Tx ring.

4 The host updates the appropriate queue tail pointer (TDT)

Odem’s DMA senses a change of a specific TDT and as a result sends a PCIe request, over the Primary IOSF bus, to fetch the
5
descriptor(s) from host memory.
The descriptor(s) content is received in a PCIe read completion,, over the Primary IOSF bus, and is written to the appropriate location in
6
the descriptor queue.
The DMA fetches the next descriptor and processes its content. As a result, the DMA sends PCIe requests, over the Primary IOSF bus,
7
to fetch the packet data from system memory.
The packet data is being received from PCIe completions, over the Primary IOSF bus, and passes through the transmit DMA that
8 performs all programmed data manipulations (various CPU offloading tasks as checksum offload, TSO offload, etc.) on the packet data
on the fly.
While the packet is passing through the DMA, it is stored into the transmit FIFO.
9
After the entire packet is stored in the transmit FIFO, it is then forwarded to transmit switch module.
10 The transmit switch arbitrates between host and management packets and eventually forwards the packet to the MAC.
11 The MAC appends the L2 CRC to the packet and delivers the packet to the integrated PHY.
The PHY performs the PCS encoding, scrambling and the other manipulations required to deliver the packet over the physical interface
12
at the selected speed.
13 When all the PCIe completions for a given packet are complete, the DMA updates the appropriate descriptor(s).
The descriptors are written back to host memory, over the Primary IOSF bus, using PCIe posted writes. The head
14
pointer is updated in host memory as well.

An interrupt is generated to notify the host driver that the specific packet has been read to Odem and the driver can then release the
15
buffer(s).
Page 5 Intel Confidential
Receive Flow basics

Page 6
 RX Flow:

1. The host creates a descriptor

Memory ring and configures one of
Host Descriptor
Descriptor
Odem receive queues with the
address location, length, head,
Descriptor Head and tail pointers of the ring
Odem (one of 128 available Rx
queues).
Tail 2. The host initializes
RDT=2 descriptors that point to
empty data buffers. The host
places these descriptors in
the correct location at the
appropriate Rx ring.
3. The host updates the
Allocated memory
0x789ABC appropriate queue tail
pointer (RDT).

5. A packet enters the Odem.

the receive DMA ask for
descriptors from host, and
EOP DDposts the packet data to
then
the location indicated by the
Allocated memory descriptor through the PCIe
interface

Packet
6. Odem writes back the
0x789ABC descriptor which contains
Length
the status of packet and
interrupts the driver that
packet received.

Page 7 Intel Confidential

 Odem Receive Data Flow
Step Description
The host creates a descriptor ring and configures one of Odem’s receive queues with the address location, length,
1
head, and tail pointers of the ring (one of 128 available Rx queues).

The host initializes descriptor(s) that point to empty data buffer(s). The host places these descriptor(s) in the
2
correct location at the appropriate Rx ring.

3 The host updates the appropriate Queue Tail Pointer (RDT).

4 A packet enters the PHY through the physical interface.
5 The PHY performs the required manipulations on the incoming signal descrambling, PCS decoding, etc.
6 The PHY delivers the packet to the Rx MAC.
7 The MAC forwards the packet to the Rx filter.
8 If the packet matches the pre-programmed criteria of the Rx filtering, it is forwarded to an Rx FIFO.

9 The receive DMA fetches the next descriptor from the appropriate host memory ring to be used for the next received packet.

After the entire packet is placed into an Rx FIFO, the receive DMA posts the packet data to the location indicated by the descriptor
10 through the Primary IOSF bus. If the packet size is greater than the buffer size, more descriptor(s) are fetched and their buffers are used
for the received packet.
11 When the packet is placed into host memory, the receive DMA updates all the descriptor(s) that were used by the packet data.
The receive DMA writes back the descriptor content along with status bits that indicate the packet information including what offloads
12
were done on that packet.

13 Odem initiates an interrupt to the host to indicate that a new received packet is ready in host memory.

The host reads the packet data and sends it to the TCP/IP stack for further processing. The host releases the associated buffer(s) and
14
descriptor(s) once they are no longer in use.

Page 8 Intel Confidential

Configuration – Config, Mem, IO space

 MGPK gets an entire bus to work with, in MGPK each port is a

differnet function (port 0 = B:0:0, port 1 = B:0:1).
 VRP translates B:0:x accesses to 0:0:x so MGPK does not have to
know what BUS it was assigned.
– The RAL code for MGPK must be given the correct bus value so commands
are routed properly.
 MGPK has 2 MEM space base address registers (BAR). BAR0/1
(32-bit registers, BAR1 is upper 32 bits) for most accesses and
BAR 4/5 for virtual address spaces (usually not needed).
 MGPK also has an IO BAR, BAR 2. IO registers are unused in
normal testing.

Page 9
Configuration – VRP programming

 First, we use the system manager (SM) and constraints manager

(CM) to ensure that the VRP and MGPK are programmed to the
same values, can be randomized, and don’t conflict with other
blocks in the SOC.
 VRP SECBUS and SUBBUS must be programmed to the bus
number assigned to the MGPK.
 VRP MEMBASE and MEMLIMIT define the memory region that
VRP will route to the MGPK, BAR0/1 and BAR4/5 for each port
must be inside this region. MEMBASE/MEMLIMIT are 32-bit, if the
BAR regions are in 64-bit space you use
PREFETCHBASE/PREFETCHLIMIT instead.
 VRP IOBASE/IOLIMIT serve a similar function for IO accesses.

Page 10
Common programming problems

 Check the “south” VRP tracker file to see if transactions are being
routed through VRP correctly.
 MGPK registers must have B:D:F programmed correctly (normally
set correctly during model boot).
 BARs should be programmed correctly (RAL automatically ties
BAR writes into the appropriate registers) but check for
misconfigured registers.
 Check for overlapping memory regions or multiple devices
getting the same bus value (will cause routing errors in PSF).

Page 11
Scoreboards

 Uses INTC scoreboard

– Based on creating Expected packets and then matching them via actual on
the output side.
– This can track missing packets, packets going through incorrect paths, and
corrupted data within the packet.
 TX Flow (HOSTx -> PHYx)
– Expected is created by the Tx packet sequence (gbe_tx_pkt_seq) when the
packet is created and placed into system memory
– Actual is created by the Cadence BFM once it detects a transmitted packet.
– Actual determines HOSTx via a lookup function using the MAC src/dest
addresses as keys (a common failure mode on MGPK was addresses not
matching causing a bad HOSTx lookup).
– PHYx is obvious based on which BFM received the packet.

Page 12
 Transmit bring up flow
Gen TX packet with 1 Read until
descriptor RX descriptor[n].status.dd==1
Platform Reset and
init sequences
Read RX packet from
Write the TX packet HM associated with this
Platform Reset and the associate Descriptor using the
Init TX data phase
descriptor to the HM RX data
packetphase
length field
and
GBE initGBE
sequences per function per function
Configuration per queue per queue
Send RX packet to
phase Increment TDT by 1 SCBD in order to
compare it with TX
Initialized RX/TX
packet
Ring descriptors

Add TX packet to SCBD

Add Compare
SCBD

Page 13 Intel Confidential

Scoreboards pt2

 RX Flow (PHYx -> HOSTx)

– Expected is created by the Cadence BFM when it detects that it has sent a
packet to the PHY.
– Actual is created by a ring monitor (gbe_rx_ring_monitor_seq) that watches
the RX ring memory location and reads the RX packet out of memory (via
the system manager) when the descriptor done bit is set in a descriptor.
– Actual HOSTx is calculated by the descriptor ring that has captured the
packet.
– Actual PHYx is determined using the address lookup function.
 Other flows
– MNG: packets going through the RMII interface. (RMII is a cadence BFM)
There are paths between MNG and both PHYx and HOSTx
– Loopback: In loopback mode the scoreboard is modified to HOSTx->HOSTx

Page 14