Hardware & Firmware work in the UK

In the UK we are focused on the phase 1& phase 2 upgrades.

 CPM
 High-Speed demonstrator
 Multi-Gbps PCB simulation

08/28/2023 Weiming Qian On behalf of UK 1

 CPM Readout

Upgraded Hit-Merger sends timing alongside data (Clock Forwarding)

encoded as the combined Clock/Parity signal

A PCB modification needed to route clock onto FPGA’s clock input

pins:
25 160 MHz ‘L’ CMM
Merger
via Backplane
‘L’
Serialiser
‘1’ Clock

Serialiser
40 MHz DAQ
Trigger DAQ ROC
G-Link Readout
Towers 40 MHz

Serialiser Clock
‘20’ 25
Merger ‘R’ CMM
‘R’ via Backplane

Scheme to be tested

08/28/2023 Weiming Qian On behalf of UK 2

 Upgraded CPM Readout – G-Link Bandwidth
Proposed Upgrade Scheme, 4x Hits: 3 bits -> 12 bits (+9 bits)

Last First
92 91 80 9 0
Parit Hits TT-D2 TT-C2 TT-B2 TT-A2 TT-D1 TT-C1 TT-B1 TT-A1
y

Threshold/ET/ROI Link Serial 8 bit TT

d11...d0 Ready Error d7...............d0
data G-Link

One data stream from each of the 20 Serialisers

40 MHz

But 8 bits are duplicated, so : + 9 bits - 8 bits = + 1 bit

08/28/2023 Weiming Qian On behalf of UK 3

 CPM – Possible Hit Merger Formats

CPM Output format 0.1  0.1, 6b cluster ET

P P P P P P P P P P P P P P P P Threshold bits ROI 1 (8B)
1L 1R 2L 2R 3L 3R 4L 4R 5L 5R 6L 6R 7L 7R 8L 8R
Cluster ET ROI 1 (6b) FP 1(2b) Cluster ET ROI 2 (6b) FP 2(2b) Threshold bits ROI 2 (8B)

Cluster ET ROI 3 (6b) FP 3(2b) Cluster ET ROI 4 (6b) FP 4(2b) Threshold bits ROI 3 (8B)

Cluster ET ROI 5 (6b) FP 5(2b) Threshold bits ROI 4 (8B) Threshold bits ROI 5 (8B)

CPM Output format 0.2  0.2, 8b cluster ET

P P P P P P P P P P P P P P P P Threshold bits ROI 1 (8B)
1L 1R 2L 2R 3L 3R 4L 4R 5L 5R 6L 6R 7L 7R 8L 8R

Cluster ET ROI 1 (8b) Cluster ET ROI 2 (8b) Threshold bits ROI 2 (8B)

Cluster ET ROI 3 (8b) Cluster ET ROI 4 (8b) Threshold bits ROI 3 (8B)

Cluster ET ROI 5 (8b) Threshold bits ROI 4 (8B) Threshold bits ROI 5 (8B)

Physics performance is very similar – so selected on implementation

08/28/2023 Weiming Qian On behalf of UK 4

 CPM – Possible Hit Merger Formats
Encoded Isolation
Encoded isolation instead of Thresholds – reaction to Simulation Studies
CPM EM. Output format 0.1  0.1, 8b cluster ET
P P P P P P P P P P P P P P P P EM Iso1 Had Iso1 HadVeto1 ROI FP1
1L 1R 2L 2R 3L 3R 4L 4R 5L 5R 6L 6R 7L 7R 8L 8R (2b) (2b) (2b) (2b)

Cluster ET ROI 1 (8b) Cluster ET ROI 2 (8b) EM Iso2 Had Iso2 HadVeto2 ROI FP2
(2b) (2b) (2b) (2b)

Cluster ET ROI 3 (8b) Cluster ET ROI 4 (8b) EM Iso3 Had Iso3 HadVeto3 ROI FP3
(2b) (2b) (2b) (2b)

Cluster ET ROI 5 (8b) EM Iso4 Had Iso4 HadVet04 ROI FP4 EM Iso5 Had Iso5 HadVeto5 ROI FP5
(2b) (2b) (2b) (2b) (2b) (2b) (2b) (2b)

CPM Had. Output format 0.1  0.1, 8b cluster ET

P P P P P P P P P P P P P P P P EM Iso1 Had Iso1 ROI FP1
1L 1R 2L 2R 3L 3R 4L 4R 5L 5R 6L 6R 7L 7R 8L 8R (2b) (2b) (2b) (2b)

Cluster ET ROI 1 (8b) Cluster ET ROI 2 (8b) EM Iso2 Had Iso2 ROI FP2
(2b) (2b) (2b) (2b)

Cluster ET ROI 3 (8b) Cluster ET ROI 4 (8b) EM Iso3 Had Iso3 ROI FP3
(2b) (2b) (2b) (2b)

Cluster ET ROI 5 (8b) EM Iso4 Had Iso4 ROI FP4 EM Iso5 Had Iso5 ROI FP5
(2b) (2b) (2b) (2b) (2b) (2b) (2b) (2b)

08/28/2023 Weiming Qian On behalf of UK 5

 CPM Firmware

Preparing design environment for developing firmware in Ph1.

Moving projects from Mentor HDL Designer to Xilinx ISE 10.1:

Tidy-up designs. Revision. Better Test-Benches (ModSim).

Hit-Merger New project created in ISE

DAQ G-Link Readout Moved – checking files/libs

ROI G-Link Readout To do

CP Chip To do

08/28/2023 Weiming Qian On behalf of UK 6

 CPM Firmware Timescales

Hope to produce code for FPGAs using just ISE (with same
functionality as current designs) by the end of 2011.

Compare behaviour of new code with current version:

Deep Simulation
B’ham Testrig -> CERN Point 1 ?

Have FPGAs debugged & ready for early 2012 cosmics running

If time, add new threshold-ing scheme to CP FPGA also.

Isolation & energy thresholds combined,
say EM16I and EM20 -> one hit.

Aim to have Phase1 upgrade designs ready by end of 2013

for big shutdown.
08/28/2023 Weiming Qian On behalf of UK 7
 Motivation for High-Speed Demonstrator
• The use of multi-Gbps data links in these upgrades is essential to handle the volume of data that needs to be
transported and shared between processing modules.

• Signals in multi-Gbps range present huge challenges to PCB design

– Signal reflection
– High-frequency attenuation
– Crosstalk
– Clock jitter
– Power decoupling difficulty
– All above problems entangled together reducing EYE opening

• We have limited experience of multi-Gbps PCB design

– eg, RAL PCB DO designed XFEL FEM with 3.25 Gbps,
very short links (a few centimetres), using rules of thumb

• In industry it is standard to model and simulate PCBs during the design process to ensure required signal integrity is
achieved

• We wish to educate ourselves with a dedicated demonstrator, that doesn’t have lots of expensive components

08/28/2023 Weiming Qian On behalf of UK 8

 High-Speed Demonstrator Overview (1)

• Primary purpose of module: gain experience of

– Multi-Gbps PCB design Op. Tx/Rx e f
– Multi-Gbps PCB simulation
– Comparison/Correlation of simulation & hardware a Custom
Op. Tx/Rx d connector
FPGA
• FPGA used as a source/sink of multi-Gbps serial signals
SNAP 12
XC6VHX255T
– 24 x GTH transceivers, 9.9 Gbps–11.1Gbps TX
c
– 24 x GTX transceivers, 0.5Gbps–6.6Gbps h 24GTX +
SNAP 12 24GTH
RX

• Implement PCB paths of a variety of topology, lengths, g

vias, and etc
b
• Propose building three of these modules, to enable
backplane transmission tests
FPGA Configuration CPU Header
Logic

ATCA
• Aim is to allow as many different path types as possible to
backplane
be tested connector
– FPGA-SMA
– FPGA–transceiver Clock Power
– FPGA–backplane–FPGA, and etc

Demonstrator Module
08/28/2023 Weiming Qian On behalf of UK 9
 High-Speed Demonstrator Overview (2)

• Secondary purposes
– Gain experience of ATCA platform
• IPMI – Intelligent Platform Management Interface
– Provide platform for exploring module control in ATCA (e.g. Ethernet)
– Provide source/sink of test data for subsequent demonstrator modules

• With regard to these, spec includes:

– SNAP12 transmitter + receiver
• Use 12 of 24 GTX transceivers
– CPU Header
• Accommodates LPCXpresso MBED CPU module.
• Connections…
– Ethernet to ATCA bus,
– I2C & General I/O  FPGA,
– USB  USB header,
– ADC  voltage monitoring.
• Not necessary for primary operation of module

08/28/2023 Weiming Qian On behalf of UK 10

 High-Speed Demonstrator Design Methodology

• The goal of this exercise is not just to produce hardware

– Producing a board that does/doesn't work at multi-Gbps will tell us little if we don't understand
how we got there.

• Also to equip ourselves with capability of producing working multi-Gbps PCBs in reliable systematic
fashion:
– Simulation
– Extraction of electrical model from hardware
– Feedback of electrical model into simulation
– Learn from our successes and failures

08/28/2023 Weiming Qian On behalf of UK 11

 High-Speed Demonstrator Status

• Conceptual design review held June 2011

– documents available at https://indico.cern.ch/event/HSDReview110615

• Schematic entry underway

• Currently designing ATCA Power system

– ATCA: board negotiates with shelf controller for power supply & backplane access
– High Speed Demonstrator hardware capable of implementing full ATCA specification
• Spartan FPGA used as board power management controller (IPMC)
• Based on XFEL Train Builder board (RAL)
– Manual over-ride enables quick commissioning of other aspects of board

• On schedule for manufacture beginning of next year

08/28/2023 Weiming Qian On behalf of UK 12

 Scaling of Physical Size vs. Scaling of time

 Equivalent in passive and lossless circuit system

• 100 times faster ~100 times bigger (X, Y, Z)

08/28/2023 Weiming Qian On behalf of UK 13

 Speed regimes of high-speed link (1)
 Up to 1Gbps
 Signal edge physical length on PCB > 5cm
 Signal loss over PCB tracks is relatively small
 PCB via effect on signal integrity is negligible

 Tight impedance control and good termination

• PCB tracks
 Channel simulation is optional
• ~1000 bit simulation will suffice
 Link validation by direct EYE measurement

 2~5Gbps
 Signal edge physical length on PCB ~ 1cm
 Frequency dependent loss on PCB causes significant ISI
 PCB via begins to distort signal integrity

 Tight impedance control and good termination

• PCB tracks , connectors and possibly long vias
 Tx pre-emphasis (Rx equalization optional)
• Linear Time Invariant (LTI)
 Open EYE achievable at Rx input pin
 Channel simulation is important
• ~1,000,000 bit simulation needed to qualify the link performance
• Time domain simulation + statistical analysis
 Validation by simulation/direct EYE measurement

08/28/2023 Weiming Qian On behalf of UK 14

 Speed regimes of high-speed link (2)
 Over 6Gbps
 Signal edge physical length on PCB ~ mm
 Signal frequency dependent loss on PCB causes huge ISI
 PCB via has big impact on signal integrity

 Tight impedance control and good termination

• PCB tracks, connectors, vias
• Differential skew
 Tx multi-tap equalization
 Closed EYE at Rx input pin
 Open EYE hidden inside the Rx after equalization
 Rx adaptive equalization + Decision Feedback Equalization (DFE)
 Non linear, time variant
 Channel simulation is essential
 ~10,000,000 bit simulation needed to qualify the link performance
 Time domain simulation + statistical analysis
 Algorithmic Model Interface for Rx DFE and CDR
 Validation by simulation/BER measurement

+
08/28/2023 Weiming Qian On behalf of UK 15
 Challenges of multi-Gbps Serdes channel simulation
 Silicon Modelling
 Can model complex analogue circuit of SERDES
 FFE, DFE, CDR
 Simulation Speed
 Can simulation enough bits within reasonable amount of time to predict BER at required level.

SPICE Lossy RLGC SPICE

IBIS Coupled differential pair IBIS
IBIS-AMI S-parameter IBIS-AMI

08/28/2023 Weiming Qian On behalf of UK 16

 Simulation Model

 Spice – Simulation Program with Integrated Circuit Emphasis (~1970)

 Accurate
 Complexity – structural transistor level
 Extremely slow
 ~100bits/hour
 IBIS – I/O Buffer Information Specification (~1994)
 Behaviour level – V/I, V/T table
 Much faster than Spice
 ~1000bits/minute
 Can not model complex analog circuit
 Equalization, DFE, CDR
 IBIS-AMI (IBIS Algorithmic Modelling Interface) ~2008
 IBIS for analog front end
 AMI for FFE, DFE, CDR
 Executable .dll
 Fast
 1 million bits/minute

08/28/2023 Weiming Qian On behalf of UK 17

 Limitation of time domain simulation

 No model can be simulated fast enough in time domain to make multi-trillion-bit (10 12) analysis possible
 Needs to extrapolate the limited time domain simulation histogram to predict BER
 Make simple assumption of channel PDF – usually Gaussian
 Not valid most of time

 Statistical Method – FastEye (Mentor)

 Channel Impulse Response
 Convolution
 Stressed BER
 Worst-case stimulus patterns
 Stressed BER rescaling

 Limited to LTI (Linear Time Invariant) channel

08/28/2023 Weiming Qian On behalf of UK 18

 Simulation Tool

 HyperLynx SI GHz (Mentor Graphics)

 Xilinx provides Eldo (Mentor’s version of spice) model for GTX (up to 6.5Gbps)
 Xilinx also provide IBIS–AMI model for GTX
 Supported by HyperLynx
 Xilinx is to release simulation model for GTH (up to 11Gbps) imminently.
 Allegro PCB SI (Cadence)
 No model support from Xilinx yet
 Not support IBIS-AMI model well
 Within the tool suit used by RAL drawing office
 Other tool
 SiSoft QCD
 ANSYS SIWave

08/28/2023 Weiming Qian On behalf of UK 19

 Simulation flow (pre-layout)
 PCB layer stackup
 Need information from Drawing Office
 Via padstack
 Need information from Drawing Office
 Channel topology

08/28/2023 Weiming Qian On behalf of UK 20

 Simulation flow (pre-layout)
 S-parameter model of the channel
 Validity check
 Passivity
 Causality
 Frequency range
 Model resolution
 SDD(2,1) differential insertion loss resonance check
 Time domain simulation sensitive to resonance
 SCD(2,1) common to differential mode conversion check

3.125Gbps 5Gbps 6Gbps

08/28/2023 Weiming Qian On behalf of UK 21

 Simulation flow (pre-layout)

 “What-if” exploration
 PCB layer stackup
 Material loss tangent
 Power/Ground/Signal layer order
 Differential pair geometry and impedance
 Differential via padstack
 Via impedance, stub length
 Channel length
 Differential skew
 Connectors
 Jitter
 Bit rates
 Tx/Rx Equalisation parameters
 Xilinx Virtex-6 Serdes Tx/RX settings:
• Transmitter settings = (16 amplitude) x (16 pre-cursor) x (32 post-cursor) = 8,192
• Receiver settings = (8 EQ) x (2 DFE on/off) = 16
• Total (Tx) x (Rx) = 131,072
 Protocol coding
8B/10B, PRBS (bit order 7, 15 or ?)
 Worst-case bit sequence
Acceptance Criteria
 Eye mask definition
 OIF-CEI standard
 Electrical and Jitter Interoperability agreements for 6G+ bps and 11G+ bps I/O

08/28/2023 Weiming Qian On behalf of UK 22

 Simulation flow (post-layout)

 Channel validation
 Crosstalk simulation between channels
 Power Integrity simulation
 DC
 AC
 SI and PI co-simulation
 Signal vias generate/pickup noise between planes
 Enhanced accuracy

08/28/2023 Weiming Qian On behalf of UK 23

 What about optical transceivers?

08/28/2023 Weiming Qian On behalf of UK 24

 What about optical transceivers?

 It is almost impossible to do a full channel simulation including optical

transceivers.
 Lacking (good) models of optical transceiver (laser driver, VCSEL, PIN, TIA)
 Optical transceivers are not passive Linear Time Invariant devices
 Not compatible with IBIS-AMI flow

 Can do partial simulations

 Serdes  optical Tx
 Optical Rx  Serdes

 Follow Industry standard

 SFP+ specification (SFF_8431)

08/28/2023 Weiming Qian On behalf of UK 25

 SEP+ Specification SFF_8431

 Specification for Enhanced 8.5 and 10 GBd Small Form Factor Pluggable
Module “SFP+”
 This specification defines the electrical interfaces and their test methods
between the SFP+ module and host board for operation up to 11.1 GBd

08/28/2023 Weiming Qian On behalf of UK 26

 SFF_8431

 Defines Transmitter output Eye Mask

 Defines Receiver input Eye Mask
 …

08/28/2023 Weiming Qian On behalf of UK 27

 Will this work?
1U = 1.75 inches = 4.4 cm
8U = 35.86 cm
ATCA RTM

8Ux280mm 8Ux70mm
SNAP12

 Maximum PCB channel length ~ 30 + 15 + 30 = 75 cm!!!

 Optical transceiver does not have built-in CDR, Pre-emphsis, equalization

08/28/2023 Weiming Qian On behalf of UK 28

 Summary on high-speed PCB simulation

 PCB simulation becomes increasingly important as signal speed marches into

Multi-Gbps range.
 PCB simulation is a very useful and powerful tool for Multi-Gbps PCB design
provided that:
 Proper simulation tool
 Proper models
 Properly used
 Properly interpreted
 We have learned a lot about multi-Gbps PCB simulation tool (MentorGraphics
HyperLynx SI GHz), and more importantly the underlying Signal Integrity
knowledge.
 We are ready to integrate PCB simulation into multi-Gbps PCB design flow
@RAL.
 We still have a lot more to learn in multi-Gbps PCB simulation.

08/28/2023 Weiming Qian On behalf of UK 29

 QUIZ

 Given no constraints, how to make current CP/JEP working @10Gbps?

 What is the name of the stadium?

08/28/2023 Weiming Qian On behalf of UK 30