Initial thoughts on “Pulsar in SRC mode” possibility

Ted Liu, Dec. 03, 2003

Some discussions with:

SRC side: Petar/Lester/Steve/Satyajit
General: Jonathan Lewis and Peter Wilson

Only discuss the technical feasibility:

Is it possible to use Pulsar as SRC IF indeed needed?

What is involved at hardware level ?
What are the options ?
What are the advantages ?
What are the concerns ?  need experts to point out !
The devil is in the details…
 Personal comments: a different view on Trigger/DAQ upgrade
 Petar is looking at the bright side, so I look at the dark side

• with both SVT and L2 upgrade & UDPS, we MAY be able to

run at 40-50KHz (RunIIa-RunIIb spec), at least there
is a fair chance at low luminosity

• at higher luminosity, with multiple interactions per BX,

there is a fair chance we may no longer be able to run
at 40KHz… if we are really unlucky we may still only be
able to run at <~20KHz if occupancy doesn’t scale as
one would like even after all SVT&L2 upgrade …

Enough safety margin && flexibility is crucial,

every micro-second in raw speed may matter
 be prepared, and invest in raw speed&flexibility early
 Some historic comments on Pulsar-SRC
• Last May, after Pulsar design was finalized, Petar
& Lester asked me to see if we can make Pulsar
compatible for SRC upgrade…
• I spent a few weeks to take care of that. In particular,
the RF clock & backup local clock (see next slide).
But the P3 connection was too late to change in order
to reuse the SRCTM
(bottom line: SRCTM cannot be reused if using Pulsar).
• Since then, there was no more discussion on using Pulsar
to upgrade SRC…
• We are now looking into this possibility again (requested
by Petar & Jonathan). maybe this time we are a bit more serious?

Backplane issue: first few empty slots of VRB P3 backplane

can be split (from the rest), or Pulsar-SRC
have to live in a different crate nearby…
Pulsar RF clock & local clock:
Use zero-delay glitch-free clock multiplexer:
ICS581-01
(found by Mircea) VME
chip
• local clock
• Tevatron RF clock DataIO FPGA 1
The two clocks
are arranged in
such a way that DataIO FPGA 2
when the RF clock
is lost, the local
clock takes over
in a few clock ticks. Control FPGA
The output clock
goes to PLL clock
pin on each FPGA)
 Designed specifically to meet the unique requirement for SRC
 Current SRC interfaces

Glink
To FIB
VRB
FIB
Beam Return
structure LVDS

RF Current SRC Current

clock SRCTM
TSI(lvds)
TS(Taxi,
Optical)

RED: already existed in Pulsar design

Black: can be absorbed on a new TM, or 1-2 mezz cards

Note: all LVDS interfaces use similar connectors

VRB

Current SRC SRCTM

 Pulsar-SRC: A few options to consider

• Option A: Simplest & cleanest

 rebuild the SRCTM for Pulsar, and add

TSI/beam structure/VRB interfaces on new TM.

 Pulsar Control FPGA “sees&controls” everything

• Option B & C: not as clean for VRB, see backup slides…

 Option A: build new TM • copy the same FIB
interface design …

Pulsar
TM is
double
New SRC
-width
….

P3:117 signals

• add TSI(LVDS)/
TS(Taxi optical)
In this case, Control FPGA sees and controls ALL. Beam structure/VRB…
DataIO FPGAs can be used for house-keeping work on the other side…
 Backplane may not (1) Split the backplane,
be an issue: remove the empty slots

Pulsar-SRC and SRC can

co-exist…can switch back
to old SRC easily

NO need to touch
the old SRC, VRB backplane
Only cables …

 ideal for
commissioning… (2) Install P2 style
backplane for
Pulsar + new TM.
(3) Pulsar-SRC and SRC
can either live next
to each in another VRB
 Option A: build new TM (cont.)

New SRC

Advantages:
• simple & cleanest from firmware/hardware point of view;
• DataIO FPGAs may be used for house keeping tasks;
• mezzanine card slots in front can be used for
expansion/diagnostics/flexibility
 Option A: build new TM (cont.)

C
P
New SRC
U

diagnostics: one possible example? just a thought last night…

• can pump information into a PC and receive PC results via
either Gigabit Ethernet (round trip time ~ 30us), or
SLINKPCI (round trip ~ a few us): maybe one can try
 resonance detection? either in firmware or software
Recall that Pulsar was designed for fast decisions using a PC
(firmware/hardware/software infrastructure/expertise all in place)
 Other comments:
A few things to consider:

• firmware design issues in each case: option A seems easiest

• hardware: option A is cleanest
• flexibility: Option A seems more flexible
• how to commission ?  think backwards!
• Option A seems the best from firmware/hardware point of view

Hardware needs seem to be simple for all options. However,

one may build a Glink Rx mezzanine card as data sink for testing
purpose. Much easier this way to develop & tune the upgrade
standalone…Crucial also for Glink Single-Bit-Error
testing (down to 10**-14) …
Also keep in mind that L2-TSI interface (i.e. L2 raw decisions),
all raw L1 trigger bits as well as SVT interface all still
available on Pulsar in case someone has good ideas to use them.
 Top 10 reasons why Pulsar is
potentially useful for SRC upgrade
• Pulsar motherboard is designed to be compatible with SRC
• Can be tailored to meet all SRC interface needs (no kludges)
P3 backplane doesn’t seem an issue
• Can leave old SRC untouched, can switch back easily,
can develop & tune the upgrade standalone
• One FPGA “sees&controls” all: firmware design easier/cleaner
Low level firmware can be shared, under same CVS…
• Fast large SRAMs (two 128K x 36) available on board
• Mezzanine card slots can be used for expansion/diagnostics
• Can communicate with a PC in semi “real time” (~10 us): e.g.
resonance detection: extra safety protection for our silicon
• L2 raw decision/L1 raw trigger bits/SVT interface available
• Many hardware/firmware expertise/modern firmware tools…
• Pulsar-SRC may be also used as a pulser for SVT …
(i.e. fake SVX fiber data into Hit Finder)
for experts: Do we have good motivation(s) for SRC upgrade ???
 Back up slides
• Option B
• Option C
• the rest of slides are mostly from my SVT talk,
keep them here for background information,
• also added Pulsar/PC round trip timing measurements (by Upenn)
using SLINK-PCI (slide 30-31), with web link to timing
measurements with Gigabit Ethernet

More detailed  information can be found at:

http://hep.uchicago.edu/~thliu/projects/Pulsar
 Pulsar-SRC: A few options to consider
• Option A: Simplest & cleanest
 rebuild the SRCTM for Pulsar, and add
TSI/beam structure/VRB interfaces on new TM.
 Pulsar Control FPGA “sees&controls” everything

• Option B: Simple & clean

 use Pulsar TM (AUX) card. Build 1-2 types of
simple mezzanine card, with one Glink + LVDS + TAXI.
 Pulsar DataIO FPGAs handle FIB interfaces, and
Control FPGA takes care the rest

•Option C: Simple & clean (avoid the P3 backplane issue completely)

 Do not use P3 and TM. Build 2 types of mezz cards:
one for FIB interfaces (2 Glink per card), the other
for the rest. One DataIO handles FIB interface, the other
takes care the rest (VRB goes from front panel, or absorb
also on mezzanine card…not as clean!)
 AUX Card

lvds
glink
lvds
BEAM
glink stru
lvds
glink TSI
LVDS/
lvds optical
glink

Option B:
Have FIB interface upfront,
and the rest on AUX card;
With 1-2 mezzanine card(s)
• DataIO FPGAs take
care the FIB interface;
• Control FPGA for the rest
 Option B: mezzanine card needs
 only need 1 or 2 types of mezz card(s) design

1-2 types mezz card: possible absorbed in one PCB design

FIB return LVDS

GLINK to FIB GLINK

TSI/beam structure LVDS

TSI (taxi, design already
TAXI
exists on Pulsar mezz)
optical
VRB interface can be absorbed on Pulsar AUX card
 If we want to
keep FIB return
glink LVDS, Glink
glink Mezzanine card
glink will be
glink double-width,
Pulsar-SRC
lvds will take 3 slots
lvds in this case.
lvds
lvds/taxi No need for P3.
DataIO FPGAs
do most of the work:
one for FIB interface,
Option C:  
The other for the rest.
Have all interface upfront
VRB can be absorbed
With two types of
on mezzanine card
mezzanine cards
 Pulsar is designed to be fully self-testable:
at board level as well as system level
For every input/output, there can be an output/input,

SVT

SRAMs
PCI

RF clock Gigabit Ethernet

This allows us to develop&tune an upgrade system standalone.

Pulsar design philosophy:

(1) In God we trust, everything else we test;
(2) One for all, and all for one;
 Pulsar design: general purpose board for HEP
3 Altera APEX 20K400_652 (BGA) FPGAs
9U VME
502 user IO pins each
(VME and CDF ctrl signals
are visible to all three FPGAs)
P1
S Mezz card (CMC)
T V Data IO connectors
S T

P2 SRAM
Control/ 128K x 36 bits
SLINK Merger
signal lines

Data IO
L S
P3 V
spare lines 1 T
SRAM

3 APEX20K400-1XV FPGAs on board = 3 Million system gates/80KB RAM per board

2 128K x 36 pipelined SRAMs with No Bus Latency: 1 MB SRAM (~5ns access time)
 Pulsar bottom view

Pulsar top view

 VME interface

• VME interface
VME
is based on the chip
same chip on all
UC boards: DataIO FPGA 1

e.g. GB board
DataIO FPGA 2
• VME interface
is available to
ALL three FPGAs
Control FPGA
• each FPGA Pulsar clocks
has four clock
inputs:
(1) CDFCLK; VME
(2) user defined chip
clock (40 MHz
DataIO FPGA 1
default, for
SLINK
interface w P3)
(2) Algorithm DataIO FPGA 2
clock (PLL,
tested up
to 114MHz);
OR Control FPGA

(4) Tevatron
RF clock
(PLL pin
on FPGAs)
 SVT interfaces
1MB SRAM(2 128K x 36): 4 ns access time, pipelined NoBL SRAM

SRAM: VME
chip
CY7C1350

DataIO FPGA 1
P2 SVT
SRAM inter-comm
Lines(5):
in DataIO FPGA 2 Master&
ta
da Slave
T
SVT input SV
d a ta in
SVT
SRAM
SVT data in Control FPGA SVT
SVT output
SVT data out

2 SLINK
As it is, Pulsar can be used as a powerful GhostBuster board
 Pulsar P3 interface
P3 connection
uses P2 style
connector, VME
117 signals chip
mapped
to all 5 rows.

Signal Pin
map can be
made
compatible 43 signal lines each
with AM board
Control FPGA

All signals P3:

directly visible 117
signal
to Control lines
FPGA
 Pulsar P2 inter-communication lines

•Pulsar has
five SVT VME
chip
style inter-
communica DataIO FPGA 1

tion lines on
P2 visible to P2
DataIO FPGA 2
all three
FPGAs.

Master Control FPGA

and
Slave
 Level 1 & TS interfaces
all Level 1 trigger bits can be directly available to all FPGAs

TS
VME
chip

DataIO FPGA 1
L1
64 bits
SRAM
DataIO FPGA 2

SRAM
Control FPGA

TS interface with Control FPGA

 SLINK/mezzanine interfaces
Each mezzanine card has up to 83 user defined signal lines;
Four mezzanine card slots up front, 2 in the back of crate.
fully compatible with ATLAS DAQ/Trigger
AUX card

mezzanine card
Mezzanine
card slots
mezzanine card
can be used
for expanded mezzanine card
SRAM,
FPGA, P3
mezzanine card
inputs/outputs
etc.
Mezzanine card slots: 5V/3.3V/2.5V power provided by Pulsar
MOAB (Mother of All Boards) Hotlink Tx
for L2 upgrade

Hotlink Rx

Taxi Tx

Pulsar Taxi Rx

SLINK
LSC/LDC
AUX Card (ODIN/HOLA)
ANL
SLINK->GBE
 PC interface: Gigabit Ethernet / SLINK-PCI
or anything one can define on AUX card
• SLINK to Gigabit LSC from ANL, or SLINK LSC/LDC from CERN
AUX card

Netgear GA621

SLINK

Tested with
ANL internal
LSCclock
up to 100 MHz NetGear GA621
Oct. 21
SLINK LSC
• Pulsar is compatible with ATLAS DAQ/trigger: via SLINK
 Round-Trip timing:

Performance with Gigabit Ethernet, see:

http://www.hep.anl.gov/reb/cdf/risto/risto_talk.pdf
 Round-Trip timing measured
& the performance looks good
w/ algo : Blue w/o algos : Red

From CERN
 P2 CDF control signals

•Pulsar sees ALL VME

P2 CDF control chip

signals
DataIO FPGA 1

•Available to
ALL three FPGAs
DataIO FPGA 2

• can even drive

P2 backplane
if necessary
Control FPGA
 Top 10 reasons for Pulsar to be
potentially useful for SVT upgrade
• Pulsar is a grandson of SVT (young & energetic);
• Same P2 inter-communication lines (SVT master&slave);
• SVT input is available to all three FPGAs;
• fast large SRAMs (two 128K x 36) already available on board;
• mezzanine card slots can be used for expansion
(extra SRAM/FPGA, extra inputs/outputs);
• Back of crate (AUX) card can be used for expansion as well;
• P3 interface can be made compatible with AM board;
• as it is, Pulsar can be used as a powerful GhostBuster board;
• with G-Link mezzanine cards, Pulsar can source/sink
SVT system (i.e. SVX fiber data)
• communication within Pulsars and to new L2 decision crate
can be done with high bandwidth S-LINK
 free Pulsars for SVT, this is business within family…
 time to brain storm … see Alex’s talk…
Pulsar Design methodology
A major fraction of the design effort
was dedicated to extensive design
optimization and verification by using
state-of-the-art CAD tools:
• Leonardo Spectrum for VHDL synthesis;
• Altera Quartus II for place and route,
FPGA level simulation
• Mentor Graphics QuickSim for board and
multi-board level simulation;
• Interconnect Synthesis tool for trace and
cross talk analysis;
• IS_MultiBoard tool for signal integrity
checks between motherboard and mezzanine
& AUX cards;

we have carefully simulated the

board(s) before sending out the
prototypes …
Example: Multi-board (9 boards) simulation to verify the design
(top level schematics)

It took 1.5 GB memory on a 2GHz/2GB modern PC

to simulate 9 boards together at the same time
Pulsar Design methodology
This design methodology allowed
us to build flawless prototype
boards
The self-test capability of the board
design made it possible for us to fully
test (all interfaces) the prototype boards
within 6 weeks after assembly.
The prototype boards were also tested
with on board clock speed up to 114
MHz and no problems were found.
So far, Pulsar has been tested with
~ 1Billion events (Tx->Rx) without
single error…
No single blue wire on all five prototype
board (Pulsar + four mezzanine cards)
All firmware (VHDL code) in CVS