You are on page 1of 53

Tile ACR training

June 2011
Intended for regular training on Thursdays,
after Tile Operations Weekly Meeting
1 Friday, July 29, 2011
Material we will cover

How to start your shift


-
Arrival time and Tile desk
-
Interaction with previous shifter and shift leader
-
Applications to open and sign on

Monitoring:
-
Tile busy
-
Tile trips and status
-
Data quality: digital errors

Calibrations

How to write useful e-log entries

When to call the experts

Other information
DAQ = Data AcQuisition
DQM = Data Quality Monitoring
DCS = Detector Control System
2 Friday, July 29, 2011

What to check before taking shifts (I)
To access the Atlas Control Room (ACR) you need :
Safety courses 1, 2, 3, 4A
ATLAS Control room ATL_CR
Check your access detail sheet on EDH
https://edh.cern.ch/Document/General/ACRQ
It should look like :
More informations on access :
https://twiki.cern.ch/twiki/bin/view/Atlas/TileCalShifterPages
For newcomers : check
your access rights at least
2 weeks before your shift!
Request your access via EDH. More information in this twiki
https://twiki.cern.ch/twiki/bin/viewauth/Atlas/OtpAtlasControlRoomShifts
For newcomers: check
your access rights at
least 2 weeks before
your shift!
3 Friday, July 29, 2011
Exercise Self-training

Minimum of 2 shadow shifts, where you ask to be


shown how things are done.
-
Take shadow during the morning shift: ~9am
-
What to do when Tile is busy
-
What to do after a module trips

This presentation is a subset of what you should read


here:
https://atlasop.cern.ch/twiki/bin/view/Main/
NewTileCalorimeterOperationManualShifter
4 Friday, July 29, 2011
Start of shift
You sit here
He controls ATLAS
partition
Shift Leader:
Ask him/her about
the plan for today
He knows about
L1Calo trigger
Arrive 10 min before your
shift starts, we HAVE to talk
to the previous shifter
5 Friday, July 29, 2011
Start of shift

Ask the previous Tile Shifter for a brieng and the Shift Leader for the plan of
the day

Log-in to RunCom.

Open (if not already open) the following:


-
GeneralATLAS Logbook. (Login as yourself.)
-
Whiteboard:
https://atlasop.cern.ch/twiki/bin/view/Main/TileCalorimeterWhiteBoard
-
TileCalDCSTile DCS Alarms
-
TileCalDCSTile DCS FSM
-
TileCalL1Calo Map
-
TDAQDAQPanel
6 Friday, July 29, 2011
7 Friday, July 29, 2011
FSM panel
Legend
Clickable / Browsable!
8 Friday, July 29, 2011
DCS Alarm Panel

Voltage trips

Temperature alarms

Yellow = warning.

Often some value ickers in/out of legal range.

Browse the source of the warning in FSM. Copy-paste it to the e-log.


(Screenshot can be taken using snapshot linux tool)

Red = Alarm!
https://atlasop.cern.ch/twiki/bin/view/Main/ShifterToolsDCS
9 Friday, July 29, 2011
L1Calo Map
Tile Modules
Masked
Tile Modules/
Channels
without signal
A side C side
10 Friday, July 29, 2011
L1 Calo Map
C side A side
11 Friday, July 29, 2011
Partition

Its an environment, appropriately congured, in which data are


taken. It has a name, and it uses some parts of ATLAS.

Partitions can exist in parallel, as long as they dont try to use the
same part of the detector.
-
TILE partition: We use it for calibrations. You control it!
-
ATLAS partition: Run Manager uses this for collisions,
cosmics, or any other combined runs. You only watch.
Examples: ATLAS partition, TILE partition.
12 Friday, July 29, 2011
Monitor/Start Partition
MRS
BUSY
OHP
DQMD
...
DAQPanel: Set partition info & laun
1
2
3
1. Congure, from whiteboard.
2. Read Info.
3. Launch the panel you need.
13 Friday, July 29, 2011
How to monitor ATLAS partition?

Congure DAQPanel according to parameters in the whiteboard.

Click Monitor Partition.


14 Friday, July 29, 2011
https://atlasop.cern.ch/twiki/bin/view/Main/ShifterToolsMonitorPartition
The TDAQ software (IGUI)
15 Friday, July 29, 2011
Look out for...

Load Panels TileIguiPanel to see how Tile is doing.

Is Tile well-congured? That will depend, but typically for physics


we use Reconstruction, and not Raw Data, especially if the L1
event rate is over ~30-40 kHz.
Conguration
for stable beams
16 Friday, July 29, 2011
Look out for...

MRS ERROR messages. TDAQ DAQPanel MRS


https://atlasop.cern.ch/twiki/bin/view/Main/ShifterToolsMrs
17 Friday, July 29, 2011
Look out for...

BUSY Panel: TDAQ DAQPanel BUSY

Is Tile BUSY? Serious incident. See:


https://atlasop.cern.ch/twiki/bin/view/Main/TroubleShooting#TileCal_is_BUSY
These would be red.
https://atlasop.cern.ch/twiki/bin/view/Main/MiscellaneousTools#Busy_Monitor
18 Friday, July 29, 2011
Tile is busy
19 Friday, July 29, 2011
Look out for...

Is Tile BUSY? Serious incident. See:


https://atlasop.cern.ch/twiki/bin/view/Main/TroubleShooting#TileCal_is_BUSY
Some ROD(s) would be red,
similar to this.
Please re-load this
panel if all crates are in
RED and no busy is
shown in the BUSY
panel
20 Friday, July 29, 2011
Ritmo Panel
21 Friday, July 29, 2011
Re-congure a module
22 Friday, July 29, 2011
DQM and OHP
Data Quality Monitor
Online Histogram Presenter

Keywords
-
Digital errors
-
Occupancy plots
-
Noisy channels
-
Masked channels
-
Known problems (in the whiteboard)
23 Friday, July 29, 2011
24 Friday, July 29, 2011
25
New histogram to monitor digital errors
Last bin is always
current LB
Time when module
was recovered
Fraction of
ANY type of
digital errors
integrates
over LB
One histogram per module and run

Browser:TileGATH-EF->SHIFT->DMUErrors->FracTileDigiErrorsEBA07
25 Friday, July 29, 2011
Back - up
26
TRIP LBC04
5 lumiblocks ago
TRIP LBA45
75 lumiblocks ago
EBC22
Known problem see
whiteboard
LBC03
Small percentage,
just report this in the
e-log
26 Friday, July 29, 2011
27 Friday, July 29, 2011
28 Friday, July 29, 2011
29 Friday, July 29, 2011
30 Friday, July 29, 2011
31 Friday, July 29, 2011
DCS &
Detector Control System

Keywords
-
High Voltage
-
Low Voltage Power Supply (LVPS or fLVPS, where f
is for nger )
-
Power trip!
-
Temperature
-
Alarm Panel
32 Friday, July 29, 2011
FSM panel
Legend
Clickable / Browsable!
33 Friday, July 29, 2011
Alarm screen
34 Friday, July 29, 2011
!"#$%&'(&'$)%*'+,-$+*+./$
01+$23"$4+)1,+'-$'%$'5-$6%..-7(%48-4'$/%8&*-$+48$8.+9-.$
:(-4$%&'(&'$)%*'+,-$(*%'$;%.$+<-6'-8$!"#$
35 Friday, July 29, 2011
!"#$%&'(&'$)%*'+,-$+*+./$012$
3%$'%$45"$(+6-*$+67$8*98:$;&<%6$!"#$%&#'()*$
=.9'-$+6$-*%,$-6'.>$?9'@$+,&-./$+67$+A-8'-7$012B$96798+'-$
8&..-6'$)%*'+,-$+67$%AC-'$)+*&-D$E@-8:$!"#$96$'@-$!"#$%&#'()*$
36 Friday, July 29, 2011
1rlpped Lv examples
A message ln Alarm panel !"#$%&'"#(()* wlll appear!
uCS Lakes auLomauc acuons and recovers Lv of Lhe Lrlpped modules
- lf module falls Lo sLarL +&,-). consecuuvely, auLomauc acuon ls .'/(()*&
AuLomauc recovery ls only for Lhe Lv.


37 Friday, July 29, 2011
Calibrations

Tile Calibration: TileCal TileCalibration


-
Pedestal, CISmono, CISscan, laser-,
laser LG, laser HG.

L1Calo: TileCal Level1 L1 Calibration


https://atlasop.cern.ch/twiki/bin/view/Main/TileCalibrationRuns
Manuall Step-by-step instructions:
Check schedule in Whiteboard Check schedule in Whiteboard Check schedule in Whiteboard Check schedule in Whiteboard Check schedule in Whiteboard
MONDAY TUESDAY WEDNESDAY THURSDAY FRIDAY
L1Calo Tile Calibration TileCalibration L1Calo
38 Friday, July 29, 2011
To take L1Calo calibration
39 Friday, July 29, 2011
To Take Tile Calibration

TileCal TileCalibration
Monitoring is not enable so,
in case histograms need to
be check you should take
manually calibration
40 Friday, July 29, 2011
We use TILE partition for calibrations
Conguration from Whiteboard.
Her we dont just monitor; we START the partition.
Step-by-step in the twiki. Please skim it.
41 Friday, July 29, 2011
To log or not to log?
When to write an e-log:
End of each run.
Include detector and data quality information.
For any significant problem requiring the intervention of an expert.
Power cycle is a significant problem, please report this
At the end of your shift, make a summary.
Rough chronology of what was done, new problems, general comments.
The goal is to have one e-log report per run for routine operations, one
shift summary, and additional posts only in case of serious problems.
E-log TEMPLATES on the whiteboard
https://atlasop.cern.ch/twiki/bin/view/Main/
NewTileCalorimeterOperationManualShifter
42 Friday, July 29, 2011
Hot issues

100% Tile Busy begin of a run:


-
Could be due to missing ECR at the start of run.
-
Tile will be busy after ~16000 events at Level1.
-
Faster solution of problem:

Stop / Un-congured (Tile) / Start run

Try at least 2 times if doesnt work call Tile DAQ


expert
-
Medium term solution is to update the software to
avoid this
43 Friday, July 29, 2011
HOT issues

Module Trips:
-
Look into FSM and ALARM panel to identify if a module trips and auto-recovers.
-
Check in FSM if module is ON again.
-
Check MRS for auto-reconguration is launched (with 3-5 minutes after module is
ON).
-
Check L1Calo for auto-reconguration succeeds.
-
Check RITMO for increasing number of discard events, if that is the case:

re-congure again:

DAQ Panel Tile Stopless Recovery

if still discarding events:

Check OHP histograms: Browse TileGather DMUErrors


FracDigErrorsXXX . If more than 20% in Last lumiblock (number 0 in plot):

Call DCS on-call to ask for a power-cycle


-
Please, write all this information in your RunSummary e-log for all the trips in the run.
44 Friday, July 29, 2011

If MRS messages from TileLaserRCD complaining about absence of laser TT


Lutin detecter that although the number of laser pulses emitted is changing, the
number of Laser Trigger Types received by LASTROD has not changed for a long
time

Check in DAQPanel TriP (TriggerPresenter) the L1_CALREQ2 PreScale


value:
-
if PS = -1 , it is trigger problems, there is nothing to do (except
complaining)
-
if PS=1 then:
-
Laser should be disable (tell Run Control Shifter, see next slide for
emergency stop of laser)
-
SHAFT conguration should be checked (send an e-mail to laser
experts)
HOT issues: LASER
45 Friday, July 29, 2011
!"##$"%$!" &'()*+,-./&01(2-342. 5
How to disable the laser in a run ?
!
On the Tile FSM, if you log in, there is a
button "Laser stop" (or similar)
!
Push the button:
"
Laser will not be fired anymore
"
A stopless removal will be triggered to
remove the associated ROBlN
46 Friday, July 29, 2011
When things go wrong

Be prepared!

https://atlasop.cern.ch/twiki/bin/view/Main/TroubleShooting
47 Friday, July 29, 2011
More information
48 Friday, July 29, 2011
TileLBA_resetshafts
This application is
ABSENT during
RUNNING
is OK it should
be like this
49 Friday, July 29, 2011
Masking a super-drawer
50 Friday, July 29, 2011
ECR Mismatch
51 Friday, July 29, 2011
TileCal
52 Friday, July 29, 2011
Basic electroncs
Basics of Tile Electronics
In order to properly understand the various problems that can occur in TileCal, it is important to understand the basics of the TileCal front end
electronics. In case of a trip of a module or barrel, the shifter must be able to identify the nature of the trip and take appropriate action. The
nature of the trip is given by the error messages in the DCS alarm panel and the colored status display on the DCS.
Each TileCal drawer has a High Voltage (830V) input that provides the voltage bias for all pmts in that drawer.
In addition, low voltage is supplied to each drawer via a two-tier system:
1. A 200 V Bulk Power Supply channel serves four tile drawers. These are the represented by the innermost ring of each barrel in the DCS
schematic.
2. The fLVPS (nger Low Voltage Power Supply) transforms the 200V into 8 different output voltages necessary for the functioning of the
TileCal Front-End electronics inside Super-Drawer(S-D). The fLVPS provides two sets of voltages: a low voltage for the readout
electronics (digitizers and motherboard) on on side of S-D and the low voltage for the High Voltage distributor system on the other side of
S-D. These two sets - "MB_side of fLVPS" and "HV_side of fLVPS" has to be distinguish / mentioned in the case of fLVPS trip.
Given this, drawer trips can be classied as follows:
Trip of the HV input channel to the TileCal drawer. One drawer becomes nonoperational and must be restarted by the DCS expert.
Trip of the fLVPS. One TileCal module becomes nonoperational and must be restarted by the DCS expert and in then the case of "MB_side
of fLVPS" recongured for data taking by the DAQ expert.
A drawer can still be operated in so called "emergency mode" if the "HV_side" of the fLVPS is tripped.
Trip of the 200V power supply. In this case, four drawers are made inoperable and each one must be treated as in a fLVPS trip. The DCS
expert must restart the 200 V Bulk Power Supply channel and fLVPS in all 4 drawers and the DAQ expert must recongure all 4 drawers for data taking.
53 Friday, July 29, 2011