You are on page 1of 32

ACME Gen II Software Systems

Experiment Monitoring, Control and Data Acquisition

Daniel G. Ang
ACME Collaboration
Harvard University
danielang@physics.harvard.edu

August 3, 2016

Daniel G. Ang (ACME Collaboration)

ACME Gen II Software Systems

August 3, 2016

1 / 25

Overview
Introduction
Experiment Monitoring
SQL Database and Logging System
Computers

Experiment Control
Data Acquisition
Structure
FPGA Scope VI
Data Streaming

Backup System
Future Plans
Summary and Conclusion

Daniel G. Ang (ACME Collaboration)

ACME Gen II Software Systems

August 3, 2016

2 / 25

Experiment Monitoring: The SQL Database

Hosted on server computer Gandalf (10.16.1.2), located in J162


Accessible manually via Microsoft SQL Server Management Studio
2008, or Microsoft Access
All ACME computers can communicate with the database via the
ACME internal network
Daniel G. Ang (ACME Collaboration)

ACME Gen II Software Systems

August 3, 2016

3 / 25

Experiment Monitoring: The SQL Database

Consists of two databases: LoggingConfigSQL and LoggingLogData


LoggingConfigSQL stores configuration parameters: IP addresses, list
of lasers, computers, Global Variables
LoggingLogData stores the live data: recorded laser wavelengths,
locked/unlocked status, beam source temperature, room temperature,
etc. - all the variables stored in Global Variables table
Daniel G. Ang (ACME Collaboration)

ACME Gen II Software Systems

August 3, 2016

3 / 25

Experiment Monitoring: The SQL Database

All interactions with the SQL Database are done via Logging.vi
Most computer-computer communication is done via the database,
e.g. through the Global Variables table
All VIs interact with the database via the Logging.vi
Daniel G. Ang (ACME Collaboration)

ACME Gen II Software Systems

August 3, 2016

4 / 25

Experiment Monitoring: Logging Workflow

Computer clock synchronization is performed constantly


Over 50 items in Global Variables table are constantly monitored
during an experimental run. (Not counting the switches on the
experimental states!)
Use the above scheme when writing new VIs!
Daniel G. Ang (ACME Collaboration)

ACME Gen II Software Systems

August 3, 2016

5 / 25

Experiment Monitoring: Interacting with the Database in


Custom VIs

Custom sub-palette (found under Programming) of LabView custom


VIs written by Brendon to easily interact with the database
Create a new field in the Global Variables table in the database in
Gandalf and use the above VIs to send SQL commands easily to and
from the database
Daniel G. Ang (ACME Collaboration)

ACME Gen II Software Systems

August 3, 2016

6 / 25

Experiment Monitoring: The Computers

J162 Gimli, or Wavemeter computer: runs wavemeter, Gimli


scanning cavity
Logs measured wavemeter values of various lasers, locked/unlocked
status
Daniel G. Ang (ACME Collaboration)

ACME Gen II Software Systems

August 3, 2016

7 / 25

Experiment Monitoring: Other Computers

J162 Laser Lab 3: Monitors the status of the ULE cavities used to
lock STIRAP lasers
Daniel G. Ang (ACME Collaboration)

ACME Gen II Software Systems

August 3, 2016

8 / 25

Experiment Monitoring: The Computers

G14 Beambox DAQ: Monitors the ion sweeper voltage, G14 room
temperature, humidity, ambient magnetometers via the Agilent
34980A Datalogger
G14 DAQ: Logs the temperature of the cell via the Lakeshore 325
cryogenic temperature controller. Also controls the actual
experimental states (coming up!)

Daniel G. Ang (ACME Collaboration)

ACME Gen II Software Systems

August 3, 2016

9 / 25

Experiment Monitoring: The Computers

G14 DAQ NEW or the FPGA computer controls data acquisition


using the FPGA
G14 Control Fingolfin Multipurpose use computer for any other
custom control VIs, also for testing various instruments
G14 General For browsing and general purpose
Total of 5 main experimental computers and 2 auxiliaries (not
counting data analysis computers)

Daniel G. Ang (ACME Collaboration)

ACME Gen II Software Systems

August 3, 2016

9 / 25

Experiment Control

Primarily done through G14 DAQ.


Three main VIs:
Ablation control, timing: Beambox Timing, Ablation, and Flow
Control.vi, Ablation Mirror and Positioning VI
Other specific control: Main Current and Voltage Control.vi,
Fluxgate Magnetometry.vi, Probulation.vi, microwave control VIs, etc.
General experimental control: Master Run VI.vi, Configure Switch
Sequence.vi

Daniel G. Ang (ACME Collaboration)

ACME Gen II Software Systems

August 3, 2016

10 / 25

Experiment Control: Beambox Timing, Ablation, and Flow


Control

Sends timing sequences to DG645 Pulse Generators to control firing


of YAG ablation laser and polarization switching
Monitors and sets voltage on ion sweeper of the beam box, cell
temperature, buffer gas flow
Daniel G. Ang (ACME Collaboration)

ACME Gen II Software Systems

August 3, 2016

11 / 25

Experiment Control: Beambox Timing, Ablation, and Flow


Control

Previous bug (timing sequence would stop when adjusting cell


temperature) found to be purely programmatic now fixed and working
smoothly
Daniel G. Ang (ACME Collaboration)

ACME Gen II Software Systems

August 3, 2016

11 / 25

Experiment Control: Configure Switch Sequence

This VI generates a switch sequence of experimental parameters to be


fed into the Master Run VI
Daniel G. Ang (ACME Collaboration)

ACME Gen II Software Systems

August 3, 2016

12 / 25

Experiment Control: Master Run VI

Master Run VI reads the switch


sequence, implements it by
launching appropriate sub-VIs
that control different
instruments
With every record, sends an
affirmative signal that launches
the trigger to the FPGA
computer

Daniel G. Ang (ACME Collaboration)

ACME Gen II Software Systems

August 3, 2016

13 / 25

Data Acquisition: Hardware Structure

AND gate ensures that experiment state is ready before data acquisition
begins.
Daniel G. Ang (ACME Collaboration)

ACME Gen II Software Systems

August 3, 2016

14 / 25

Data Acquisition: Mechanics

Polarization switching occurs every 20 ms


Data is recorded for 10 ms (50 % duty cycle) = 1 record.
Take data in batches of 25 records (500 ms), which are
summed/averaged into 1 trace = 1 experimental state.
Daniel G. Ang (ACME Collaboration)

ACME Gen II Software Systems

August 3, 2016

15 / 25

Data Acquisition: Mechanics

64 traces are averaged into one block = 1 measured value of the


EDM.
FPGA VI saves traces in two forms: the individual PMTs and the
sums. Both are written into the network drive.
Speed: 5 MB/s stream of data.
Daniel G. Ang (ACME Collaboration)

ACME Gen II Software Systems

August 3, 2016

15 / 25

Data Acquisition: FPGA Scope VI status

FPGAs initially investigated by Adam, VIs were incorporated into the


database and overall exp. scheme by Brendon
FPGA Scope.vi now refined, troubleshooted, ready to take data
Stress test: able to acquire and save data for at least 12 continuous
hours without error
However, FPGA computer must not be simultaneously used for other
heavy tasks (e.g. editing other VIs) when running
Otherwise will start to lag, fall behind the triggers and need to restart
LabView.

Daniel G. Ang (ACME Collaboration)

ACME Gen II Software Systems

August 3, 2016

16 / 25

Data Acquisition: Header Data


Generated by the G14 DAQ
control computer every trace
(0.5 s), stored on NAS drive
Records the experimental state
and some important monitored
variables

Daniel G. Ang (ACME Collaboration)

ACME Gen II Software Systems

August 3, 2016

17 / 25

Data Acquisition: Software Structure

Global variables record 50 constantly monitored variables: e.g. trace name,


save status, laser lock status, laser wavenumbers, beam box parameters,
etc.
Daniel G. Ang (ACME Collaboration)

ACME Gen II Software Systems

August 3, 2016

18 / 25

Data Acquisition: Streaming

Want to look at traces without having to operate the FPGA


computer or save data to the NAS.
Solution: allow for streaming from FPGA computer to any ACME
network computer

Daniel G. Ang (ACME Collaboration)

ACME Gen II Software Systems

August 3, 2016

19 / 25

Data Acquisition: Streaming

Created an FPGA computer server VI that monitors database


constantly while acquiring data
When a computer wants to stream, the client sends a command to
the database
Daniel G. Ang (ACME Collaboration)
ACME Gen II Software Systems
August 3, 2016
20 / 25

Data Acquisition: Streaming

FPGA server VI picks up, sends data directly to client computer as it


acquires and saves it
For now, limited to one client computer at a time to improve latency
and reduce load on FPGA computer
Basic capabilities demonstrated, need mostly cosmetic refinements

Daniel G. Ang (ACME Collaboration)

ACME Gen II Software Systems

August 3, 2016

21 / 25

Backup System
General Computers Backup scheme:
Bought a second NAS, a Synology 5-bay RAID array to store
computer backups and miscellaneous files: ACME Storage
All computers except for Gandalf are fully backed up every 2 weeks
during the weekend
Backup process takes less 1 hour (for all computers, simultaneously)
Contents of ACME Storage are backed up to the cloud.
Use ACME Storage to store non-EDM data files and share them
across different computers.
Rescue media for each computer are to be prepared in the event of a
crash or hard drive failure

Daniel G. Ang (ACME Collaboration)

ACME Gen II Software Systems

August 3, 2016

22 / 25

Backup System

SQL Database backup scheme


Gandalf is fully backed up every day at 4 AM to an external drive
Database only is also separately backed up every week to ACME
storage

Daniel G. Ang (ACME Collaboration)

ACME Gen II Software Systems

August 3, 2016

22 / 25

Backup System

EDM data backup scheme


EDM data from the FPGA is recorded in the older 8-bay NAS drive,
ACME Data
ACME Data is constantly mirrored onto a second, identical NAS.
It is also mirrored onto the cloud.

Daniel G. Ang (ACME Collaboration)

ACME Gen II Software Systems

August 3, 2016

22 / 25

Near-Future Plans: Notification System

A notification system that alerts relevant personnel when certain


experimental parameters are outside of the allowed range.
Appropriate time resolution depends on response time - hopefully at
most several minutes

Daniel G. Ang (ACME Collaboration)

ACME Gen II Software Systems

August 3, 2016

23 / 25

Near-Future Plans: Notification System

Two types of data being monitored:


1 Some parameters (cell temperature, pressure, room temperature) can
be read from the header files being generated in the NAS every 0.5 s
(to reduce load on the database by not querying constantly)
2 Other parameters (e.g. laser lock status) can be queried every 5-10
minutes from the database
Probably best to run it on an independent, unoccupied computer.
Daniel G. Ang (ACME Collaboration)

ACME Gen II Software Systems

August 3, 2016

23 / 25

Summary

Critical VIs for spin precession data taking are ready for operation (in
LabView 2014, with only minor bugs left): timing VIs, current and
voltage control VIs, laser control VIs
FPGAs and their VIs are ready for spin precession data taking

Daniel G. Ang (ACME Collaboration)

ACME Gen II Software Systems

August 3, 2016

24 / 25

To Do List

Fix minor bugs on a few VIs (e.g. laser locking)


Refine and optimize streaming VIs with more features
Investigate Master Run VI in more depth, troubleshoot any problems
(needed for automated experimental state switching)

Daniel G. Ang (ACME Collaboration)

ACME Gen II Software Systems

August 3, 2016

25 / 25