IEEE Transactions on Nuclear Science, Vol. NS-30, No.
5, October 1983 3925
MULTI-PROCESSOR DATA ACQUISITION AND MONITORING SYSTEMS FOR PARTICLE PHYSICS
V.White, B.Burch, K.Eng, P. Heinicke, M. Pyatetsky, D. Ritchie
Fermi National Accelerator Laboratory
P.O. Box 500, Batavia
Illinois 60510
Abstract
A high speed distributed processing system, using
PDP-11 and VAX processors, is being developed at
Fermilab. The acquisition of data is done using one
or more PDP-lls. Additional processors are connected
to provide either data logging or extra data analysis
capabilities. Within this framework, functional
interchangeablity of PDP-11 and VAX processors and of
the PDP-11 operating systems, RT-11 and RSX-11M, has
been maintained. Inter-processor connections have
been implemented in a general way using the 5 megabit
DR11-W hardware currently selected for the purpose.
Using this approach we have been able to make use of
several existing data acquisition and analysis
packages, such as RT/MULTI, in a multi-processor
system.
History
The majority of experiments at Fermilab have, in the
past, carried out the acquisition of data and the
monitoring of their apparatus with a single dedicated
minicomputer. Despite wide differences in event
sizes, event rates and in the monitoring requirements,
a large number of experiments have been done using a
PDP-11 processor and one of the two online packages
supported by the Fermilab Computer Department. These
packages are the RT/MULTI data acquisition and
monitoring package (1) and an RSX Bulk Memory data
acquisition system (2) combined with an RSX/MULTI
analysis package (1). Indeed, the use of MULTI has
extended beyond Fermilab, with various forms of the
package in use at SLAC, LBL, Saclay and other
institutions.
Some experiments have used a package "off the
shelf" and unchanged. In fact one of the advertised
features of RT/MULTI is that one can be taking data
and analysing it within hours of obtaining the
software package, and without programming. Other
experiments have expended considerable time and effort
in tailoring the package to their particular needs.
Over ninety percent of the experiments with Fermilab
software support have used the RT/MULTI system (more
than 50 experiments in total). The rest have opted
for the package running under the multi- tasking
operating system RSX-11M.
Why Multiple Processors?
Several factors have served to highlight inadequacies
of both single processor and single user systems.
Experimental apparatus is growing in size and
complexity, increasing the need for a greater amount
and depth of monitoring of the apparatus. Upgrades to
the Fermilab accelerator will change the online
software environment to one where data must be
accumulated from the apparatus for approximately
thirty percent of the time. This contrasts with the
previous beam spill of one second out of every ten.
Even sociological factors in large experimental groups
can make the use of a single processor, or worse still
* Work supported by the U.S Department of Energy,
Contract DE-AC02-76ZH03000.
0018-9499/83/1000-3925$01.00©1983 IEEE
a single user system, difficult.
More data, higher rates, more software point
clearly to a need to distribute functions over
multiple processors.
Experimenters normally are anxious to minimise
the experiment "dead-time". Some experimental groups
with very high data rates nay choose to buffer data
prior to transferring it to the processor memory.
Where such "hardware" solutions are inappropriate, the
parallel readout of event data, either by the use of
multiple CAMAC branch drivers, or multiple data
acquisition processors, or both, can significantly
reduce experiment dead-time. The recombination of the
part events and their logging to tape can be handled
as a non time-critical operation, provided sufficient
processor memory is available.
The desire to analyse a small fraction of tihe
data in some depth, while continuing to look briefly
at a significant percentage of events, also indicates
a need to distribute and modularise functions. A
32-bit machine having compatibility with "offline"
analysis of the data and a gigabyte of logical address
space, is an attractive and in some cases essential
ingredient for data analysis. A VAX is the obvious
choice for such a machine, given the existing
extensive use of PDP-lls.
Why not just mnove to the VAX as a single
processor sstem for each experiment? There are
experiments with data rates and dead-time requirements
such that a single VAX could well do the job. There
is however much to be argued in favour of using
PDP-lls for the highly time-critical data acquisition
part of an experiment. Typical total event interrupt
and readout overhead, measured for real-world PDP-11
data acquisition systems are of the order of 300-500
microseconds. Many people have handled highly
time-critical applications within the framework of a
large multi-user operating system. However interrupt
response, on a VAX, to match that of the PDP-11
system, even if possible requires a high degree of
care in both bypassing and coordinating with the
operating system. This is especially true when a
general purpose data acquisition system is the goal,
with widely different performance requirements for
different experiments. The PDP-11 MULTI packages
handle the acquisition of data efficiently and
flexibly. Their simplicity means that even special
purpose hardware can relatively easily be interfaced,
tested and incorporated into the data acquisition
system. This has been done successfully by several
experiments using RT/MULTI, for such devices as
FASTBUS (3) and non Fermilab-standard CAMAC
interfaces.
Non-technical considerations such as funding and
manpower and the existence of many PDP-lls , often
with specialised or tailored software, could also be
cited as contributory arguments toward using PDP-lls
for data acquisition machines. A consideration not to
be overlooked is conservatism. Many physics
experiments metamorphose from a previous experiment.
3926
A data acquisition system which is based on an old and
trusted system, is often desirable from the human
point of view.
Multi-processor systems
The goal of our multi-processor systems is to allow
the functional, practical and performance requirements
of a physics experiment to be satisfied by configuring
both a hardware and software system of processors and
packages. It is a )uilding block approach, in which
some of the building blocks are themselves complete
data acquisition and monitoring packages.
Functional requi rements
Nearly all high energy physics experiments have the
same broad functional requirements from a data
acquisition and monitoring system.
1. Fast acquisition of data from the apparatus, with
minimum dead-time and adequate buffering.
2. Logging of data to a secondary medium, normally a
magnetic tape.
3. Monitoring of the apparatus by either direct
checks on the hardware or analysis of the event
data.
4. Analysis of some fraction of the events.
5. Display of graphic and other information.
6. Control of the starting/stopping of data taking
and tape writing.
Additionally data acquisition systems often
require the ability to playback events previously
written to tape.
Practical requirements
These vary considerably from experiment to experiment,
but some of the following may be required:
1. A complete data acquisition and monitoring package
which requires little or no programming.
2. A framework data acquisition and monitoring
package to which specialised software and/or
hardware can be easily added.
3. The ability to quickly and easily change or add to
the system without affecting other critical parts.
4. A program development capability whilst the
experiment is in progress.
5. The ability to write detailed analysis programs
without the limitations of a 32K word logical
address space. Analysis of the data may require a
large histogram space.
6. Transportablity of analysis software to other
systems.
Performance requirements
Events may vary in size from less than 100 16-bit
words to in excess of 30000 16-bit words. Event rates
may vary from just a few per second to 1000 per
second. A system may be required to read in and
buffer up to 100,000 16-bit words per second.
Normally the primary performance measurement of
interest to the experiment is the event dead-time.
This is the time interval between when the computer is
interrupted to read in an event and when it is again
ready to read in the next event. The ability to
monitor a large fraction, or all, of the events may be
required.
The Links between the Processors
The communication links between processors are
fundamental to the success of a distributed processing
system.
Tightly coupled processors with shared memory
regions were considered and were rejected largely
because of the lack of suitable hardware and also
because of the difficulty of extending the system to
loosely coupled, physically separate systems.
DECNET was also considered and rejected because
of its large requirements on address space and its
high time overheads.
Link Hardware
We chose pairs of DR11-Ws to provide a high speed
parallel link. The DR11-W is a DMA controller capable
of sustaining DMA transfers at a rate of 333
Kwords/second (non-burst mode) and 500 Kwords/second
(burst mode) (4). Pairs of DR11-Ws may be used to
link any two processors with a UNIBUS, which are not
more than 50 ft. apart. It is an inexpensive,
Digital supported device, also available from MDB.
The latter also provides for links between machines up
to 1000 ft apart.
Network Architecture
Our network architecture is designed for typical
multiple processor configurations such as shown in
Figure 1. We adopted a layered approach with the
following ground rules in mind:
1. point-to-point connections only,
2. bi-directional use of the link,
3. direct transfer of data to a program, without
intermediate buffering, for maximum speed,
4. interchangeablity of processors and operating
systems at each node,
5. capability of switching to alternate link hardware
with minimal program change.
Link Layer
The link layer executes a series of transactions with
its counterpart in the other machine. This protocol
accomplishes the sending of a single block of data (a
packet) or a single byte of control information (a
signal) over the link. In addition it guarantees fair
bi-directional use of the link regardless of the
varying speeds and interrupt response times of the end
processors.

Logical Link Layer
The logical link layer is concerned with the routing
of messages to the appropriate program, or part of a
program within a processor. It uses a system-wide
identifier, termed a packet type code (PTC) to
determine the destination of a message. Each message
has to be given a code which describes its intended
destination. Any program or part of a program may
declare itself the exclusive owner of messages of a
particular packet type code, and will then receive all
messages with that particular destination packet type
code. This philosophy allows messages to be sent over
the link, categorised by their function. They are
directed to the piece of software which is responsible
for those types of messages. Whether this i's a
dedicated special purpose program (on a VAX say), or
part of RT/MULTI is transparent to the sender of the
message.
Application Layer
Programs wishing to exchange messages must develop
their own higher-level protocols, which include
destination identifiers, for replies to messages where
necessary. It is at this layer that the use of data
acquisition and monitoring software packages enters.
Implementation of the Communication software
For all three operating systems (VMS, RT-11 and
RSX-11M) we have chosen to implement the link and
logical link layers using device drivers (5),(6). In
addition we have provided a FORTRAN callable package
of routines (CDPACK) to interface to the device
drivers. These provide a uniform operating system
independent interface to the communications software
(7). The driver and the FrORTRAN package both serve to
insulate application programs using them from future
changes in the hardware.
Multi-Processor System Architecture
Hardware
A multi-processor system consists of some combination
of PDP-11 and VAX machines. Currently, one or more of
the PDP-lls are data acquisition machines. Connected
machines must be less than 50- ft apart and linked by a
pair of DR11-Ws. Connections to LSIs has not been
explicitly tried but should be possible with little or
no change to software. Similarly all our tests to
date have been with PDP 11/34, 11/45 and upwards range
of processors, excluding 22 bit machines. RT-11
software should in principle function on 11/03, 11/05,
11/10 type machines.
One machine is normally equipped with at least
one tape drive, either a standard Digital supported
800 or 1600 bpi tape drive or a STC1900 1600/6250 bpi
drive.
Display of graphic information is done using
Tektronix 4010 terminal (or compatible substitute),
and/or Tektronix 613 on a PDP-11 processor. A Requests for events (either single events or a
Versatec printer/plotter or Printronix printer/plotter
device may be used on any processor. The latter may
be shared between multiple processors, using a
hardware switching mechanisin (8).
Data acquisition PDP-lls are equipped with one or more
Jorway 411 CAMAC branch drivers through which event
readout is done. With an RT/MULTIL software package an
(EGG) BD011 CAMAC interface may be used instead.
3927
Fast "event-data-present" interrupts are
presented via a DR11-C general purpose device
interrupt interface module.
Any adequate combination of manufacturer
supported discs may be used on the systems. Disc
requirements range from one double density floppy disc
(onto which RT/MULTI can be fitted) to a more norinal
pair of RL01 or RL02 discs on a data acquisition
PDP-11.
Bank switchable bulk memory may be used on arny
PDP-11 processor (9). This may be the only memory on
the UNIBUS or may co-exist with normal PDP-11 memory.
Addition of one or more 128K word bulk memory boards
to a PDP-11 processor allows the system to access a
large amount of memory, as data buffers, in addition
to the normal memory.
Software
All machines in a multi-processor configuration are
expected to run under one of the manufacturer-supplied
operating systems: VMS (for VAX) and either RT-11 or
RSX-11M (for PDP-lls).
The basic building blocks available for a
multiprocessor system are:
1. Communications software for all machines and
operating systems.
2. RT/MULTI data acquisition and/or analysis package.
3. RSX bulk memory data acquisition package.
4. RSX/MULTI analysis package.
5. VAX/MULTI analysis package (10)
Communications "hooks" have been added and are
still in the process of being added to the above basic
data acquisition packages. The FORTRAN callable
interface to the coi-munications software is being used
to implement these inter-processor dialogues,
following the system design philosophy outlined below.
Multi-processor system design
Each of the functional requirements of a system,
listed above, are considered as logically distinct
subsystems. Each subsystemnvhich needs to communicate
with a subsystem in a connected processor must do so
in a well-defined and operating system independent
manner.
We have defined a system-wide format of such
communications between subsystems for:
1. Providing event data, on request, to an analysis
subsystem. A systemwide network identifier exists
for a subsystem which can provide such event data.
buffer of several events) consist of sending a 10
word request block to this event provider,
specifying the type of events required, the size
of buffer available to receive them and the
network identifier of the requester.
2. Passing a part-event, on request, to a subsystem
dedicated to the re-assembly of part events and
their logging to magnetic tape. This is for
systems where more than one PDP-11 is used to read
3928
out the data for a single event.
Table 1
Standard protocols for the following subsystems have .
yet to be defined: RT-ll/ RT-Il/ RSX/ VAX/ VAX/
RT-11 RSX RSX PDP-11 PDP-1l
1. Run and tape control software communication with * * * +(I) +(2)
the data acquisition and tape logging subsystems, - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

and with other run and tape control sub-systems.

-

Per message time 4-5 5-6 4.5-6 4.5-6

2. Error and message subsystems communication across
processors.
Often the subsystems are, or will be, actually
implemented as separate processes (VMS), or separate
tasks (RSX-11M), which must be able to communicate
with other subsystems in the same machine. In this
case communication software may be used for the
inter-process cominunication too. This is achieved by
the use of an internal communication link feature
provided in both the RSX-11M and VMS device drivers.
Complete functional interchangeability of
software between processors and operating systemsX
demands more than just well defined communications
protocols between the various data acquisition and
monitoring subsystems themselves. It also means
providing certain services normally available within a
machine to a program on a connected machine. In
particular, remote file access and the ability to
remotely manipulate and read from CAMAC, come into
that category. General purpose communication software
will in the future form the basis for the provision of
such services. In the meantime multi-processor
configurations are also constrained by the requirement
that a monitoring program,which directly accesses the
CAMAC must be implemented as a part of the data
acquisition machine software. Also data and histogram
files produced on one processor are not immediately
and transparently available for use on a connected
machine.
Special purpose user written software may make
use of the general communications software to transfer
data, of any type, between connected machines.
The Building Blocks
Overview, Status and Performance
General communications software
Device drivers and a standard interface package exist
for all three operating systems (VMS, RT-11 and
RSX-11M).
Any program in one processor may receive or
transfer data directly from/to the data buffer of any
program in a connected processor.
Much test software for the link has been written
in FORTRAN common code. The link has proven to be
error-free and completely reliable.
Some timing figures for data transfers between
processors (using the FORTRAN callable CDPACK) are
given in Table 1. These depend on the particular
processor and memory speed.
overhead
(milliseconds)
Time overhead per 3 3 3 5.5 3.6
word of data
transferred
(microseconds)
* Loopback tests over several thousand transfers
between 11/34 and 11/50 processors
+ Tentative figures (performance still under study),
of several thousand one way transfers from VAX to
11/50 running RSX-11M.
(1) DMA using VAX direct data path(11/780)
(2) DMA using VAX buffered data path. ( 11/780)
Using connected RSX-11M systems we have measured
the CPU utilisation when executing continuous loopback
communications software test programs. The percentage
of the CPU available to other programs was found to be
between 50 and 65%. This compares favourably with the
CPU utilisation of writing continuously to a 6250 bpi
magnetic tape.
Data acquisition and analysis packages
RT/MULTI is a complete software package incorporating
data acquisition, analysis and run control, with an
emphasis on interactive analysis of the data. It now
has two possible "network" implementations. In one,it
may be used both as a data acquisition system and a
provider of events over the link to an analysis
subsystem. Analysis may also be carried on in
parallel in the RT/MULTI data acquisition machine. In
the other it may be used as an analysis-only package,
obtaining its data from the link. LINK commands have
been incorporated which allow control of
inter-processor communication. They also provide for
control of the supply of data buffers for the analysis
subsystem of the data acquisition machine, relative to
the supply of buffers available to over-the-link
requesters. RT/MULTI sends and receives buffers of
events.
The performance of the system is quite dependent
on various configuration parameters. It depends on
the size of buffers, the number of events per buffer,
and above all on the extent and type of data
acquisition activity. Requests for events are treated
as a part of an analysis-type subsystem and as such
data acquisition activities normally take precedence.
 3929
We have measured the performance in both RT-RT References
and RT-RSX cases. The result is an 11 ms per buffer
overhead in both cases, to which the 3 microsecond per
word transfer time from Table 1 must be added. Large 1. J.F.Bartlett, et. al., RT/RSX MULTI: Packages
buffers containing several events are obviously the for Data Acquisition and Analysis in High-Energy
most efficient way to use the link, since the per Physics, IEEE Transactions on Nuclear Science,
buffer overhead is constant. RT/MULTI link additions Vol. NS-26, No. 4, Aug 1979.
provide for this.
2. J.R.Biel and R.J.Dosen, An RSX-11M High-Rate CAMAC
Two experiments at Fermilab are currently setting Data Acquisition System Using a Bank-Switchable
up dual RT/MULTI systems. Both are using existing and Bulk Memory, IEEE Transactions on Nuclear Science,
heavily tailored RT/MULTI systems, to which they are Vol. NS-28, No. 5, Oct 1981.
adding an RT/MULTI analysis machine.
3. D.Harding,J.Kohlmeier,J.Filaseta, D.Ritchie, A
Interprocessor communication software takes up High Speed Data Acquisition System for Fermilab
more than 2K words in an RT/MULTI system. This means Experiments. Paper presented to this conference.
that existing systems may have to either re-configure
the software or move some of the analysis functions to 4. Digital Equipment Corporation, DR11-W Direct
the connected machines. Memory Interface Module Users Guide.
Any program in an RSX-11M system or on a VAX may 5. M.Pyatetsky, P.Heinicke, D.Ritchie, V.White, Using
also obtain a buffer of events from an RT/MULTI data the DR11-W DMA device for Interprocessor
acquisition system, using CDPACK routines and obeying Communications in RT-11, 1982 Fall DECUS U.S.
the defined "event request" protocol. Work is still Symposium, Anaheim, Ca.
in progress to provide a user interface routine for
such event requests and to incorporate that into both 6. D.Burch, V.White, An RSX-11M Device Driver
the RSX/MULTI and VAX/MULTI analysis programs. implementing Network protocols on the DR11-W, 1982
Fall DECUS U.S. Symposium, Anaheim, Ca.
Three experiments are already planning to take
data this Fall using an RT/MULTI system connected to a 7. J.Biel et. al., High Speed Interprocessor Data
VAX analysis machine. Links using the DR11-W, 1982 Fall DECUS U.S.
Symposium, Anaheim, Ca.
Work has been done on adding logging of part
events protocols to the RSX data acquisition system. 8. R.Knowles, Scanning Printer Multiplexer (SMUXBOX),
Two RSX data acquisition systems may take data, in Fermilab Computer Department Hardware note HN-48.
parallel, for a single event,and later each send the
data to a third RSX "data acquisition" system to be 9. Digital Pathways, Mt. View, California 94043,
recombined and logged to tape. This work is currently Bank-Switchable Bulk Memory Users Manual.
being tested and at present there is one potential
user of such a configuration. 10. K.Eng, B.Burch, D.Ritchie, VAXMULTI System
Generation, Fermilab Computer Department Program
Note PN-146.
Conclusions and Future Plans
We will continue to add communication "hooks" to all
subsystems. The RSX data acquisition system must be
equipped to provide events over the link. RT/MULTI
and possibly some VAX subsystem will have part-event
logging capabilities added. Work will proceed on
integrating system-wide run and tape control and a
system-wide error and message system.
We will be closely watching the performance of
these multi processor systems as they go into
operation in experiments. Our user community has, in
the past, given us valuable feedback both in the form
of operational experience and program contributions.
With similar input, we will adapt these multi
processor systems to user needs and incorporate our
users' additional developments.
3930

Poss ib le Processor Topolo2ies

'-~~~~~~~~~~~~~~~~L --e

IIS

AQIJISITION AND LOGGING

CAMAC / FAS-iUS
(A) EXTRA ANALYSIS

MACHINES,

| BE

(B) LOW DEAD-TI ME PARALLEL READOUT

MACHINES,

(C) PARALLEL READOUT AND EXTRA ANALYSIS