293

IEEE TRANSACTIONSON NUCLEAR SCIENCE,VOL. 37, NO. 2, APRIL 1990

A HIGH SPEED FIBER OPTIC SERIAL NETWOFtKING SYSTEM FOR DL4TL4

ACQUISITION AND CONTROL
Robert Skegg, TRIU R4F
4004 Wesbrook Mall, Vancouver B.C., Canada VGT 2A3'

A bs t ra c t
A serial ring coininunication system is described which
inay be used t o link data acquisition and control devices

and, via suitable interconnects, other serial rings, FASTBUS segments and CAhiIAC crates. The protocol used is
transparent t o FASTBUS [l] and contains very little management overhead. T h e system has applications from sim1)ly linking two closely spaced devices t o interconnecting
many FASTBUS and CAR4AC systems in a wide area network. Performance calculations are presented showing the
effects of varying the link bit-rate, the link distance and
the number of connected devices.

cannot gain control of the ring. It may be addressed by

a master and transfer data on demand. A slave may also
be capable of transmitting a siinple asynchronous service
request t o indicate that it requires attention. The ring
CONTROLLER supervises access to the ring, generates
the primary clock signal and responses for broadcast operations. Masters and the ring controller may contain slave
devices. An interconnect device, used t o allow commnunication between two rings, or between a ring and a FAST13US
segment or CARlAC crate, contains a combination of master and slave devices which are coi~nectetlto each ring or
segment .
111. TIIE RING STRUCTURE

I. INTRODUCTION
Existing inter-crate communication schemes for modul a r control or data acquisition syst,eiiis suffer from various
tlra\vlncks: they can be complex and expensive, difficult
to extend past looin length, have only one-way commuiricat.ion or may require sta.ckiiig of communication protocols. The result for physically large or complex systems is
a Ijmit,ed data carrying capacity and, sometimes, limited
communication ability.
In order t o meet the demand for effective coinmunicat,ioiis bet.ween devices in data acquisition or control systems a new serial ring coinmunication scheme has been designed. The structure is a mono-directional single ring ideally (but not necessa.rily) using high-speed fiber-optic links
between devices. Fiber-optic links provide a cost-effective
iiieaiis of transporting large volumes of time-critical data,
and they eliminate electro-magnetic radiation susceptibilitmyand ground noise problems associated with copper wire
connections. This proposed communicatioii system may be
effectively used t o link a very large number of devices over
great distances or just two devices over a short distance.
The protocol includes operations such as arbitration
for ring usage, addressing and d a t a transmission, and is
designed t o transparently 1ia.ndle FASTBUS communications. Using suitable interconnect devices, it may also pro\.icle communications for CARIAC.

11. DEVICES
Three types of devices may be connected in a serial ring.
h MASTER device may gain control of the ring, address
slave devices and control data transfers. A SLAVE device
'From December 1989: SSC Laboratory, 2550 BecMeymeade Ave.,

Dallas, Texas 75237.

A serial ring consists of one ring controller, one or more

masters and some slaves. A simple system is sho\vn in Figure 1, consisting of one master, one slave and a ring controller device which could he included in the master. One
fiber optic link connects the mastcr/controller to the slave,
and a second connects the slave to the master/colitroller,
completing the ring.

Figure 1: A simple serial ring system consisting of one

master, one slave and a controller
Figure 2 shows a system which uses interconnect, devices
to provide transparent communication between FASTI3US
segments which may be separated by a great distance.
IV. SIGNAL ENCODING
Both d a t a and data timing information (clock signals)
must be passed through the links. The data ant1 clock are
therefore Manchester encoded for transmission as slio\vii
in Figure 3. Manchester code has some features which are
used t o advantage by the protocol. Firstly data Os aud 1s
occupy easily recognizable locations and are of the same
length which allows on-the-fly bit replacement. Secondly,

0018-9499/90/0400-0293$01.000 1990 IEEE

294

Fastbus Segment

Interconnect
Device

our
Q

Ring
Controller

Interconnect
Device

IN

OUT

IN

Fastbus Segment

1N

OUT

Figure 2: Interconnection of FASTBUS segments by means of a serial ring

because of the reliability of fiber-optic links a Manchester
code violation can be used to indicate a special condition,
if this condition does not occur too often.
When the system is idle a 0,1,0,1 pattern is transmitted
to maintain synchronization of clock regenerators located
in each device. Signals are re-timed by each device before
being passed on.

V. PROTOCOL
The protocol allows for arbitration between contending
masters, broadcast and individual addressing and various
modes of data transfer.
The general structure of the protocol is shown in Figure
4. The start of each operation is marked by a Manchester
code violation. This is followed by an eight-bit control
word which indicates the nature of the operation. Up to
32 bits of data may follow. The clock regeneration system
i n each device has a compensation mechanism to prevent
perturbation due to a missing clock edge.
The significance of each bit of the control word depends
to some extent upon the higher level protocol being passed.
A mapping which provides compatibility with FASTBUS
is shown in Table 1.

A. Arbitration
To gain control of the ring a master must arbitrate with
any other contending masters. The winner of the arbitration operation may use the ring to address slaves and
transfer data between itself and attached slaves. It may
be that the ring has only one master. In this case either
arbitration is dispensed with or an arbitration operation is
conduct,ed during system initialization and mastership of
the ring is never released.
Each master has a single bit register which indicates if
the ring is free for use. This bit is cleared at system initialization as discussed later. If a master wishes to use the ring
and its ring-free bit is set then it transmits an arbitration
request control word (see table 1) to the ring. If during
transmission of this control word the master detects an incoming control word (easily detected by the initial Manchester code violation start indicator) then it aborts its own
transmission and repeats the incoming control word, complete with new start indicator. This will happen if, for
example, an up-stream master has also transmitted an arbitration request.
Each device on the ring repeats the arbitration request
which eventually arrives a t the ring controller where it is

absorbed. The ring controller then transmits an arbitration grant control word to the ring followed by eight bits of
zeroes which specify the initial winning arbitration level.
As each master receives the arbitration grant it clears its
ring-free register bit, thus preventing transmission of further arbitration requests. Masters (and all other devices)
repeat the control word and following arbitration level. If
the master was requesting use of the ring it compares the
following eight bits with its own arbitration level. This is
loaded into a register in the master at system initialization
time and specifies the priority of the master. Each master
has a unique priority level and the larger the number, the
greater the level.
If a requesting master has a greater prioiity level than
that on the ring then it replaces the ring arbitration level
with its own. This check and replace function can be done
on a bit-by-bit basis because the arbitration level following
the control word is transmitted with the most significant
bit first. The master simply compares each bit as it arrives
with the corresponding bit of its arbitration level. IVhilst
the incoming bits have the same value as the inastcrs then
the arbitration levels are the same. If the incoming value
is 1 and the masters is 0 then the incoming arbitration
level is higher and the master simply repeats the remainder of the incoming level bits. If the incoming value is 0
and the masters is 1 then the master has a greater arbitration level, and replaces the remainder of the incoming
arbitration level with its own value.
Table 1. Control words, FASTBUS mapping
5

0
I

PE
0
O

0
0
0
0
0
0
0
0
0
0
0
0

0
0
0
0
0
0
0
0
0
0
0
0

0
0
O

0
0
O

0 3
PA
0
l

2
MSZ
SSZ
*

1
1
1
1
1
1
1
1
1
1
1
1

1
1
1
1
1
1
1
1
1
1
1
I

0
0
0

0
0
0

1
0

0
0
1
1

0
0
0

1
l

0
I

0
0
1
1

1
1
0
1

0
1
RD
RD

1
PA

P
P
P
P

E
E
E
E

PA
PA
PA

SSZ
SS2
MSZ

MSZ

0
MSO
SSO

0
1
1
0
0
1
1

0
1
0
1
0
1
0
1
0
1
0
1

0
I
1
0

1
1
1

1
MS1
SS1

0
1
1
SS1
SS1
MS1
MSl

1
0
1
SSO
SSO
MSO
MSO

Addrcrr s y n c
Address acknowledge

Reserved
Release address
Release acknowledge
Wait
Reserved
Arbitration request
Arbitralioii grant
Graut acknorl+,lgr
Release b u r
S e r v i c e request 1

service qYF.1 2
Halt r C q Y C s t
B u r halted
Reiet bus
Release r e q u e $ t
Relcarc b u s

Dua free
Data
Data
Data
Dsla

acknowledge 0
acknowledge 1
sync 0
sync 1

* = dont care, RD = Read / not write indicator PE = Enable parity

checking, PA = Parity check bit, MS = Mode select bits, SS = Slave
status bits.

296

T\\
Control Word

Idle

31

w a s a read operation, follows that with the 32 bits of data.

In order to accommodate the full significance of FASTBUS Mode Select Bits, data sync 0 and corresponding data
acknowledge 0 control words are provided for continuing
data transmissions. These are equivalent to FASTBUS DS
(down) and DI< (down) timing signals.
All FASTBUS data transfer modes are possible, but
when using pipelined transfer (in which the master does
not wait for the confirming DI< before commencing the
next transfer) the masters must allow time for both the
slave to respond and for the insertion of the response in
the serial bit stream. For broadcast write operations each
slave picks off the data as it passes around the ring. The
ring controller supplies the DK handshake. For broadcast rea.d operations the DS control word from the master
primes the slaves and the ring controller that a read operation is about to occur. The master (not the ring controller)
then transmits a DI< control word followed by 32 Os. Each
slave in turn ORs its da.ta into the 32-bit stream which
eventually arrives back a t the master. Slaves may also OR
status bits into the DI< control word.

D. Other Operations
A slave which is unable to respond but expects to do
so in a short while may transmit a Wait control word to
the master rather than a DI< response. This will cause the

master to suspend its response timer which would otherwise, a t expiration, indicate an error condition. The slave
should eventually transmit the required DK response and
when the master receives this it then proceeds normally.
Any device may, when the ring is free, attempt to transmit a Service Request (SR1) control word, provided that
transmission is aborted if an incoming start bit is detected.
This allows a device which does not have full master capability to asynchronously request service by a master. For
SR signalling to be effective, requesting masters must leave
a gap for SR1 signals after the bus-free control word is received, before transmitting an arbitration request control
word. SR1 is absorbed by the ring controller which then
t.ransmits an SR2 control word. This passes once around
the ring and is then absorbed by the ring controller. In
transit it informs the servicing master of the request and
the originating slave that its request has been passed on.
VII. SYSTEM INITIALIZATION
At power-up all devices must be disabled by clearing
t,heir bus-free bit. The ring controller transmits an idle pattern to the ring, and monitors the returning signal. When

S
at Control Word
r
1
t o
,

Idle

the received idle pattern becomes stable the clock regenerators in each device are in lock. The ring controller then
sends a bus-free control word which allows enabled inasters
to request arbitration.
Masters power-up with arbitration disabled as they must
first be loaded with an arbitration level. This may be performed simply through an external connection to the master. In systems having a number of distributed master
devices, first one master is externally loaded with an arbitration level and enabled. It then arbitrates for use of
the ring and gains control because it is the only enabled
master. It proceeds t o connect geographically to the slave
part of each other master in turn, down-loading and the
arbitration level and enabling arbitration.
If logical addressing is implemented, slaves power-up
with logical address recognition disabled. During system
initialization a master device must attach to each slave in
turn using its geographical address, down-load a logical address, and then enable the feature by setting an enabling
bit in the slave.
If device substitution, removal or addition in an active
system is necessary, a halt request (IIR) control word can
be transmitted from any device immediately folloiviiig a
bus-free control word. HR is absorbed by the ring controller where it causes transmission of a bus halt (BII)
control word to the ring and disables further operations
until a release request control word is received. BII clears
the bus-free bit in each device preventing further activity.
At this point the ring may be broken to insert or remove
devices.
The system is re-started by transmitting a release request control word to the ring controller which responds,
when the received ring signal is stable, by transmitting a
bus-free control word to the ring. Halt request and release
request may be generated by any device, therefore access
is only required to a nearby suitably equipped device i n
order to halt or release the ring.
If system re-initialization is required, a reset bus control
word will disable logical address recognition in devices and
disable arbitration in masters.
VIII. INTERFACE STRUCTURE
T h e structure of a serial ring interface is sho\vn in Figure 5. High speed serial protocol processing is performed
in a semi-custom integrated circuit. This can include most
of the phase-locked oscillator necessary to regenerate the
clock signal, and the on-the-fly bit replacement mechanism used during arbitration and broadcast read operations. This semi-custom circuit has byte-wide connections

295

Y
Idle Pattern

1
Start Bit

v
Control Word
(Arbitration Request)

Y
Idle Pattern

Figure 3: Manchester code format used to encode clock and data. Each data bit contains a clock synchronization
transition. The absence of a clock transition is used to indicate the start of a control word

VI. OPERATIONS
When the arbitration grant control word arrives at the
ring controller it is therefore followed by the arbitration
level of the contending master with the highest priority.
The ring controller then transmits a grant acknowledge
control word followed by that arbitration level to the ring.
Each contending master compares the arbitration level
following the grant acknowledge control word with its own.
The winning master is the one that finds a match and
becomes the current master. After passing the grant acknowledge and following arbitration level the winning master changes mode of operation to absorb subsequent control words and data. The grant acknowledge control word
eventually arrives a t the ring controller which absorbs it
and changes to a transparent operating mode.
When the master has completed all its operations it
transmits a release bus control word which sets the bus-free
bit in other masters. The ring controller passes the release
bus control word but then re-enables its absorption of subsequent control words. When the release bus control word
arrives at the current master it sets its bus-free bit and
cancels its current master status placing it in transparent
inode again.
To allow space for service requests and bus halt coinmaiids which are described later, masters may not transmit an arbitration request control word during the 15 bits
following a bus-free control word.

B. Addressing
Once a master has gained control of the ring it may
at,tempt t o gain connection to one or more slave devices
by performing one of three types of address operation.
Any single slave may be LOGICALLY addressed by
transmission of an address select (AS) control word followed by the address contained in the following 32 bits. As
each slave receives this information it checks the address
against its own logical address. This is contained in a register in the slave and is loaded during system initialization.
Not all the 32 bits need be used for device identification.
For example, the upper 20 bits only could be used to identify the device and the lower 12 bits to identify a feature
therein. The address information is passed from device to
device around the ring until it arrives back at the issuing
master where it is absorbed.
If a slave recognizes the address it sets its single bit
'attached' reeister and transmits an address acknowledge

(AK) control word following the the end of the address

sync information. When the address acknowledge reaches
the master it is absorbed and the master knows that a
connection to the slave has been established. If the master
does not receive a response within a defined period then an
error condition exists and the master should take corrective
action.
Slaves may also be addressed GEOGRAPIIICALLY. In
order to maintain compatibility with FASTBUS, addresses
having the top 24 bits set to 0 are reserved for geograpliical addressing. The lowest eight bits contain the address
which the slaves compare to a juniper or switch-configured
on-board address. Up to 256 devices may therefore be coiinected in one ring. Interconnect devices used to connect
a FASTBUS segment to a serial ring must zero the top 2.1
bits when a FASTBUS master geographically addresses a
device on the ring, because there is no equivalent to tlie
FASTBUS Geographic Address bit.
FASTBUS style BROADCAST ADDRESSING inay be
used to attach to more than one slave. This mode is indicated by the MS bits embedded in tlie AS control word.
The following 32 bits are interpreted by the slave according
to FASTBUS rules. Addressed slaves do not respond with
AI<. This is produced by the ring controller after a delay
suitable for the response speed of the slaves connected to
the ring.
When the master has finished its operations it releases
the slave to which it is attached by transmitting a release
address control word. The slave then clears its 'attached'
bit and transmits a release acknowledge (RA) control word
to the master which may then address another slave or release the ring for other master to use. \\'lien broadcast
operations are terminated the RA control word is generated by the ring controller.

C. D a t a Transmission
When the master is attached to one or more slaves it
may initiate data transfers by transmitting a data sync
(DS) 1 control word (see table 1.) This contains a n RD
bit which indicates if the operation is to read or write tlie
slave. If it is a write operation then the following 32 bits
contain the data. The control word also contains inode
select bits as used in the FASTBUS protocol.
When the slave has completed the requested operation
it responds with a data acknowledge (DI<) 1 control word
containing status information (FASTBUS SS bits) and, if it

297

High Speed
Serial Protocol
Processor

Table 2. Serial ring system performance

1-1

System 1
loom
100 Mb/s
1 master
1 slave

System 2
100m
20 Mb/s
1 master
1 slave

System 3
85 km
100 Mb/s
200 devices

System 4
85 k m
20 Mb/s
200 devices

Arbitration
Time, ps

764

so9

Primary
Address, ps

2.1

4.4

389

407

Single data
Transfer, ps

3.9

7.1

777

81 1

Block
Transfer of
1000 32-bit
words, ms

2.1

4.4

389

407

Controls and
Byte-wide
Data

Address-Data
Separator

Address

Fiber
Optic
Transmitter

General Serial
Protocol Control

Data

To Slave Device

Figure 5: Structure of a serial ring interface

to slower devices handling the general protocol and separating data and address information. These are similar
to the PCL [2] and AD1 [3] devices for FASTBUS parallel
segments.
IX. ERROR DETECTION AND RECOVERY

Effective
8-bit Byte
rate,
k Bytes/s
1900
906
10
9.8
Effective rate calculated with overhead of arbitration, primary &
secondary address, block transfer of 1000 32-bit words. Calcula-

The ring controller continuously monitors the signal

stream returning from the ring. This phase of this should
be stable when compared to the out-going stream generated by the ring controller. Any rapid phase change or loss
of signal indicates a fault on the ring and should cause the
controller to halt operations.
The ring controller can also keep track of the sequence
of control bytes on the ring and take corrective action if
abnormal operation is observed. For instance, if an arbitration sequence is initiated but after a certain period the
grant acknowledge control word has not been received then
it knows that the arbitration sequence has failed and can
take appropriate action.

The serial ring communication system described offers

advantages over existing communications systems when
large distances are to be covered or when copper wire connections present a problem.

X. SYSTEM PERFORMANCE

XIII. ACKNOWLEDGERIENTS

The performance of a serial ring system depends mostly

on the bit-rate at which it is operated and the length of
the ring. The number of devices in the ring also affects
performance. Table 2 compares the performance of several
systems.
System 1 is a small link between a single master and
slave device, as might be used to connect a FASTBUS segment in an experimental area to one in a counting room.
It operates a t 100 Mb/s (which is one transition every 5ns
for continuous 1s or Os) and the master and slave are separated by 100 meters.
System 2 is the same as system 1, but operates at 20
Mb/s. It can be seen that reducing the bit rate by a factor
of five lowers performance by a factor of two.
System 3 consists of 200 devices connected in a large
ring 85 km long. It operates at 100 Mb/s. This is over 600
times the size of system 1, but the performance is reduced
by a factor of only 200.
System 4 is the same as system 3, but operates at 20
Rlbls. It can be seen that in a large system propagation
t.ime dominates; bit rate has only a small effect.

The author wishes to thank Ken Dawson, Peter Bennett

and Ruth Risto of TRIURlF for their valuable assistance
in preparing this paper.

tions assume velocity factor of fiber optics=0.68, fiber optic transmitter+receiver propagation delay=4Ons, bit delay through devices=2,
slave+master response time=lps.

XII. CONCLUSIONS

References
[l] Modular High Speed Data Acqusition and Control
system, IEEE Std 960-1986 with Addenda and Errata
of May 1987, U.S. NIM Committee.
[2] R. Skegg and A. Daviel, A Semi-custom Protocol
Control Logic Device For FASTBUS, IEEE Transactions on Nuclear Science, February 1988, Vol. NS-35,
page 306.
[3] R. Skegg and A. Daviel, A General Purpose FASTBUS Interface Chipset, IEEE Transactions on Nuc1ea.r
Science, February 1985, Vol. NS-32, page 305.