AI & Soc (1998) 12:78-86

@ 1998 Springer-VerlagLondonLimited A] ~ S O C l l l l I ' r ~

M e a s u r e m e n t of Vessel Traffic Service Operator

Performance

E. Wiersma ~ and N. Mastenbroek 2

1Safety Science Group, Delft University of Technology, Delft, The Netherlands,
2MarineSafety International Rotterdam, Rotterdam, The Netherlands

Abstract: To meet the growing demands for yet more efficient and safer traffic, traffic
control is deployed in all modes of transportation. In maritime transportation, traffic control
is performed by Vessel Traffic Services (VTS). This paper describes research which is
focused upon measurement of VTS operator performance. The concept of situation awareness
is introduced as a means to describe and quantify VTS operator performance. Situation
awareness is tested in a full scale interactive simulator. A scoring system for VTS operator
performance accounts for the specific characteristics of VTS operator work, such as accuracy
and relevance of information.

Keywords: Human-machine interaction; Management systems; Maritime safety;

Methodology; Performance characteristics; Traffic control

1. Introduction

Traffic is intensifying in all modes of transportation. Increased quantity and complexity

of traffic, combined with the necessity to optimise traffic streams and logistics in
airports and harbours make traffic control ever more needed, In maritime trans-
portation, traffic control is carried out by Vessel Traffic Services (VTS). VTS operators
are responsible for a safe and efficient handling of traffic in harbours, rivers and
approach areas. They guide traffic in and out of harbours, provide information on
request and coordinate movement of ships in (emerging) conflict situations.
For gathering information on traffic situations, vessels' intentions, weather con-
ditions, and other relevant parameters, VTS operators have sophisticated tools at
their disposal, such as double radar displays of traffic combined with synthetic
images of harbour lay-out and vessels, and information systems of ships, harbour
etc. A large number of communication tools of all kinds enable VTS operators to
communicate with ships, other VTS posts, and harbour services. Apart from making
use of tools available, operators often have a direct view on traffic through the
window, possibly with binoculars.
Measurement of Vessel Traffic Service Operator Performance 79

The instruments are subject to constant development and improvement, aimed at

supporting the operator. To ensure that newly developed tools enhance VTS operator
performance and that safety levels are maintained, new equipment and technological
devices have to be tested prior to implementation. An EC-funded research project is
initiated with the purpose to develop a method which will enable the evaluation of
the effect of these new devices for the work of VTS operators. This paper describes
part of this project which aims at providing a theoretical framework that can serve
as a tool for evaluation of modifications to a VTS System and can also help in
designing new VTS support systems (Wiersma and Heijer, 1996). Ideally, this theor-
etical basis should be complete enough to describe optimal VTS support system
characteristics. It should take into account:

9 situation characteristics: harbour layout, type of traffic, weather conditions etc.;

9 the characteristics of the human controller in the supervisory role that VTS requires;
9 the characteristics of the VTS clients;
9 nature and quality of operator/client communication.

Wiersma and Heijer (1996) give a review of applicable theoretical concepts and tools
and the consequent functional demands that follow from these concepts. They conclude
that at present a complete theoretical understanding of human operators, in this
case both the ship's pilots and the VTS operators is not available. Theoretical models
required to describe with sufficient accuracy how human operators will interact
with all sorts of tools and systems are not available.
Wiersma and Heijer propose to develop a series of tests (test bank), which take
VTS operator performance as a measure, in order to evaluate the effect of changes in
the VTS operator task or task performance, due to new tools. In this test bank, several
performance measures are combined, including situation awareness of VTS operators,
task load measurements, and a classification of VTS operator errors. This paper describes
how situation awareness is tested in VTS operators in an experiment and how a
base-line for VTS operator performance has been established. This concept has evoked
enthusiastic reactions from VTS trainers. It gives them the opportunity to evaluate
aspects of VTS operator performance which were previously difficult to evaluate.

2. Measuring VTS Operator Performance

VTS operators are well trained professionals with a maritime background, which
can be either civilian or naval. This background ensures that VTS operators are well
aware of manoeuvring abilities of ships, and that they can "think together" with captains
and navigators aboard ships. Extensive simulator training with equipment and
communication protocols, as well as on-the-job training are used to establish a high
level of professional competence among VTS operators. Obligatory triennial qualification
examinations ensure that this level of competence is maintained and that VTS operators
keep up with new developments in equipment and (communication) procedures.
The role of VTS operators in traffic is more informing and advising than controlling,
because ships have a large freedom in accepting or rejecting orders given by VTS.
Awareness of manoeuvering possibilities of ships and communication with traffic
80 E. Wiersma and N. Mastenbroek

are therefore of utmost importance. In training and qualification examinations a

strong emphasis is placed on the communication skills of VTS operators. However,
focus on communication and communication protocols constitutes a one-sided view
of the issue. In order to provide a more comprehensive (and a more correct) view,
other factors are incorporated in the project as well. One of these factors is the VTS
operator's understanding of a current traffic situation. Although a thorough descrip-
tion is given of traffic situations used in training, understanding of these situations
is usually examined by a secondary measure: communication with traffic. In a walk-
through-talk-through session afterwards, a simulation run is reviewed, an operator
is shown his mistakes and is given the opportunity to justify his actions. Here the
operator can show that he was aware of specific circumstances in the traffic situation
and that he has communicated correctly.

3. Defining Situation Awareness

Recently the concept of situation awareness has been put forward as an objective
measure of the understanding of supervisory situations, such as VTS. Developed in
military aviation some 15 years ago, situation awareness has since been applied in
numerous fields of application,varying from Air Traffic Control (Mogford, 1997),
weather services (Dyck and Gilson, 1994), and medicine (Adams et al., 1996)
Over the past few years a vast body of literature has emerged on the topic of
situation awareness, both on theoretical and applied research. Numerous attempts
have been made to define the concept, see for instance Sarter and Woods (1991),
Endsley (1995), Wickens (1995), to name just a few. Instead of proposing a new
definition, the one offered by Endsley (1995, 36) is adopted in this paper:
Situation awareness is the perception of the elements in the environment within a volume of time and
space, the comprehensionof their meaning, and the projection of their status in the near future.
According to Endsley's definition, situation awareness can be seen to consist of
three levels:
1. Perception of elements in the current situation: the state of the environment
which has to be supervised can be defined as a large collection of different para-
meters with specific, momentary, values. The completeness of the mental picture
of the operator depends in part on the possibility to obtain information on these
parameters: personal abilities and supporting equipment determine the subset of
obtainable parameters and hence the limits of completeness of the picture;
2. Comprehension of current situation: numerous state parameters in the envir-
onment will be related by causal mechanisms. In any given situation some of
these mechanisms are latent and others are active in effecting the way state para-
meters change over time. Comprehension of the situation can be described in
terms of the available repertoire of mental models of these causal mechanisms,
the choice of model(s) from this repertoire that the operator is currently applying
and the quality (relevancy, accuracy, completeness) of the chosen models with
respect to reality;
3. Projection of future status: apart from an assessment of the current state of the
environment, a prediction of how this state will change in the near future is also
Measurement of Vessel Traffic Service Operator Performance 81

important for the decisions and actions a VTS operator may take. The quality of
these predictions is in part determines the quality of the decisions.
It must be noted however that situation awareness is not the only factor which
determines operator's actions. Other factors include a formally defined set of goals
and objectives, operator's interpretation of these goals and objectives, as well as his
preconceptions about his role.

4. Measuring Situation Awareness

As stated before, there are many definitions of situation awareness, in many fields of
application. Likewise, many methods have been developed to measure situation aware-
ness. These methods can be categorised into four groups: physiological techniques,
performance measures, subjective techniques and query techniques (Endsley, 1995b).
A method, SATEST, for measuring situation awareness of VTS operators (Wiersma
et al., 1997) has been developed, based upon the Situation Awareness Global Assess-
ment Technique (SAGAT), developed in aviation research (Endsley, 1995b, 1996).
SAGAT is a global tool for measurement of situation awareness in military aviation.
In SAGAT, pilots run a range of simulator flight scenarios. At specified times during
these scenarios the system displays are blanked and the simulation is stopped, and
pilots are requested to answer a number of questions about their perception of the
situation at that time, related to their situation awareness. Questions pertain to all
three levels of situation awareness, level 1 (perception of elements); level 2 (compre-
hension of current situation) and level 3 (projection of future status). This includes a
consideration of the system status as well as relevant aspects of the external
environment. The accuracy of the responses is then compared with logged situation
data. Thereby a measurement of situation awareness on all three levels is derived.
A method, SATEST, has been developed to measure situation awareness in a
VTS context. SATEST uses the same "freeze" technique in simulated VTS trials~
These trials are conducted at the simulator of MarineSafety Int. Rotterdam (MSR),
which is used for training and formal qualification examinations. The simulator
used consists of a full operational VTS terminal which is linked to a very extensive
maritime traffic simulator. In this simulator years of research effort has been put
in designing and developing realistic traffic, where ships behave as they would in
reality, taking into account both ship characteristics and weather conditions, river
flows and tidal effects. Numerous traffic scenarios can be chosen, both in real maritime
areas and imaginary areas, with pre-programmed traffic. Oral communication with
ships, other VTS-posts and services is supplied by a ghost operator, who can also
apply manual control of any of the ships within the scenario.
Basic requirements for the scenarios used in SATEST are that they are:
9 of a sufficient quality to enable the VTS operator to forget that it is just a simulation;
~ long enough to enable the VTS operator to build his situation awareness of
the situation;
9 complex enough to differentiate between operators or different experimental settings
A typical simulator run used in SATEST comprises of a forty-five minute scenario
preferably in an imaginary harbour area (the last is advisable to avoid effects of differences
82 E. Wiersma and N. Mastenbroek

in familiarity with the area between different operators). In the scenario several conflicting
situations develop, which have to be handled by the operator to avoid conflicts.
At specified times the simulation is stopped and questions, which are relevant for
VTS, are asked about the situation. Responses are recorded by means of computer.
A score form has been developed on which operators can fill in a number of aspects
of each situation. The score form consists of two parts. The first part is a map of the
VTS area. This map is used to indicate the exact position of all ships and the speed
and course vector. With this form a complete reproduction of the traffic situation is
made as it is seen on the VTS monitor. Ships are denoted by small circles; their speed
and course are indicated by a vector. An example of form (1) is given in Fig. 1.

@@@@ ~@ ~J @@
@ @
5 e e e ,2....-.....~:--~
~
5 9 -'~
~.~vRo. ~ I
e} o"" @

?
.~5-.s ::;. . . . . }>)~ ,,v~ - % - -- __
...... -, / ; / .... -_.

................ .:0\ =~

~EOTOR NO01ZD~ZI~EHA~N

Fig. 1. Form(1) used in SATESTto describe traffic situation.

A second form is used for each ship individually. This form opens every time a
single ship is selected by clicking the mouse. On this form (2a) several aspects are
to be filled in. These include:
9 Information about ships (has there been communication with ship; name or label
of ship; origin and destination; type of ship; speed; special remarks);
9 Focus of attention;
9 (Potential) conflicts. Particulars about the conflicts can be indicated in a separate
form (2b)
An example of the forms (2a) and (2b) is given in Fig. 2.
In this manner, information is collected on the operator's mental picture of the
traffic situation, including elements such as conflicts and potential conflicts between
ships and between ships and infrastructure, and predictions of where conflicts are
likely to develop.
Measurement of Vessel Traffic Service Operator Performance 83

Fig. 2. Forms (2a) and (2b) used in SATESTfor information of individual ships and conflicts.

5. Validation of Experimental Material

Validation of experimental material consists of two distinct elements: validation

of material presented to VTS operators, and validation of the scoring system. Both
aspects deserve adequate attention.
The material presented to the VTS operators needs to be validated so as to estab-
lish that the reactions of the subjects reflect their normal behaviour. Otherwise, the
results obtained may be of questionable value.
In choosing scenarios for experiments one can focus on behaviour in extreme
situations or select scenarios which reflect usual conditions. In this project the latter
approach is chosen. In normal VTS working conditions, conflicts are rare events.
Situations with potential for developing into conflicts are solved as soon as they
arise. Research conducted with scenarios which include an unrealistic number of
conflicts may lead to decreased credibility of the testing environment, biases and
thus reduce value of results.
The simulated traffic scenario presented to the operators must provide a realistic
picture of traffic, in a realistic setting, i.e. traffic situations VTS operators have to
deal with in daily practise. The researchers in this project have been in a fortunate
position that they can make use of the large accumulated experience with the simulation
of traffic at MSR. In this experiment a scenario is taken from the regular training
program (used for the qualification training of VTS operators at MSR's simulation
centre) and slightly moderated. This scenario has been created after extensive analysis
of an array of actual traffic situations. Types of conflicts presented, are those which
VTS operators encounter in their work. The number of conflicts per unit of time and
their seriousness are slightly raised, although they do not exceed limits of credibility.
Fortunately maritime accidents are rare events. Even in confined waters as rivers
and harbours, collisions do not happen often. Ships are rather inert objects and are
slow to change in course and speed. Especially when sailing with low speed, as
is required in confined areas, manoeuvrability of ships is low. Therefore evasive
84 E. Wiersma anclN. Mastenbroek

action is taken at the earliest moment, when a potential conflict is detected several
minutes ahead.
Likewise VTS operators are used to intervene in potential conflict situations
between ships in the earliest moment and do not wait upon the developments. In
daily routine, VTS operators are hardly ever confronted with conflict situations which
develop into accidents. Action has been taken long before that moment. Therefore,
in order to design an experimental setting which does not deviate too much from
reality, three situations with only potential for conflicts are distinguished. Figure 3
shows one of them. Ships are denoted by small circles; their speed and course are
indicated by a vector. The three ships circled are potentially in a conflict situation.
The possible accident lies fifteen to twenty minutes ahead in time.

$A-session /%I~NvsJ!~ STOP [ g o I O• Help

|174 ~@ ~J |

ooo
..... @@:~ &
Conflict ,oo ...... =o ~ .....
~ ~ ......... alind~

L..
s~a~r

9S ~ I'IOOtI~AV~N

Fig. 3. Typicalpotential conflictsituation.

The setting of the scenario is in an artificial harbour area, which is part of a larger
simulated area. This area was designed more than ten years ago and has been used
ever since. Much effort was put into creating an area which is realistic and has the
characteristics comparable to those of the areas where VTS operators usually work.
The map on Fig. 1 presents this area.
The moment in time presented in Fig. 3 is considered the first time VTS operators
can be expected to act in this potential conflict situation. The required action of the
VTS operator would be to contact all ships involved in the situation and communicate
the potential conflict with them and agree on a solution. The follow up of this action
would be to keep an eye on the development of the situation and intervene only if
deviations of the agreement are noticed (maintaining situation awareness of the ships'
positions, and the follow up of the agreement). Most operators act upon a situation
like this at the earliest moment. However, some operators wait for the situation to
develop instead of interacting immediately, thus giving the ships the possibility to
Measurement of Vessel Traffic Service Operator Performance 85

solve the situation among themselves. This enhances the risk of the situation, since
it is unclear to all participants how the situation will develop. They still will have to
maintain a proper situation awareness of the situation.
In the traffic scene in Fig. 3 there also is one ship which has not contacted the
VTS operator at all. This ship does need special attention of the VTS operator at all
times, since it is unknown what it will do and where it will go.
These potential conflicts that are offered in the scenarios, differ in severity, which
is expressed in the number of conflicts involved and the complexity of the situation.
Throughout the set-up of the experiments there have been discussions between trainers,
VTS operators, and researchers about the definition of conflicts and the interpretation
of significance of certain events. Emphasis has been put upon reaching common
ground for these aspects among different groups participating in the experiments.

6. Establishing Scoring System

In the tests, operators get scores for their performance. These scores are based upon
recognition and solution of potential conflicts in traffic. Establishing a scoring system
for VTS operator performance though, is not a straightforward task. Since in the
simulator exact positions of ships and other information are known, a maximum
obtainable score can be derived about each traffic situation tested. An individual
score can be calculated by comparing operator's scores with this maximum. This
score can be presented as a percentage of the maximum. However, this method is far
from perfect, since it does not account for the fact that some information is essential
for correct performance of VTS operators, while other information is practically
superfluous. This can be easily illustrated by the exact location, course and speed
of a ship entering a conflict situation versus a ship which has passed all points
of interaction with other traffic. The scoring system must account for this relative
"weight" of information.
The scoring system must also account for seriousness of a conflict. Commonly
applied numerical methods for determination of conflicts between ships, such as
TCPA (Time to Closest Point of Approach), are usually based on distances between
ships and information on speed and course. Criteria for acceptable TCPA depend
upon manoeuvring abilities of the ships involved. They do no take into account the
fact whether or not there has been communication with the ships about the pending
conflict. VTS operators, on the other hand, in their daily practise use a very different
definition. To them, a potential conflict which has been communicated, is not a
conflict anymore. On the other hand, a situation with potential for developing into a
conflict, even as much as fifteen or twenty minutes ahead, must be attended and
communicated. The scoring system therefore must take communication into account.
In an effort to establish a scoring system which is acceptable for all parties
involved, currently expert opinions on appropriate weighing scores are gathered and
combined. For each trial in the experiment VTS trainers determine the maximum
scores and deviations from maximum scores that are allowed. VTS operators who
serve as subjects in the experiments are confronted with the normative scores and
have a chance to defend their responses. This is useful for clearing up differences in
interpretation. Although scoring this way is a very time-consuming exercise, it is of
86 E. Wiersma and N. Mastenbroek

eminent importance for coming to a meaningful scoring system. This weighing takes
into account the importance, relevance and accuracy of responses. At the same time,
the importance of the conflict concerned is examined.

7. Conclusion

The task of VTS operators is well understood in terms of communication and

communication protocols, In this project this view is refined with the concept of
situation awareness.
A scoring system for assessment of situation awareness must account for the
importance of information. In this project importance has been expressed in accuracy
and relevance of responses given by VTS operators. Establishing a well balanced scor-
ing system is achieved by deployment of weighing factors. Expert opinions are used
to derive these weighing factors. In this context, not only trainers and researchers, but
also VTS operators are included as experts in arriving at a well balanced scoring system.

References

Adams, S.K., Babin, S.P. and Sabri, Z.A. (1996) The virtual patient - an application of situation awareness
for system design and training in intensive care. In: Garland, D.J. and Endsley, M.R. (eds.) Experimental
analysis and measurement of situation awareness. Embry-Riddle Aeronautical University Press,
Daytona Beach F1. USA. 329-338.
Dyck, J.L. and Gilson, R.D. (1994) Situation awareness in marginal weather conditions. In: Gilson, R.D.,
Garland, D.L and Koonce, J.M. (eds.) Situation awareness in complex systems. Embry-Riddle
Aeronautical University Press, Daytona Beach F1. USA., 251-259.
Endsley, M.R. (1995) Toward a theory of situation awareness in dynamic systems. Human Factors. 37(1)
32 -64.
Endsley, M.R. (1995) Measurement of situation awareness in dynamic systems, Human factors, 37(1)
65 - 84.
Endsley, M.R. (1996) Direct measurement of situation awareness in simulations of dynamic systems:
Validity and use of SAGAT. In: Garland, D.J. and Endsley, M.R (eds.) Experimental analysis and
measurement of situation awareness. Embry-Riddle Aeronautical University Press, Daytona Beach FI.
USA., 107-114.
Mogford, R.H (1997) Mental models and situation awareness in Air Traffic Control. International Journal
of Aviation Psychology, 7(4) 331- 341
Sarter N.B & Woods, D.D. (1991) Situation awareness: a critical but ill-defined phenomenon. International
Journal of Aviation Psychology, 1(1) 4 5 - 5 7
Wickens, C.D. (1995) Situation awareness: impact of automation and display technology. In: AGARD
Aerospace Medical Panel (eds.) Situation awareness: Limitations and enhancement in the aviation
environment. AGARD-CP-575 K2.1 - 13.
Wiersma, E. and Heijer, T. (1996) Human factors models and methodologies for vessel traffic service
operator performance. Report to: E.C./D.G. 7, Delft University of Technology, Delft.
Wiersma, E., Heijer, T. and Hooijer, J. (1997) SATEST, a method for measuring situation awareness in
Vessel Traffic Service operators. In: Soares, C.G. (ed.) Advances in safety and reliability. Pergamon,
Lissabon, 901-908.

Correspondence and offprint requests to: E. Wiersma, Safey Science Group, Delft University of Technology,
Kanaalweg 2 B, 2628 EB Delft, The Netherlands. Tel: +31 15 2787818; fax: +31 15 262235; email:
Wiersma@ wtm.tudelft.nl