You are on page 1of 48

Research Programme

Operations
Psychometric Testing - A review of the train driver selection process
Psychometric Testing – Review of Train Driver Selection Process
Recommendations for standardising and improving selection processes

Psychometric Testing – A review of


the Train Driver selection Process

Recommendations for standardising and improving the


selection process

© Copyright 2006 Rail Safety and Standards Board.

This publication may be reproduced free of charge for research, private study or for internal
circulation within an organisation. This is subject to it being reproduced and referenced accurately and
not being used in a misleading context. The material must be acknowledged as the copyright of Rail
Safety and Standards Board and the title of the publication specified accordingly. For any other use of
the material please apply to RSSB's Head of Research and Development for permission. Any
additional queries can be directed to research@rssb.co.uk. This publication can be accessed via the
RSSB website

www.rssb.co.uk

Produced by Competence Assurance Solutions Ltd Page 1 of 46


Psychometric Testing – Review of Train Driver Selection Process
Recommendations for standardising and improving selection processes

Contents

EXECUTIVE SUMMARY ..................................................................................................3

1 INTRODUCTION .....................................................................................................6

1.1 SUMMARY OF THE WORK CARRIED OUT IN THE PRECEDING DELIVERABLES .........................6

2 KEY FINDINGS .....................................................................................................10

2.1 SIFTING, SHORTLISTING AND FINAL SELECTION DECISIONS .........................................11


2.2 ASSESSMENT CENTRE PROCESS ...........................................................................14
2.3 ASSESSMENT CENTRE SELECTION METHODS ...........................................................19
2.4 USING PSYCHOMETRIC TESTS FOR DRIVER MANAGEMENT ...........................................24
2.5 CURRENT AND FUTURE TRAIN DRIVING ROLE ..........................................................27

3 RECOMMENDATIONS.............................................................................................28

3.1 SELECTION CRITERIA .......................................................................................28


3.2 THE DRIVER SELECTION PROCESS ........................................................................35
3.3 CONTENT OF DRIVER SELECTION METHODS ............................................................38
3.4 THE USE OF PSYCHOMETRIC TESTS IN DRIVER MANAGEMENT .......................................41

4 NEXT STEPS ........................................................................................................45

Produced by Competence Assurance Solutions Ltd Page 2 of 46


Psychometric Testing – Review of Train Driver Selection Process
Recommendations for standardising and improving selection processes

Executive Summary

Introduction
In November 2004, RSSB commissioned CAS to undertake a review of the processes
currently used for the selection and recruitment of train drivers with particular
reference to the use of psychometric testing.

This report presents a summary of the findings of the review and recommends
changes to the selection criteria and selection methods currently being used and to
the way psychometric tests are used in driver management.

Key findings
The British rail industry uses a recruitment process for train drivers which is
recognisably similar to those used for train driver recruitment overseas and in other
industry sectors. Almost all companies in Great Britain use the same four stage
process for recruitment, namely:

Stage 1: Attracting applicants.

Stage 2: Sifting / shortlisting.

Stage 3: Assessment Centre (involving psychometric testing and Criterion Based


Interviewing).

Stage 4: Final selection decision.

The assessment centres are recognised to provide a standardised way of ensuring


that all recruits to the industry meet minimum acceptable standards. However, a
number of issues have been identified of which the most important are:

• Nationally, there is no shortage of applicants for train driving. Some companies


experience very high application rates. The pass marks applied at the
assessment centres are fixed and only reject a small percentage of applicants.
This means that the bulk of selection is done at the sifting and final selection
stages using methods which have often not been technically evaluated and
carried out by staff who have not always been trained.

• Almost all companies go through the same stages, although the methods and
approaches used during the sifting and final selection stages vary.

• Some of the selection methods used at the assessment centres have proven to be
reliable and valid predictors of train driver performance. For others, evidence of
their validity is either weak or inconsistent.

Produced by Competence Assurance Solutions Ltd Page 3 of 46


Psychometric Testing – Review of Train Driver Selection Process
Recommendations for standardising and improving selection processes

• The selection criteria used at the assessment centres needs to be revised to fully
encompass the skills needed for modern train driving. Discussions with industry
representatives and comparison with overseas practice and other industry
sectors have identified a consistent set of new selection criteria. These are
related to the current set but give wider coverage of key requirements. However,
not all of the selection criteria are associated with common national
requirements. Only those which are required nationally should be assessed in
the assessment centres.

• Psychometric tests are used for four main purposes in driver management:

− Training and development.


− Post trauma counselling.
− Pre-incident counselling
− Post incident investigation.

Test use in the first two of these is sensible and conforms to good practice. Practice
in pre-incident counselling and post incident investigation is more problematic. It
is not clear that companies fully understand the purpose of testing in these
circumstances. As a result, some may not be using tests effectively.

Recommendations

The main recommendations of this project are that:

• The selection criteria should be updated to give better coverage of the abilities
recognised to underpin good performance in modern train driving.

• Selection criteria which address safety and train handling performance should
form the core of driver selection and be assessed by all companies in the same
way either by the assessment centres or by qualified individuals in companies.
Companies should have flexibility in the way in which they assess criteria which
relate to personal effectiveness.

• The effective parts of the current assessment centre process should be retained
for national use.

• The parts of the assessment centre process which have not proved effective
should either be replaced or upgraded.

• New tests should be identified or developed to assess the new or extended


selection criteria that have been identified

Produced by Competence Assurance Solutions Ltd Page 4 of 46


Psychometric Testing – Review of Train Driver Selection Process
Recommendations for standardising and improving selection processes

• Companies need to give greater thought to the use of psychometric tests for
post incident investigation. A process is outlined for how tests can be better
integrated into the investigation process.

Produced by Competence Assurance Solutions Ltd Page 5 of 46


Psychometric Testing – Review of Train Driver Selection Process
Recommendations for standardising and improving selection processes

1 Introduction

This report is the ninth deliverable of a project to review the processes currently
used for the selection and recruitment of train drivers, with particular reference to
the use of psychometric testing. It unfolds as follows:

• Section 2 sets out the key findings of the project. The findings have come from
research and data collected in five work packages. The main contributing
deliverables, described in more detail in section 1.1, are:

− Deliverable 2a: An analysis of current processes based on findings from an


industry questionnaire.
− Deliverable 4: Current recruitment processes and the use of psychometric
tests in selection (part A).
− Deliverable 5: The effectiveness of current recruitment processes and the
use of psychometric tests in driver management.
− Deliverable 6: Recruitment and selection practices in other industries and
overseas.
− Deliverable 7: Validation study of the current recruitment process and
review of the future train driving role.

In addition, work package 5 included an industry briefing to key stakeholders


where the findings from the research to date were presented and a number of
draft recommendations communicated. Stakeholder feedback was noted and
has been incorporated in this report as appropriate.

• Section 3 contains a number of recommendations from this project concerning


the selection criteria, selection process, selection method and the use of
psychometric tests in driver management.

1.1 Summary of the work carried out in the preceding deliverables

1.1.1 Deliverable 2a – An analysis of current processes based on findings


from an industry questionnaire

Deliverable 2a was based on findings from a questionnaire survey which covered:

• Driver recruitment processes.

Produced by Competence Assurance Solutions Ltd Page 6 of 46


Psychometric Testing – Review of Train Driver Selection Process
Recommendations for standardising and improving selection processes

• Processes and methods used in the initial sifting and shortlisting of applicants.

• The process of assessment centres (including psychometric testing and the


Criterion Based Interview (CBI)).

• Processes and methods used in the final selection decision.

• Recruitment difficulties.

• Use of tests in continued driver management and development.

Questionnaires were sent to a mailing list comprising members of the Driver


Manager Group and participants in a previous study undertaken for RSSB looking at
driver management practices. In total, 38 companies were emailed questionnaires
of which 31 replied.

1.1.2 Deliverable 4 – Current recruitment processes and the use of


psychometric tests in selection (part A)
Deliverable 4 was also partly based on information from the questionnaire survey
supplemented with 22 follow-up interviews. The interviews involved Driver
Managers, Operations Managers and Human Resources staff and had 3 main aims,
as follows:

• To clarify responses in the questionnaires which were ambiguous or unclear.

• To explore specific issues further.

• To identify how companies assess driver performance and how willing they were
to provide access to that information for validation purposes (Deliverable 7).

Data was also analysed from the eight national assessment centres which carry out
psychometric testing and interviewing on behalf of the industry. Analyses were
based on 1347 applicants for train driver posts assessed between April 2003 and
November 2004, for whom complete assessment centre data are available, and
4606 applicants between November 2000 and April 2003 for whom assessment
centre results and limited personal information is available.

1.1.3 Deliverable 5 – The effectiveness of current recruitment processes and


the use of psychometric tests in driver management
This report reviewed the effectiveness of the current recruitment process,
comparing findings from Deliverables 2a and 4 with good practice in selection and
recruitment and psychometric testing.

Produced by Competence Assurance Solutions Ltd Page 7 of 46


Psychometric Testing – Review of Train Driver Selection Process
Recommendations for standardising and improving selection processes

1.1.4 Deliverable 6 – Recruitment and Selection Practices in Other Industries


and Overseas
Data for this stage of the research was gathered in two phases:

• A literature review to compare practice and findings in the use of psychometric


testing and other recruitment and selection methods in overseas rail
organisations and other industries.
• A questionnaire sent to rail organisations overseas to gather detailed information
on driver recruitment and selection practices.

The research on other industry sectors looked at recruitment and selection practices
and methods in car, tram and bus drivers, pilots, air traffic controllers, fire fighters,
the Royal Navy, the Merchant Navy, the Offshore Oil and Gas industry, shipping
industry, the Royal Air Force and the nuclear industry (BNFL).

The overseas research looked at recruitment and selection practices and methods in
Australia, Austria, Belgium, Denmark, Estonia, France, Finland, Germany, Hungary,
India, Italy, Lithuania, Luxembourg, Netherlands, Northern Ireland, Norway, Poland,
Spain, Sweden and Switzerland.

1.1.5 Deliverable 7 – Validation study of the current recruitment process and


review of the future train driving role
This report considered the validity of the selection methods used in the current
assessment centre process and whether the selection criteria and selection methods
currently being used should be changed in light of recent and anticipated changes
to the train driver role.

The validation exercise involved collecting performance data from all the companies
which had expressed an interest in contributing to the validation study. The data
included train handling performance, safety records, performance in following
procedures, work attitude and training records. The performance data were then
matched to assessment centre records. Eighteen companies are represented in the
analyses with performance data available for 373 drivers who had been recruited
since 1999. Most of the sample were drivers of passenger trains (both conventional
and high speed). There were some freight and on-track machine drivers but
insufficient to undertake separate statistical analyses.

Current and future train driver job demands were explored through desk research
and three workshops with 24 industry representatives from 14 separate companies

Produced by Competence Assurance Solutions Ltd Page 8 of 46


Psychometric Testing – Review of Train Driver Selection Process
Recommendations for standardising and improving selection processes

(including RSSB). The workshops involved focused discussions, which took into
account both the current competence standards for drivers and the implications of
the introduction of ERTMS, and a Repertory Grid exercise.

Produced by Competence Assurance Solutions Ltd Page 9 of 46


Psychometric Testing – Review of Train Driver Selection Process
Recommendations for standardising and improving selection processes

2 Key findings

There is considerable variety in the recruitment practices used by companies when


recruiting train drivers. They vary in terms of:

• The methods used to attract applicants.

• The types of applicants encouraged to apply (e.g. a small number of companies


only recruit experienced train drivers).

• The ratio of applicants to job vacancies (e.g. several companies report occasions
when they have had more than a thousand applicants per vacancy while others
typically only have five or six).

• The methods used to select applicants.

• The staff used in the selection process.

• The selection criteria used across the whole selection process.

Nonetheless, all but two companies use the following four stage recruitment
process:

Stage 1: Attracting applicants.

Stage 2: Sifting / shortlisting.

Stage 3: Assessment Centre (involving psychometric testing and CBI).

Stage 4: Final selection decision.

The two companies which do not follow this process only recruit experienced drivers
and do not require applicants to go to an assessment centre or to sit psychometric
tests. This four stage process is the same as that used in most train driver
recruitment in other countries and in other sectors where safety critical staff are
recruited. However, the selection methods used at each stage can vary
considerably.

All companies use application forms in the second stage for sifting/shortlisting
although the sophistication of their use varies markedly. A number of companies
use a scored checklist approach while others rely on subjective assessment. A
number of selection methods are used in the sifting and final selection stages. For
example, some companies use additional psychometric testing at either or both
stages. It is also quite common for interviews to be carried out at both these
stages. Where interviews are carried out at the sifting stage, these are often done

Produced by Competence Assurance Solutions Ltd Page 10 of 46


Psychometric Testing – Review of Train Driver Selection Process
Recommendations for standardising and improving selection processes

by driver managers or production managers. Otherwise shortlisting is usually


undertaken by Human Resources (HR) staff.

The final selection stage usually involves a wider range of staff, typically involving a
mix of HR staff, driver managers and production or operations managers.
Sometimes, particularly in smaller companies, senior managers and directors will
also be involved in the final stage.

2.1 Sifting, shortlisting and final selection decisions

2.1.1 Applicant ratios and selection


Although the emphasis in this project has been on the selection methods used at
the assessment centres, it is important to note that most selection decisions are
made at the sifting and final selection stages. There are three main reasons for this:

1. On average, there are 317 applicants for every vacancy. This number has to be
reduced for cost reasons before applicants are sent to an assessment centre.

2. Many companies report problems with large numbers of poor quality applicants.

3. Many companies also report problems with large numbers of applicants


dropping out after they have been invited to interview or attend an assessment
centre. On average, drop outs account for up to a third of the number of
applicants at every stage of the process. Part of the reason for this is the
common practice of companies holding applications on file, sometimes for
significant periods of time. Understandably, many applicants will already have
found other employment by the time they receive an invitation to attend for an
assessment. Unwillingness to travel to assessments and self-assessment against
job specifications may also play a significant part.

The following table shows an example of both the selection ratios and the number
of applicants affected at each stage in a typical recruitment scheme.

Recruitment / Number of % passing to Number Number failing


Selection Stage Applicants next stage passing stage stage
Advertise
317 - - -
Vacancy
Drop outs
317 66% 211 106
before sift
Sifting Stage 211 31% 65 146

Produced by Competence Assurance Solutions Ltd Page 11 of 46


Psychometric Testing – Review of Train Driver Selection Process
Recommendations for standardising and improving selection processes

Drop out
65 66% 43 22
before A/C
Assessment
43 41% 17 26
Centre
Drop outs
before final 17 66% 10 7
stage
Final selection
10 48% 4 6
stage
Drop outs
4 33% 2 2
before medical
Medical
2 80% 1–2 0–1
Examination
Recruited
- - 1 -

Table 1: Example of pass rates in a typical recruitment scheme

In this example, the assessment centres are only responsible for rejecting 26
applicants or just over 8% of applicants who responded to the advertisement. By
comparison, drop outs account for 43% of applicants while the sifting and final
selection stages account together for 48% of applicants.

2.1.2 Effectiveness of methods used for sifting, shortlisting and final


selection decisions
Given that so much selection work is done in the sifting and final decision stages, it
is important to consider the relationship between the psychometric tests and CBI
used at the assessment centres and the selection methods used in other recruitment
stages. A wide variety of methods are used, including:

• Application form sifts.

• Interviews.

• Psychometric tests of ability.

• Psychometric tests of personality.

• References.

• Exercises such as role play simulations and writing tasks.

Little is known about the reliability and validity of these methods for train driver
recruitment. A variety of staff are involved in the delivery of these methods. Where

Produced by Competence Assurance Solutions Ltd Page 12 of 46


Psychometric Testing – Review of Train Driver Selection Process
Recommendations for standardising and improving selection processes

psychometric tests are used, staff trained to at least British Psychological Society
Level A standard always seem to be used. However, staff are not always trained in
the use of other methods or in processes for making assessment and selection
decisions. That interviewers are often untrained is a key issue given the extensive
use of interviews. This is clearly different from practice at the assessment centres
where only trained interviewers are used.

2.1.3 Reasons for use of other methods


There are four main reasons why other selection methods are used:

1. The pass marks at the assessment centres are pre-set. This means that the
rejection rate is more or less fixed although there is some variation from
company to company depending on the quality of their applicants and the
amount of sifting they do. This in turn means that companies with high ratios of
applicants to vacancies have to use other selection methods to reduce applicants
to manageable numbers.

2. Although all companies are aware of the recommended selection criteria for
driver recruitment, many either introduce new criteria during the recruitment
process or extend the coverage of the recommended criteria. An example of the
former is requiring applicants to have experience of technical or engineering
work. An example of the latter is requiring applicants to undertake a writing
exercise.

3. Most companies recognise the value in having their own staff involved in the
selection process. This gives staff a sense of ownership in the selection
decisions and allows companies to recruit staff who will fit the company culture.

4. Sending applicants to an assessment centre is considered to be expensive. Many


companies, therefore, put effort into restricting the numbers of applicants who
attend an assessment centre.

2.1.4 Maximising pass rates at the assessment centres


The obvious way to restrict the number of applicants attending assessment centres
is to shortlist only those applicants who are likely to pass the assessment centre. A
number of companies have adopted such a strategy and claimed success. One
passenger train company reported a success rate of over 95% of applicants passing
the assessment centre, compared to the national average of 41%. In a number of
cases, companies use the psychometric tests which would be given at the
assessment centres as part of their initial sifting process, using the same pass

Produced by Competence Assurance Solutions Ltd Page 13 of 46


Psychometric Testing – Review of Train Driver Selection Process
Recommendations for standardising and improving selection processes

marks. Applicants in such cases are excused taking the tests again at the
assessment centres although the results are entered in their assessment centre
results sheet.

This practice calls into question the need for such companies to use assessment
centres. It seems that most companies treat the assessment centres as a mandated
process rather than a process recommended in RACOP GO/RC3551. In practice,
there is no reason why companies with the required competence to administer
psychometric tests and conduct the CBI should send applicants to an assessment
centre at all. Alternatively, it is also reasonable for some companies to require
applicants only to undertake part of the process if they can demonstrate that they
have covered other parts of the process adequately elsewhere.

2.2 Assessment centre process


Data was available from eight national assessment centres which carry out
psychometric testing and interviewing on behalf of the industry. The data was
provided by Springfield Training which maintain a database for the industry of all
candidates who attend the assessment centres. Analyses are based on 1,347
applicants for whom complete assessment centre data are available assessed
between April 2003 and November 2004, and 4606 applicants between November
2000 and April 2003 for whom partial assessment centre data was available.

The standard assessment centre process consists of four psychometric tests and a
CBI. The tests provide between 22 and 33 scores, depending on which versions are
used, of which 10 or 13 are used to make assessment decisions. These assessment
scores are sometimes individual test scores and sometimes combinations of scores.
They are used to provide assessments for six of the twelve recommended selection
criteria. The other six selection criteria are covered by the CBI. Table 2 shows
which selection methods are used to assess which selection criteria.

Selection criterion Assessment Method


Ability to maintain vigilance and Concentration Test (Group Bourdon).
concentration at all times.
Ability to react quickly and safely to Reaction & Co-ordination Test
changing situations. (Determinations Gerat or DTG).
Ability to operate hand and foot Reaction & Co-ordination Test
controls. (Determinations Gerat or DTG).
Ability to recall and retain job related Trainability for Rules & Procedures Test
information. (TRP parts 1 and 2).

Produced by Competence Assurance Solutions Ltd Page 14 of 46


Psychometric Testing – Review of Train Driver Selection Process
Recommendations for standardising and improving selection processes

Ability to learn new information Trainability for Rules & Procedures Test
within appropriate time limits. (TRP parts 1 and 2).
Ability to understand mechanical Mechanical comprehension test (MT4).
principles.
Ability to communicate clearly and Criterion Based Interview.
effectively verbally and in writing.
Motivation to follow rules and Criterion Based Interview.
procedures.

Ability to remain calm in emergency Criterion Based Interview.


and/or stressful situations.
Conscientiously works to meet Criterion Based Interview.
training course and job demands.
Is proactive and tenacious. Criterion Based Interview.

Can spend time alone and do so Criterion Based Interview.


effectively.

Table 2: Assessment Centre selection criteria and assessment methods

It is important to note that the assessment centre design conforms to the


educational definition of an assessment centre, that is, a location where
assessments are carried out. It does not conform to the definition of an assessment
centre current in most occupational recruitment and development practice. This
would require the centres to use a greater range of assessment methods,
particularly simulation exercises such as role plays, and for each selection criterion
to be assessed using more than one method.

It can be argued that many of the selection criteria are assessed more than once in
the complete recruitment process and that the complete recruitment process does
conform to the definition. However, this depends on individual companies and it
cannot be guaranteed currently that selection criteria are systematically assessed by
more than one method. Not all companies have systematically considered the
relationships between the methods used at different stages and how coverage of the
selection criteria is provided. This has important implications for the reliability and
validity of the recruitment process.

It should also be noted that although most companies send applicants to an


assessment centre there is a small number which do not. These are companies
which only recruit experienced drivers, although it should be noted that some

Produced by Competence Assurance Solutions Ltd Page 15 of 46


Psychometric Testing – Review of Train Driver Selection Process
Recommendations for standardising and improving selection processes

companies which recruit experienced drivers nonetheless require them to attend an


assessment centre.

2.2.1 The purpose of the assessment centres


One of the benefits of the assessment centre process recognised by companies is
that it provides a standardised process and sets minimum acceptable requirements
for the whole industry. What is less well understood is that the assessment centre
process is not designed to identify the best or even good drivers. It is designed to
ensure that applicants who would be unsuitable for employment as drivers, either
because they are at risk of being unsafe drivers or of being unable to cope with the
demands of the training and the work, are not recruited. This can be seen in a
number of features of the assessment centre process:

• Pass rates are high on all the individual assessment centre selection methods.

• Many companies reject applicants as soon as they have failed one test (at least
25%) and do not require them to complete the process. Only about 25% of
companies require applicants to complete the process whether or not they have
passed all the tests.

• Indeed, the failure rate for all the criterion based interview criteria combined is
only 5%.

• The pass rate is particularly high on one of the new selection criteria, being
proactive and tenacious. In our assessment centre sample, not one applicant
was failed on this criterion.

• The only reason that the average success rate at the assessment centres is as
low as 41% is because there are so many relatively independent selection criteria.

• There are more fail grades than pass grades with poor differentiation amongst
the pass grades.

Note that the high pass rate on the CBI does not mean that it is a poor assessment
method. There are a number of reasons why the pass rates should be so high. The
most obvious is that applicants have already being selected on the selection criteria
addressed by the CBI during the shortlisting and sifting stage. In particular, many
companies interview applicants before sending them to an assessment centre.
Nonetheless, this does raise the question of whether the CBI is providing all the
value that it could or should.

Produced by Competence Assurance Solutions Ltd Page 16 of 46


Psychometric Testing – Review of Train Driver Selection Process
Recommendations for standardising and improving selection processes

2.2.2 The validity of the recommended selection criteria


With one exception, all the current selection criteria are recognised as being
important aspects of train driving competence. The exception is the ability to react
quickly. Reacting safely is seen as a crucial ability for drivers but speed of reaction
is not thought to be so important. There are few situations where driver reaction
times need to be measured in anything other than several seconds. Anticipation
and planning are thought to be more important abilities.

Although the relevance of the selection criteria is recognised, this does not mean
that the recommended criteria are the best possible selection. The current selection
criteria are a mix of safety, performance and trainability criteria pitched at varying
levels of generality. For example, “ability to communicate clearly and effectively” is
a very general criterion which makes no reference to the specific sorts of
communications train driving involves. “Ability to operate hand and foot controls” is
an example of a very specific criterion.

Having both very general and very specific criteria can create problems:

• Very general criteria usually need to be accompanied by more specific contextual


guidance; otherwise it is difficult to accurately assess performance against them.

• A very specific criterion is usually just one of a subset of criteria which make up
a more general criterion. The question arises as to why one or two specific
criteria should be chosen and not others. For example, “ability to operate hand
and foot controls” is just one of a range of train handling skills which might be
assessed.

The current selection criteria, and the methods used to assess them at the
assessment centres, do not appear to have got the balance of generality and
specificity right. This is one reason why individual companies find it necessary to
add extra criteria to the selection process or to assess further dimensions of the
existing criteria. For example, it is not clear that it is necessary to assess separately
for both “ability to recall and retain job related information” and “ability to learn new
information within appropriate time limits” when they are such closely related
aspects of trainability.

2.2.3 Assessment against selection criteria at assessment centres


The current assumption is that all selection criteria should be covered and assessed
for at the assessment centre. This does not necessarily follow. For example, there
is not a consensus that “ability to understand mechanical principles” should be

Produced by Competence Assurance Solutions Ltd Page 17 of 46


Psychometric Testing – Review of Train Driver Selection Process
Recommendations for standardising and improving selection processes

retained in the selection criteria. Most passenger and freight companies consider
fault diagnosis to be a more important skill than mechanical comprehension.
Furthermore, most drivers no longer take responsibility for even minor fixes and
repairs. This trend has been recognised before, with a fault finding test included in
trials in a previous validation study1. However, other companies, particularly those
employing track machine drivers, still see a need for mechanical comprehension
ability.

There is a clear case for setting minimum acceptable standards for the whole
industry on selection criteria concerned with safety, such as vigilance. The case for
general performance and trainability criteria is less clear. The reasons for including
such criteria are often commercial. Companies differ in the level of ability they
require on such criteria even if they are recognised to be important criteria by all.
Where companies have different requirements for performance and trainability
criteria, there is a case for them to either set their own pass marks or to use their
own selection methods as long as these decisions are taken by qualified individuals.

2.2.4 Maintenance of the assessment centre database


A master copy of the assessment centre applicant database is maintained on behalf
of the industry. In the course of analysing the assessment centre data, a number of
problems with the accuracy and reliability of the assessment centre database have
arisen. The most important are as follows:

• A number of candidate records are missing.

• A significant proportion of the dates for date of birth and date of assessment are
incorrect.

• In almost 20% of the cases the recording of job experience is inaccurate.

• A significant number of test scores entered in the records are outside the
possible range of scores for the tests.

• There is inconsistency in the way test scores are entered.

• The old and new versions of the Mechanical Comprehension Test are kept
separate in the records but the old and new versions of the TRP test are being
entered in the same field making it difficult to distinguish them.

• When applicants re-sit, sometimes scores from previous sittings of tests are re-
entered in the relevant fields but sometimes the fields are left blank.

1 Fletcher, S. (2002) Predicting the Effective and Safe Train Driver: Report of Validation
Findings and Recommendations for Action. Occupational Psychology Centre Ltd., Watford.

Produced by Competence Assurance Solutions Ltd Page 18 of 46


Psychometric Testing – Review of Train Driver Selection Process
Recommendations for standardising and improving selection processes

Some of these problems, for example missing candidate records, resulted from
problems at the time the database was set up and should not be a problem in the
future. Others, such as recording test scores on different versions, will disappear
once stocks of old test materials have run out. For the other problems, two steps
need to be taken:

• Consistency needs to be achieved across the different assessment centres


wherever possible.

• Many of the data entry errors could be trapped by having field validation in the
database.

2.3 Assessment centre selection methods


Technical evaluation of the selection methods involves assessment against three
main criteria:

• Reliability – do the selection methods consistently produce the same results?

• Validity – do the selection methods accurately measure what they are supposed
to measure?

• Utility – do the different selection methods all add value to the selection process?

2.3.1 Reliability
Although reliability information was not available to us for all the selection methods,
where it does exist, the indications are that most individual measures are highly
reliable. However, there are two concerns:

1. The score bands used for making grading decisions at the assessment centres
are not reliable for all test scores. In some cases, the difference between a grade
A and a grade C is not statistically significant. This is particularly a problem with
the ‘error’ and ‘omission’ scores in the DTG and Group Bourdon.

2. A number of new versions of the tests have been introduced. Although care was
taken when introducing these new versions to make the pass marks and score
bands equivalent, different versions of the tests produce different pass rates.
This affects three of the tests, the Group Bourdon, the Mechanical
Comprehension and the Trainability for Rules and Procedures and occurs for
technical reasons to do with the way the scores are used for making decisions.
In particular, the computerised version of the Group Bourdon and the paper and
pencil version are not equivalent tests.

Produced by Competence Assurance Solutions Ltd Page 19 of 46


Psychometric Testing – Review of Train Driver Selection Process
Recommendations for standardising and improving selection processes

As noted previously, problems related to the existence of different versions should


resolve themselves as stocks of old materials are used up and all the assessment
centres are using the same versions.

2.3.2 Validity
The validity of the psychometric tests used at the assessment centres has been
examined several times since their introduction. A variety of performance criteria
have been used in these studies including both objective and subjective measures.
The current study mainly used performance data collected from driver managers
under the following 15 headings:

• Train handling (Acceleration and braking technique / Speed and distance


judgment)
• Application of rules
• SPAD record
• Collisions or derailment incidents (inc. buffers)
• Speeding record
• Station overrun record
• Station disregard record
• Operation / isolation of safety systems
• Preparation, disposal and handover of trains
• Workplace formal assessment (both practical and underpinning knowledge)
• Attitude to work performance or people
• Written / spoken communications - e.g. submission of reports relating to
safety/performance incidents or train handover arrangements.
• Handling abnormal events / calmness under pressure
• Classroom training
• Practical training

In addition, for a subset of 157 drivers more detailed performance data was
collected from personnel records, on attendance and timekeeping, and examination
and assessment records from training.

Statistical analysis showed that these performance criteria could be grouped under
five main headings:

1. Overall train handling ability


2. Procedure and rule based work

Produced by Competence Assurance Solutions Ltd Page 20 of 46


Psychometric Testing – Review of Train Driver Selection Process
Recommendations for standardising and improving selection processes

3. Safe performance
4. Formal competence and work attitude
5. Classroom examinations

Validation of the psychometric tests and interviews was conducted against these
performance groupings. Details of how the data were collected and the guidance
given to driver managers can be found in Deliverable 7: Validation study of the
current recruitment process and review of the future train driving role.

All the selection methods used at the assessment centres have shown some
evidence of being valid predictors of either driving performance or trainability, but
the results have been inconsistent. This inconsistency is also apparent in the
current study. For example, the DTG has been reported in the past to have a
positive relationship with training performance, job performance and safety
incidents. However, the previous validation study found no predictive relationships.

This study found no positive relationships for the DTG with training outcomes, nor
with overall train handling (e.g. control of acceleration, braking and speed), but did
find some modest significant correlations (in the range 0.13 – 0.16) with aspects of
safety performance and procedure based work such as the operation/isolation of
safety systems and train preparation, disposal and handover. Three of the four DTG
scores is associated with a significant predictive correlation but in each case with
only one aspect of driving performance.

The results for the Group Bourdon test are also inconclusive. There are significant
positive correlations for both the computer and paper and pencil versions of the test
but little consistency in which parts of the tests produce these results. So, for
example, the Time measure on the computer version of the test shows some
moderate correlations (in the range 0.21 to 0.24) with procedure based work,
classroom training performance and safe performance, but the Omissions and Error
scores produce no significant correlations. In contrast, the paper and pencil version
of the test produces no significant correlations for the Production score (regarded as
similar to the Time score) but several significant correlations for the Omissions and
Error scores, particularly with classroom training performance, in the range 0.31 to
0.43). Two important conclusions can be drawn from this result:

1. The computer version and paper and pencil version of the Group Bourdon are
not equivalent tests.

Produced by Competence Assurance Solutions Ltd Page 21 of 46


Psychometric Testing – Review of Train Driver Selection Process
Recommendations for standardising and improving selection processes

2. Although the tests may have some value they do not measure what they are
expected to measure. Although they may be measures of attention and
information processing, they are almost certainly not measures of vigilance.

Both parts of the TRP test consistently correlate positively with relevant criterion
measures. For example, both show significant correlations with classroom training,
practical training, procedure-based work and safe performance (in the range 0.16 to
0.39). There is also evidence that the new version of the TRP is a better predictor
than the old version, almost certainly because there is a wider score range which will
make the tests more reliable.

The mechanical comprehension tests (MT4.1) correlates significantly with both


classroom training and practical training (0.23). However, there are no significant
correlations with any of the measures of driving performance, either train handling
or safe performance. Nor are there any significant correlations with the softer
measures of performance such as attitude or commitment.

There is little evidence for the validity of the CBI. Only two of the criteria assessed
using the CBI (“follow set rules and procedures” and “being proactive and tenacious”)
show any significant predictive relationships, the former producing two low but
statistically significant correlations of 0.13 with safe performance and classroom
training, the latter correlating 0.35 with procedure-based work. Otherwise, there is
no consistent evidence for the validity of other parts of the CBI although some
evidence has been reported in previous studies. The very high pass rates and the
lack of differentiation amongst the pass grades almost certainly reduce the chances
of finding statistically significant results.

Nonetheless, there is evidence that the CBI has some construct validity. It
differentiates better between the various selection criteria than is often the case
with interviews, the correlations between the ratings on the six criteria being
moderate (averaging 0.32). Furthermore, the CBI seems to provide sufficiently
different information to the tests, the correlations again typically being in the range
0.3 to 0.35.

Overall, the selection methods appear to be better at predicting trainability rather


than train handling or safety performance. However, the TRP seems to also predict
driver performance well. There is evidence to suggest that parts of the DTG and the
Group Bourdon may be valid predictors but these parts also have moderate to high
correlations with the TRP and do not seem to have much added value. This effect is
more pronounced for the computerised Group Bourdon. Indeed, both the results of

Produced by Competence Assurance Solutions Ltd Page 22 of 46


Psychometric Testing – Review of Train Driver Selection Process
Recommendations for standardising and improving selection processes

this study and other published research suggest that the paper and pencil version of
the test is of greater value than the computer version. The inconsistency in the
validity findings over time suggests that other parts of the process do have some
value but the predictive validities are probably low (0.20 or less) which is why they
are not found in every validation study.

2.3.3 Test resits


Applicants who pass the assessment centres after their second attempt appear to
perform as train drivers as well as applicants who pass first time. They are also
significantly more committed to both training and the job. This finding is consistent
with research in other sectors. For example, Hausknecht et al2 argue that this is an
example of how persistence and commitment can be recognised in applicants. This
suggests that applicants ought to be allowed to resit the assessment centre tests
more than once. However, they should be allowed no more than three attempts in
total since inability to reach the required standard by the third attempt may indicate
weaknesses which will be difficult to overcome. Applicants do have a slightly higher
pass rate on resit. However, performance on resits only improves on some of the
application methods (the TRP and the interviews) but not on the Group Bourdon,
DTG or MT4.1.

2.3.4 Other selection methods


Other types of selection evidence may need to be given more attention. As noted,
little is known about the validity of some of the assessment methods used during
sifting and final selection. However, the validation study has indicated that
academic achievements are an important predictor of train driver performance and
trainability, with some reservations. Better qualified recruits tend to perform better
on average in most aspects of train handling and safety performance but seem to be
less dependable as employees because they have poorer time-keeping and
absenteeism records. There is also evidence from previous studies and from
practice overseas and in other sectors that other tests and personality measures
may be worth considering

2.3.5 Utility
Four criteria need to be considered when considering the utility of a selection
process:

2 Hausknecht, J.P., Trevor, C.O. and Farr, J.L. (2002) Retaking ability tests in a selection
setting: Implications for practice effects, training performance, and turnover. Journal of
Applied Psychology. 87 (2), 243-254.

Produced by Competence Assurance Solutions Ltd Page 23 of 46


Psychometric Testing – Review of Train Driver Selection Process
Recommendations for standardising and improving selection processes

1. The validity of the selection methods.

2. The cost of putting applicants through the selection process.

3. The benefits resulting from making better selection decisions.

4. The added value of each assessment method.

For train driver recruitment the costs associated with psychometric testing and
interviewing are trivial compared to the benefits of making better selection
decisions. The cost of a train driving incident far outweighs recruitment costs.
Ensuring that an applicant who is at risk of having an accident is not recruited easily
pays for itself.

However, recruitment costs could be reduced without affecting the validity of the
process. With some important exceptions, the different methods used at the
assessment centres, both tests and interviews, are relatively independent of each
other. This means that, as long as the methods are valid, each assessment method
will add value to the selection process. The exceptions are some parts of the paper
and pencil version of the Group Bourdon, the `good’ and `omissions’ scores on the
DTG, and the Time measure of the computerised Group Bourdon where there are
moderate to large inter-correlations. There is little chance of one of these scores
adding value to the others. Also, a number of scores are recorded for the various
tests but not used in the selection decision. The scoring, recording and use of these
correlated scores should be rationalised and simplified.

The key finding is that several of the selection methods appear to have poor, or at
best modest, validity. In such circumstances, the utility of the selection process
must also be limited. This fact has been recognised in the past by lowering the pass
marks on some tests (e.g. the Group Bourdon). Such a step further reduces the
utility of the selection method since, if the pass rate gets very high, there is less
chance of the method realising its full benefit. As noted, the pass rates on all the
methods used at the assessment centres are high so the utility of the total process
is almost certain to be relatively low.

2.4 Using psychometric tests for driver management


The use of psychometric tests for driver management is not required practice in the
industry although it is recommended in GO/RC3551(Approved Code of Practice –
Train Driving) Part F as one of the possible interventions to support drivers in the
Driver Care and Support System (DCASS). This partly explains why practice is

Produced by Competence Assurance Solutions Ltd Page 24 of 46


Psychometric Testing – Review of Train Driver Selection Process
Recommendations for standardising and improving selection processes

inconsistent across the industry. Industry opinions vary as to whether tests give
value.

Tests are used for four purposes:

• Training and development, including promotion and career progression.

• Post trauma counselling.

• Pre-incident counselling (special monitoring)

• Post incident investigation.

Just over half of companies use tests for one or other of these purposes. About a
third use tests for post incident investigation, 30% for training and development and
15% for post trauma counselling and pre-incident counselling. However, even in
those companies where tests are used for these purposes, there is a degree of
unease about their appropriateness.

2.4.1 Use in post trauma counselling


This is regarded as a specialist activity. Companies using psychometric tests for
this purpose send drivers to competent counsellors who make the decision about
which tests to use. The choice will depend on the counsellor and the circumstances.
Therefore the tests administered vary from one use to another. It would make sense
in some circumstances to include re-testing on the recruitment tests if it is believed
that the traumatic event may have led to deterioration in performance in these
areas.

2.4.2 Use in training and development


Tests are used in two ways for training and development:

1. Performance on the assessment centre tests is sometimes used to guide and


tailor driver training. The most usual approach is to identify areas where there
are concerns that the new recruit may struggle and to offer extra help or
guidance as necessary.

2. Tests are sometimes used as part of the promotion process from driver to roles
such as driver instructor and driver manager. Indeed, GO/RC3551 (pp132)
contains recommendations for the use of various tests (e.g. RAAT, VT5.1,
SAFEPQ) which are offered as ‘examples of suitable assessment tools” although
“there may be other exercises or assessment tools that would assess these key

Produced by Competence Assurance Solutions Ltd Page 25 of 46


Psychometric Testing – Review of Train Driver Selection Process
Recommendations for standardising and improving selection processes

criteria and are available off the shelf or could be designed bespoke by the
assessor’. When used in this way, the tests should be used to assess ability to
cope with demands of the new role. It follows that there is no point in using the
same tests as were used in driver recruitment. In fact, a wide variety of different
tests are used by companies including ability and personality tests and practice
appears to be entirely appropriate.

2.4.3 Pre-incident counselling and post incident investigation


The appropriateness of the use which is made of tests for pre-incident counselling
and post incident investigation is less certain. It is not clear that companies
properly understand the purpose of using tests for these purposes. Nor is it always
the case that the recipients and users of test information have been trained in this
area. Companies also need to consider whether the use of test scores contravenes
the Data Protection Act. The Act seems to require that test takers should be aware
of and agree to any future use which might be made of their test scores at the time
that they sit the tests. It is not clear that this happens as a matter of course.

Typical practice is to re-test drivers on the tests used at the assessment centres.
Except in some specific circumstances, this practice makes little sense. Performance
on tests tends to be stable over time. Unless there are reasons to believe that the
driver has either experienced a traumatic event or suffered a deterioration in his or
her mental or physical health there is no reason to suppose that test performance
should change over time. If there is a psychological reason for the incident and no
evidence of a significant life event, the task for the investigator is to identify aspects
of individuals’ performance which are not assessed by the recruitment tests. As
noted, the assessment centre tests only measure a subset of the skills and abilities
involved in train handling and safety performance. Coverage of all these abilities is
a necessary element of good practice in post incident investigation.

Where there are reasons to suspect that a driver’s personal circumstances have
changed significantly, a case can be made for re-testing on the recruitment tests.
The assumption would be that, as in post trauma assessment, the driver’s
performance may have deteriorated. Note, however, that deterioration may occur in
other areas of performance and it is important to ensure that assessment is
comprehensive. Furthermore, deterioration in performance may result from the
stress of being tested again rather than from other causes. In any case, the fact that
there are some concerns about the validity of some of the tests and their ability to
predict safe performance suggests this should be undertaken with caution.

Produced by Competence Assurance Solutions Ltd Page 26 of 46


Psychometric Testing – Review of Train Driver Selection Process
Recommendations for standardising and improving selection processes

Since testing needs to be tailored to the circumstances of the incident and the
individual, it follows that testing should not be the first port of call in an
investigation. Interviews, observation and performance on simulators can all be
used to provide performance evidence and to help investigators develop hypotheses
about the causes of incidents.

2.5 Current and future train driving role


As noted in section 2.2.2, there are reasons to suggest that the current
recommended selection criteria may not be the most appropriate. Discussions with
industry representatives, research into practice in the recruitment of train drivers
overseas and research into the recruitment of staff to safety critical roles in other
industrial sectors, have consistently identified a number of relevant selection criteria
which are commonly used elsewhere but not in Great Britain. These criteria fall into
three main classes:

• Criteria associated with other aspects of safe driving performance. These


include being able to estimate time to collision, coping with multi-tasking,
decision making and interpreting instrument readings.

• Personality characteristics not covered in the current selection criteria. These


include extraversion, sensation seeking and sociability.

• Wider conceptualisation of characteristics which are currently assessed such as


including dependability in the definition of conscientiousness and reactivity to
stress in the definition of remaining calm in stressful situations.

In addition, many Human Factors issues were identified which are related to the
introduction of ERTMS (the European Rail Traffic Management System) and other
anticipated changes to the driver role. Interestingly, consideration of these issues
did not result in the identification of any additional selection criteria. All the
demands which these changes will make on driver performance already exist even
though the level of demand may vary.

Produced by Competence Assurance Solutions Ltd Page 27 of 46


Psychometric Testing – Review of Train Driver Selection Process
Recommendations for standardising and improving selection processes

3 Recommendations

The recommendations are presented in four sections:

Section 3.1: Recommendations concerning selection criteria.

Section 3.2: Recommendations for the driver selection process.

Section 3.3: Recommendations for the content of driver selection methods.

Section 3.4: Recommendations on the use of psychometric tests in driver

management.

3.1 Selection Criteria


On the basis of the evaluation of current and future train driving demands, it is
recommended that the following selection criteria should be adopted:

• Attention. • Conscientiousness.

• Vigilance. • Rule Compliance.

• Train Handling Aptitude. • Achievement Need.

• Decision Making. • Emotional Stability.

• Trainability. • Tolerance of working alone.

• Communication skills.

Figure 1 below shows the relationships between the current selection criteria and
the recommended replacements. The key points to note are that:

• All the existing criteria are covered somewhere in the new criteria.

• In some cases several criteria have been combined in one selection criterion (e.g.
trainability) while in others one criterion has been split into two (Attention and
Vigilance).

• Where appropriate, the same or similar terminology has been retained, but where
there is a significant change in emphasis, more substantial changes have been
made to the wording.

Produced by Competence Assurance Solutions Ltd Page 28 of 46


Psychometric Testing – Review of Train Driver Selection Process
Recommendations for standardising and improving selection processes

• The number of selection criteria has been reduced.

• Some of the new selection criteria (e.g. achievement need) might best be
assessed by companies themselves rather than at an assessment centre. There
are several criteria for deciding which criteria might best be assessed by
companies which include:

− the need for specialist or specially trained assessors


− whether a criterion can be applied to all companies at the same level
or not
− whether the interpretation of a criterion is influenced by company
culture
− whether the criterion applies to all recruits or only some (e.g. are
different types of driver employed)

Produced by Competence Assurance Solutions Ltd Page 29 of 46


Psychometric Testing – Review of Train Driver Selection Process
Recommendations for standardising and improving selection processes

• Current criteria • Suggested criteria


– Vigilance and concentration – Attention
– Vigilance
– Quick and safe reactions
– Operate hand and foot controls – Train handling aptitude
– Recall and retain job related – Decision making
information
– Learn new information – Trainability
– Understand mechanical principles
– Communicate clearly and – Communication Skills
effectively
– Follow rules and procedures – Rule compliance
– Remain calm – Emotional stability
– Conscientiously work to meet – Conscientiousness
training and job demands
– Proactive and tenacious – Achievement need
– Can spend time alone – Tolerance of working alone

Figure 1: Mapping of current selection criteria to new selection criteria

The following definitions explain the range of concepts to be covered by each of the
new criteria:

Attention: Three different types of attention have been identified which are relevant
to train driving. These are:

• Selective attention (the ability to focus on one thing and avoid distractions).

• Attention switching (the ability to change the focus of attention as and when

necessary).

• Sustained attention (or vigilance, the ability to remain attentive for long periods

of time).

The three types are sufficiently different to be assessed separately. It is


recommended that the “attention” selection criterion is restricted to selective
attention and attention switching since these are more closely related to each other.

Produced by Competence Assurance Solutions Ltd Page 30 of 46


Psychometric Testing – Review of Train Driver Selection Process
Recommendations for standardising and improving selection processes

Vigilance: Problems with vigilance or sustained attention have frequently been


identified as contributory causes of train driving incidents. It has also been shown
that tests of other types of attention do not give particularly good measures of
vigilance. For this reason, vigilance is considered to be sufficiently important to
retain as a separate selection criterion assessed using specific assessment methods.

Train handling aptitude: Train handling aptitude consists of a number of


contributing abilities. A distinction is made here between physical abilities and
decision making abilities which are considered in the next selection criterion. Some
of the physical abilities are addressed in the current selection criteria and remain
relevant. Others are not covered in the existing criteria. It is recommended that
this criterion should cover the following abilities:

• Hand, eye, foot co-ordination.

• Differential perception of both visual and auditory information.

• Estimation of time to collision (including estimation of speed and distance).

Decision making: Decision making is considered an important aspect of driving


generally and is highly relevant to train driving. One aspect of decision making is
captured in the current selection criterion “ability to react quickly and safely” but
reaction is only a part of the decision making demands made on drivers. Other
demands, such as anticipation and interpretation of information, may be more
important. It is recommended, therefore, that the following abilities are included in
the definition of this selection criterion:

• Multi-tasking.

• Prioritisation.

• Planning and anticipation.

• Fault diagnosis.

• Instrument interpretation.

Trainability: There are two existing selection criteria which relate to trainability,
“Ability to recall and retain job related information” and “Ability to learn new
information within appropriate time limits”. Between them, these criteria address all
but one of the main abilities which underpin trainability. These are:

Produced by Competence Assurance Solutions Ltd Page 31 of 46


Psychometric Testing – Review of Train Driver Selection Process
Recommendations for standardising and improving selection processes

• The understanding and assimilation of new information.

• The retention of information for acceptable periods of time.

• The recall of information when required.

• The appropriate application of recalled information.

These are four related stages. You cannot be good at one unless you are good at
the previous stage. Therefore, trainability is better considered as one selection
criterion but where assessment has to address all four aspects.

Communication skills: Communication skills are covered in the existing selection


criteria as the “ability to communicate clearly and effectively verbally and in writing”.
This criterion can remain unchanged although strictly it should be worded “ability to
communicate clearly and effectively in speech and in writing”. Attention may need
to be paid to how this ability is assessed. Most companies have identified a growing
need for drivers to be able to communicate effectively with passengers and to be
able to write acceptable reports. These skills are required in addition to the need to
follow safe communication protocols. These abilities may require greater emphasis
in the selection process and also imply a requirement for a degree of sociability not
currently addressed.

Conscientiousness: Conscientiousness is partially covered in the current selection


criteria through “conscientiously works to meet training course and job demands”.
However, recent research on conscientiousness suggests that there are six distinct
aspects of which five have relevance for train driving. The relevant aspects are:

• Industriousness, which includes the motivation to work hard and commitment to

work.

• Responsibility, which includes dependability and professionalism.

• Desire for order, which includes attention to detail.

• Self-control, which includes ability to check and not make assumptions and

attitude, sensation seeking and behaviour towards other people.

• Virtue, which includes attitude to rule compliance (see next section) and the

desire to work to the best of your ability.

Rule Compliance: As noted, rule compliance is a specific aspect of


conscientiousness. It is included in the current selection criteria as “motivation to

Produced by Competence Assurance Solutions Ltd Page 32 of 46


Psychometric Testing – Review of Train Driver Selection Process
Recommendations for standardising and improving selection processes

follow rules and procedures”. Because rule compliance is so important in safety


critical activities, this criterion should be retained as a separate item for assessment
purposes.

Emotional Stability: There are three main aspects of emotional stability to consider:

• The ability to remain outwardly calm when under stress.

• Physiological reactivity to stress.

• The ability to remain intellectually effective when under stress.

The current selection criterion “ability to remain calm in emergency and/or stressful
situations” addresses the first of these but does not explicitly cover physiological or
intellectual reactivity to stress.

Achievement Need: This criterion is concerned with having the motivation to


develop, improve and succeed at what you do. It includes concepts such as drive
and determination and is partly addressed by the current selection criterion “is
proactive and tenacious”. The desire to progress in your career is also part of this
concept. However, although it is important for some drivers to move into roles such
as driver instructor or driver manager, it is not expected of all drivers. Companies
may vary in the emphasis they wish to place on this criterion and on the extent to
which they want to identify future driver managers / instructors. Since it may not be
possible to identify a common standard for all companies, it may be best to remove
this criterion from the set assessed at the assessment centres.

Tolerance of working alone: The current selection criterion “can spend time alone
and do so effectively” addresses this requirement but it is important to be clear
about what it entails. Train drivers do spend significant amounts of time on their
own on a regular basis. It is not sufficient, therefore, to assess whether they “can
spend time alone” since that wording does not capture the regularity of the activity.

Conversely, most drivers also need to interact with passengers, other train drivers
and / or other railway workers on a regular basis, so it is not desirable for drivers to
be unsociable or to lack social skills. The key issue is that drivers need to be able to
stay alert during periods of time when there is little happening to stimulate or
interest them. Extraverts are more likely to have problems with this than introverts.
As importantly, however, you do not want drivers who, in the absence of
stimulation, try to create interest by taking risks or seeking thrills. The key dangers

Produced by Competence Assurance Solutions Ltd Page 33 of 46


Psychometric Testing – Review of Train Driver Selection Process
Recommendations for standardising and improving selection processes

here, therefore, are extraversion and sensation seeking and not the need for
socialising. So, train drivers must have:

• Sufficient social skills for the social interaction involved in the role.

• The ability to remain alert for significant periods of time without social or other

external stimulation.

• The ability to carry out the same tasks in the same way repeatedly.

Again, companies are likely to vary in the demands made on their drivers. For
example, drivers on commuter lines have more varied day to day work experiences
than those on long haul freight journeys. As with achievement need, it may be
preferable for companies to take responsibility for this selection criterion since there
may not be a common, minimum standard which all companies require.

The following table summarises the recommended new selection criteria and the
sub-criteria which need to be taken into account:

Criteria Sub-criteria
Attention • Sustained attention (vigilance)*.
• Selective attention (including perceptual differentiation).
• Attentional switching.

*Vigilance is sufficiently important to be assessed as a separate


selection criterion.
Train Handling • Hand / eye / foot co-ordination.
Aptitude • Estimating time to collision.
• Estimating speed.
• Estimating distance.
Decision-making • Multi-tasking.
• Prioritisation.
• Planning and anticipation.
• Fault diagnosis.
• Instrument interpretation.
Trainability • Ability to retain, recall and apply job related information.
• Ability to learn new information in appropriate time limits.
• Openness to experience.
• Understanding of mechanical principles^.
• Understanding of electronic principles^.

Produced by Competence Assurance Solutions Ltd Page 34 of 46


Psychometric Testing – Review of Train Driver Selection Process
Recommendations for standardising and improving selection processes

• Ability to use computers and informatics^.

^ These sub-criteria may only be relevant to certain companies.


Communication • Spoken communication.
Skills • Written communication.
• Production skills (i.e. writing and speaking)
• Comprehension skills.
Conscientiousness • Rule compliance*.
• Dependability.
• Attitude to work and people.
• Commitment to work.
• Attention to detail.
• Ability to check and not make assumptions.

*Rule compliance is sufficiently important to be assessed as a


separate selection criterion.
Achievement Need • Desire to succeed and do your best.
• Proactive.
• Tenacious.
• Drive / energy.
Emotional Stability • Calmness under pressure.
• Reactivity to stress.
Tolerance of • Sociability.
working alone • Sensation seeking.
• Need for external stimulation (extraversion).
Table 3: Suggested ‘selection criteria’ and sub-criteria

3.2 The driver selection process


The assessment centres should focus on those selection criteria or those aspects of
selection criteria which require minimum standards to be met by all train drivers.
Selection criteria which only apply to some types of train operator, where it is
acceptable for companies to require different levels of performance or where the
criteria are company specific (e.g. fit with company culture) should be assessed by
individual companies.

The selection criteria which apply to all companies are those concerning safety and
train handling performance. The other criteria deal with personal effectiveness, that
is, the ability to be a good employee. The following table indicates which of the
recommended criteria fall into these categories.

Produced by Competence Assurance Solutions Ltd Page 35 of 46


Psychometric Testing – Review of Train Driver Selection Process
Recommendations for standardising and improving selection processes

Selection Criteria Criterion Group


Attention Safety Performance

Vigilance Safety Performance

Train Handling Safety Performance and Personal Effectiveness


Aptitude

Decision Making Safety Performance and Personal Effectiveness

Emotional Stability Safety Performance and Personal Effectiveness

Rule Compliance Safety Performance and Personal Effectiveness

Trainability Safety Performance and Personal Effectiveness

Conscientiousness Safety Performance (e.g. desire for order, self


control, virtue) and Personal Effectiveness
Communication skills Safety Performance (safety critical communications)
and Personal Effectiveness
Achievement Need Personal Effectiveness

Tolerance of working Personal Effectiveness


alone
Table 4: Selection criterion categories

• Companies should have the flexibility to ask the assessment centres to


undertake only those assessments they want. There is no reason in principle
why companies with qualified test users and trained interviewers cannot
undertake all the assessments themselves although in practice they may not
have access to the necessary equipment or materials. As long as all companies
operate to at least the minimum selection standards on the safety and train
handling performance criteria, there will be no loss in the rigour of the process.

• As noted in 3.1, many of the selection criteria need to be adjusted to take


account of other relevant dimensions of performance or expanded definitions.
Section 3.1 suggests the areas which should be covered by the new selection

Produced by Competence Assurance Solutions Ltd Page 36 of 46


Psychometric Testing – Review of Train Driver Selection Process
Recommendations for standardising and improving selection processes

criteria but they will need to be reworded to capture the wider definitions of what
needs to be assessed and the standards expected.

• Applicants should be allowed more than one re-sit. The validation evidence
suggests that applicants who pass at the second attempt perform as well as
those who pass first time and are more committed to the work. However,
applicants should be allowed no more than three attempts in total since inability
to reach the required standard by the third attempt may indicate weaknesses
which would be difficult to overcome.

• Test scores should remain current for more than one year. Indeed, this
requirement is contradictory at present since the test scores of existing drivers
are included in the data which is transferred when drivers change company.
Either, test scores should remain current for at least five years or should remain
current until there is reason to believe that there has been a significant change in
an individual’s life. The latter is more in keeping with what is known about
changes in test performance over time.

• There is some evidence of practice effects on some of the selection methods.


For that reason, it is sensible to retain the six month gap between applications
since most evidence suggests that memory of previous test sittings has
dissipated by then.

• Best practice requires that more than one assessment, preferably using different
methods, is carried out for each criterion. This is particularly true for the criteria
which address safety and train handling performance. These multiple
assessments need not happen at the assessment centres. Companies should
design their selection processes to ensure that they occur somewhere in the
three selection stages (sifting / shortlisting, assessment centre and final
selection).

Note, this does not mean that a different selection method should be used for
every assessment against every criterion. One test (or other method of
assessment) may be able to give you evidence on two or more of the selection
criteria. Companies should aim to make the total selection process as efficient
as possible. This is also true at the assessment centres. Any changes to the
selection methods used should be based on an evaluation of what are the most
important criteria to assess there.

Produced by Competence Assurance Solutions Ltd Page 37 of 46


Psychometric Testing – Review of Train Driver Selection Process
Recommendations for standardising and improving selection processes

3.3 Content of driver selection methods


Validation exercises carried out as part of this project and in previous studies
suggest that the selection methods used currently at the assessment centres are of
variable effectiveness. In addition, analysis of current and future driver roles and
practice overseas and in other sectors suggest that the selection criteria need to be
updated which also has implications for the selection methods used. The following
recommendations outline the changes required to the selection methods.

• The TRP and the “motivation to follow rules and procedures” section of the CBI
should be retained as they are in the recommended assessment centre process.
Both have been found to predict safety performance as well as aspects of
personal effectiveness.

• The Computerised Group Bourdon should be replaced and the DTG replaced or
upgraded. Although the Group Bourdon is a test of selective attention, there is
reason to doubt it is an effective test of vigilance. Furthermore, the correlation
between the time measure on the Computerised Group Bourdon and both the
TRP and the `good’ scores on the DTG are high (0.55 and 0.76 respectively).
The DTG may need replacing or upgrading to give better coverage of the range
of abilities needed in modern train handling.

• The Mechanical Comprehension Test (MT4.1) measures abilities which are only
required by a minority of companies. This probably explains why the validation
evidence for the test is so inconsistent. It should, therefore, be removed from
the assessment centre process but individual companies may wish to retain it for
use where they have a specific requirement for mechanical aptitude.

• One or more tests should either be identified or developed to give assessments


of the following abilities:
− Improved assessment of vigilance.
− Assessment of decision-making (incorporating multi-tasking, fault
diagnosis and instrument interpretation).
− Differential perception.
− Reactivity to stress.
− Estimating time to collision.

Some of these abilities could be covered in an upgraded DTG.

Produced by Competence Assurance Solutions Ltd Page 38 of 46


Psychometric Testing – Review of Train Driver Selection Process
Recommendations for standardising and improving selection processes

• Ineffective parts of the CBI should be improved to give better coverage of the
recommended selection criteria. This is particularly true for “ability to
communicate clearly and effectively” where it seems not enough attention is
being paid to production skills, and “ability to remain calm in emergency and/or
stressful situations” where the assessment criteria seem to be inappropriate.
Consideration should be given to including role play or simulation exercises to
cover these criteria.

• The introduction of personality testing should be considered as an adjunct to the


CBI (British Psychological Society guidance is that personality tests should never
be used on their own in making selection decisions).
− Personality testing could provide useful confirmatory evidence for a number
of safety performance and personal effectiveness criteria, particularly “rule
compliance”, “conscientiousness” and “tolerance of working alone”.
− Personality testing need not be carried out at an assessment centre but note
that it requires a higher level of test user qualification which may make it
impractical for companies to do such testing themselves.

• More use should be made of academic achievements in selecting drivers.


However, unlike many other countries, it is not recommended that minimum
academic requirements are set. In Great Britain this would probably be counter-
productive since there is no tradition of setting such requirements and it might
result in adverse impact on older applicants. Instead, academic attainments
should be combined with other types of selection evidence for making final
decisions, particularly where performance on other measures is borderline.

Figure 2 below illustrates the envisioned changes to the content of the assessment
centres using the following colour keys:

• The pink boxes show the current selection methods.

• The blue boxes indicate recommended methods for the revised assessment

centres.

• The black arrows indicate the selection criteria, current at the top, recommended

at the bottom, for which the various assessment methods seek to provide

evidence.

Produced by Competence Assurance Solutions Ltd Page 39 of 46


Psychometric Testing – Review of Train Driver Selection Process
Recommendations for standardising and improving selection processes

• The green arrows indicate where changes or upgrades to selection methods are

recommended.

• The pink arrow indicates where the existing method can be carried over

unchanged into the assessment centres.

Produced by Competence Assurance Solutions Ltd Page 40 of 46


Psychometric Testing – Review of Train Driver Selection Process
Recommendations for standardising and improving selection processes

Communication
Follow rules Hand and foot Recall and retain
Remain calm control information
Conscientious React quickly Learn new
Proactive / tenacious Mechanical Vigilance and
and safety Comprehension information Concentration
Spend time alone

CBI DTG MT TRP Group


4.1 Bourdon

CBI + Personality
Perception Decision
Role TRP Vigilance
Test ? Test (DTG Making
Play? Test
upgrade?) test

Train Handling
Emotional stability Multi - tasking
Aptitude:
Communication Instrumentation Trainability Vigilance
Hand and foot control
Rule compliance interpretation Rule compliance
Concentration
Conscientious Planning / Communication
TTC estimation
Achievement need anticipation
Perceptual
Tolerance of working alone Train handling
differentiation
aptitude

Figure 2: Route map of recommended assessment centre content changes

3.4 The use of psychometric tests in driver management


Psychometric testing can be a useful addition to driver management practices but
only if they are used appropriately. Psychometric tests are used to support four
aspects of driver management:

• Training and development.

• Post trauma counselling.

• Pre-incident counselling.

• Post incident investigation.

Current use of psychometric tests in the first two of these seems sensible and no
changes are recommended in these areas.

Produced by Competence Assurance Solutions Ltd Page 41 of 46


Psychometric Testing – Review of Train Driver Selection Process
Recommendations for standardising and improving selection processes

The use of tests in post incident investigation and pre-incident counselling (special
monitoring) is more problematic. It is not clear that re-testing drivers on the tests
used at the assessment centres is always, or even usually, an appropriate strategy.

This conclusion is strengthened by the finding that three of the psychometric tests
used at the assessment centres have, at best, only modest validity and that they
only address a subset of the abilities required of a good driver. Furthermore, post
incident testing should focus on the safety and train handling aspects of the role.
Tests and other assessments of personal effectiveness will often be irrelevant.
Nonetheless, tests are widely used in other sectors for similar purposes. Indeed,
they are one of the main tools used by clinical psychologists.

Therefore, tests can be used effectively in these areas of driver management,


particularly if used in the following ways:

• Following an incident to help confirm or disconfirm explanations of driver

behaviour.

• Before an incident occurs to identify whether performance is deteriorating in

drivers who have been identified as “at risk”.

Table 5 below sets out a recommended process for each of these:

When you may Why you may How you might use tests
want to use a want to test
test

Produced by Competence Assurance Solutions Ltd Page 42 of 46


Psychometric Testing – Review of Train Driver Selection Process
Recommendations for standardising and improving selection processes

When you may Why you may How you might use tests
want to use a want to test
test
Following an To help • Psychometric tests should not be the first
incident. establish choice for assessment.
explanations of • Observation, interviews and assessment on
behaviours. simulators should be used first to establish
likely causes which explain the driver’s
actions.
• Where these possible explanations can be
investigated using tests, identify suitable tests
which may help you to understand better why
a driver behaved in a particular way.
• The purpose of testing will be to support or
challenge the possible explanations identified
earlier.
• These tests will often not be the ones used for
recruitment since they cover only a subset of
the abilities required by a good driver e.g. you
may want to test for a suspected memory
problem using a specific test.
• Such testing often requires specialist
knowledge.
• Remember that re-testing, post incident is
itself a stressful event that may result in a
driver performing less well than they would in
normal circumstances.

Produced by Competence Assurance Solutions Ltd Page 43 of 46


Psychometric Testing – Review of Train Driver Selection Process
Recommendations for standardising and improving selection processes

When you may Why you may How you might use tests
want to use a want to test
test
If you have To find out if • If you suspect that a driver’s performance has
reasons to believe they are deteriorated but you are unsure why.
a driver’s unlikely to be • You will need to assess them to find out if this
performance may able to continue is really the case.
be introducing doing their job • Consider using the following sequence of
risk. properly. assessment methods to help you do this:
Interview the driver.
Observation.
Assessment on a simulator.
Re-test, using the selection tests – you
can compare these results with the
original ones to see what, if any,
changes there are in performance.
• Based on these outcomes you may wish to use
other psychometric tests to look at particular
areas of concern or to further explore areas
where performance may be deteriorating.
Table 5: Use of psychometric tests for driver management

Produced by Competence Assurance Solutions Ltd Page 44 of 46


Psychometric Testing – Review of Train Driver Selection Process
Recommendations for standardising and improving selection processes

4 Next steps

The full implementation of all the recommendations outlined above will take some
time. Furthermore, although stakeholders who attended the industry briefing held
on 20 October 2005 were broadly in agreement with the principles underpinning the
recommended changes, agreement has not been reached with regard to the detailed
implementation of the recommendations. Final decisions about which tests and
methods to upgrade, which to introduce and which to remove, will considerably
affect the time to implement revised processes. For example, developing new tests
or new versions of tests will take considerably longer than buying commercially
available tests. The recommendation of the stakeholders at the industry briefing
was that a steering committee should be set up, facilitated by RSSB, to make these
detailed decisions on content and process.

The questions which the steering committee will need to answer are:

• Does the industry needs a core assessment process for all train driver

recruitment?

• If a core recruitment process is needed, which selection criteria should be

covered by that process?

• Given that it is impossible to test for all the criteria and sub-criteria, which might

be included in the core process at the assessment centre i.e. which criteria must

be covered?

• Which assessment methods should be used to assess the selection criteria

included in the core process?

• Is greater consistency required in the use of psychometric tests in driver

management?

• How might greater consistency be achieved in test use in driver management?

In making its decisions, the committee will need to take into account regulations
which may be introduced as part of the implementation of the European directive on
rail system interoperability and changes to the train driver group standard (GO/RT
3251) which may result from RSSB’s ongoing review of standards.

Produced by Competence Assurance Solutions Ltd Page 45 of 46


Psychometric Testing – Review of Train Driver Selection Process
Recommendations for standardising and improving selection processes

In the short term, however, there are a number of steps which can be taken to
improve the total recruitment and selection process:

1. Continue to operate the current assessment centre process until such time as a
revised process has been developed. There is sufficient evidence for the validity
and utility of parts of the process to suggest that it delivers value.

2. Companies can begin selecting against the new selection criteria as soon as they
are agreed.

3. To make this happen, the extent of the coverage of these selection criteria by the
current assessment centre process will need to be agreed.

4. Following this, companies will need to review their sifting and final selection
processes to ensure completeness of the coverage of the new selection criteria.
This may involve revisions or additions to their selection methods.

5. Companies need to make sure that all participants in the various stages of the
total recruitment process are properly trained.

6. Implement the recommend improvements in database management, namely


consistency in test score recording and trapping of data entry errors.

There are also a number of steps that can be taken in the short term concerning the
use of tests in driver management:

1. Use tests where appropriate but not as the first choice of assessment.

2. Outsource testing for this purpose if the company does not have appropriate
expertise in-house.

3. Always use test data in combination with other sources of information.

4. Train the recipients of psychometric test information in-house so that:

− Managers understand the purpose of testing e.g. for testing hypotheses


about human factor influences on incidents.
− Managers understand what tests scores can and cannot tell them about
skills, abilities and behaviours.

Produced by Competence Assurance Solutions Ltd Page 46 of 46


Rail Safety and Standards Board Evergreen House 160 Euston Road London NW1 2DX
Reception Telephone +44 (0)20 7904 7777 Facsimile +44 (0)20 7904 7791
www.rssb.co.uk

Rail Safety & Standards Board Registered Office: Evergreen House 160 Euston Road London NW1 2DX. Registered in England and Wales No. 04655675.

Rail Safety & Standards Board is a not-for-profit company limited by guarantee.

You might also like