ELICITING DISPERSED EXPERT KNOWLEDGE Via DELPHI

Ian Miles, MIoIR Manchester, 2007

(this note was prepared in connection with a research proposal – IM)

Why Delphi?

Delphi studies involve surveying expert opinion. What makes Delphi unusual is
that the survey is designed to be reiterates, feeding information back to its
respondents, in contrast to most polls of opinion. The survey is circulated, to the
same set of respondents, at least twice (in the original studies, more iterations
were common). The respondents in later rounds are not only asked to answer
the same set of questions again, but also receive feedback on the structure of
responses at previous rounds. They should be able to look at the pattern of
results – how many people replied in each possible way to a given question.
Ideally, they should additionally receive information on why judgements, and
especially extreme judgements, were made.

The rationales for Delphi are that:

• Experts are liable to make better-informed judgements when they can
compare their own views with those of others.
• An anonymous survey should be freer of the influence of loud voices and
reluctance to disagree with authority than is a conventional face-to-face
meeting.
• The distribution of replies should provide all respondents with access to
special information that only a few possess, especially when these underpin
judgements that diverge from the average.
• A Delphi survey typically allows for much greater outreach than a
conventional meeting, allowing for inputs from a huge range of experts.
• Well-formulated Delphi statements allow for more precise comparison of
views and assessment of their relevance for forecasts than do the vaguer,
more qualitative statements that typically emerge from open-ended
discussion and workshops.

Delphi studies can be organised in numerous different ways, with practically any
topic being studied. The core feature of a Delphi, as noted, is that the survey
questionnaire is reiterated. That is, after a first “round” of questioning, it is
returned with feedback to the original respondents, who are then able to revise
their answers in the light of this information about the distribution of responses in
the first round. The process may be further reiterated, with second and even
third round results being fed back to respondents.

1
Often the formulation of topic statements for a Delphi study may be constructed
from an earlier sounding of respondents’ views as to the important topics to
investigate. In an ideal world, Delphis will do more than just feed back
quantitative information to respondents. They will allow for some open-ended
input of material, which may also be fed back so as to enlighten respondents
about reasons for specific responses, other categories of response that might be
included, and so on. However, inn practice many Delphis give poor feedback of
a qualitative nature, because of the costs in labour and time this involves.

Delphi studies have traditionally been administered by post, pen and paper. It is
now practicable to use online surveying. PREST/MIoIR has used this approach
in cross-national projects such as EUFORIA and FISTERA. Occasionally, the
survey is administered in a workshop of some form; the online methods can also
be used in this way, with rapid feedback permitting a second round to be
undertaken on the spot. (In FISTERA, the online survey was actually launched in
a workshop. This test bed led to detailed revision of several survey questions.
The participants’ first round responses were presented to them for comment and
to inform the workshop process.)

In what follows, we present a number of illustrative examples of the sorts of

topics that might be addressed in this project, and the ways in which they might
be handled. The commentary will explicate what some of the design choices are,
and why specific options might be chosen in relation to particular objectives for
the study. If a Delphi is to be undertaken, it will be helpful to build support for it,
for example by illustrating what the technique has been able to bring to earlier
Foresight exercises. Designers must be prepared to deal with several common
misapprehensions about Delphi, too – that it is always seeking consensus, that it
is always a matter of forecasting, that one is seeking the same kind of statistical
representativeness as in an opinion poll.

What would be a common feature to any Delphi is the need to be assiduous in

locating and mobilising relevant expertise. The most impressive quantitative
results, displayed in the most striking charts and graphs, are worth next to
nothing if the opinions and information used do not come from the appropriate
sources of knowledge. It is necessary to define who the experts are that need to
be sampled – and especially when they are fairly few in number, to motivate
them to participate in the survey. Often this is best accomplished by personal
contact from an esteemed source. Locating the appropriate experts may involve
different ways of “snowballing” a list, for instance by contacting a few known
authorities and relevant associations and communities, and asking for their
recommendations concerning whom to contact.

We now present a set of options for Delphi studies.

2
Delphi Example 1 - “Impacts of development” - FISTERA

The FISTERA study included a large-scale online Delphi survey, in which over
500 experts contributed their views as to the future development of IST in the
EU. The study was particularly concerned to examine how far specific
applications of IST could contribute to a number of European objectives (inspired
by the Lisbon Agenda), and to elicit views as to the major challenges that will be
confronted, areas where research is needed, and how far European capabilities
exist to be excellent in these topics.

The EU goals were specified for the survey as: job creation; wealth creation;
competitiveness; social cohesion; social inclusion; environmental quality. (Brief
descriptions of each were available for participants in the survey.) Drawing on a
literature review conducted for FISTERA, the survey focused on twelve IST
application areas: 1) Social / family relationships; 2) Cultural diversity; 3)
Transport; 4) Ageing; 5) Health; 6) Education and learning; 7) Social welfare /
public services; 8) Leisure and recreation; 9) Security, 10) Government; 11)
Management; 12) Work organisation. (Again, brief documentation on each topic
area was prepared and placed online.)

Participants were asked to select 5, from the list of 12, IST Application Areas that
they considered to be most likely ones to contribute to the achievement of the
each 6 specific EU objectives in turn. It also asked participants to indicate which
sort of applications within each area would be more likely to improve quality of
life and wealth creation. The result was a set of data which could be presented
in terms of which applications are seen as being most beneficial ()and which
most beneficial to subclasses of goals). Correlation analysis indicated that there
were two broad clusters of goals in terms of applications impacting on one item in
a cluster tended to impact on the others in the same way. “Social inclusion” and
“Social cohesion” were extremely highly correlated; as were “Job Creation” and
“Wealth Creation” which were also strongly related to “Competitiveness” (which
in turn is fairly closely related to “Environmental Quality”).

The Delphi survey asked respondents:Which of a series of impediments

(developed on the basis of earlier discussions and analyses) were thought to be
most important obstacles to development of IST applications.
• What were the top actions contributing to making Information Society
Technologies more effective and socially beneficial.
• Which major challenges in IST R&D would need to address.
• What EU R&D capabilities are in IST in comparison to the World, and how
prepared the key EU research communities in the public and private
sectors are, in the the 12 application areas.

3
 • Which stakeholders are most likely to play a key role in improving
applications of IST in each area.
Full results are presented on the FISTERA website.

Delphi Example 2 - “Impacts of development” - EUFORIA

The EUFORIA Delphi was generated in large part on the basis of inputs from
workshops and other soundings undertaken by the Finnish, German and Greek
teams in EUFORIA.
This topics are presented in terms of future developments (in this case of
Knowledge Society, rather than of technologies –examples are: “The balance
between immigration and emigration in the EU15 causes the percentage of non-
EU15 immigrant workers to more than double by 2015” and “Widespread use of
ICT in e-governance enhances transparency in the procedures concerning the
relationship between the citizen and the state in my country”. For each topic
statement, the respondents were asked about the likelihood or extent of the
development actually applying in the year 2015. This contrasts with the most
common form of Delphi, in which respondents are asked to specify at what
period (if ever) the topic would be realised- such Delphis are used to get a sort of
timeline of prospective future developments of different technologies and
applications. But if we are interested in the “impacts” of developments by a
particular time point, alternative approaches may be more useful.
Respondents were thus asked to specify whether each topic statement:
• Underestimates the situation in 2015;
• Is about right for 2015
• Overestimates the situation in 2015;
• Developments will not follow this path;
(the option was also provided of answering “I don’t know / Too uncertain”, with
the explanation that this means that the respondent has no opinion to express).
For the proposed study, the date of 2030 would be used, of course.

Respondents were also asked to rate the influence of the development

represented by the topic statement, assuming that it did materialise, on the
following set of factors (that corresponded to Lisbon Objectives and other EU
goals):
• Social cohesion;
• Social exclusion or divides;
• Sustainability/environmental quality;

4
• Employer-employee relations;
• Economic growth/wealth creation;
• Entrepreneurship and innovativeness;
• Employee exercise of autonomy and responsibility at work;
• Work-life balance;
• Job creation.

Advantages and Disadvantages of this approach

This approach provides an assessment of a “most likely” future. It does not
necessarily tell us directly about the changes required to get to a more desirable
future. (However, by illustrating which developments were felt to be most liable
to be achieved by 2030, and which are felt unlikely to be so, we have a rapid
insight into where policy intervention might be required.) The impact factors are
based on the sponsor interests for the specific study. The present project might
be concerned with social and economic indicators such as:
• The economic performance of small and medium-sized enterprises;
• Requirements for higher levels of skills at work;
• Carbon emissions.

The approach provides an assessment of a most likely future. It does not

necessarily tell us about the changes required to get to a more desirable future,
though it can be used to identify areas where action is necessary.

Delphi Example 3 - A Policy Delphi

Policy Delphis feature much more emphasis on how particular outcomes are
reached. (In contrast, Decision Delphis are much more about prioritising
particular immediate actions.1) One use of a Policy Delphi would be as part of an
attempt to identify critical objectives for Europe – for example, there could be a
number of alternative visions of the future, and respondents asked to select
among and/or combine these. An approach that requires rather more consensus
about a desirable future to be assumed would begin with a brief “vision
statement” of what the goals are for Europe's future.

This sort of Delphi begins with a brief statement or set of statements of

what the goals are for Europe's future. These are very important
formulations. Each needs to be concise yet to evoke a good deal of
shared understanding. If a single integrated vision is being examined, then
the goals should be ones that are widely viewed as feasible and desirable,

5
 and they should not be too numerous. If part of the exercise is to
prioritise or assess the degree of support that a number of (sub)goals
receive, then they need to be expressed in broadly comparable terms.

The key point of the method is that it focuses on how the goals are achieved.
One way in which this can be done is to ask about how important various actions,
strategies, or circumstances would be for this goal to be achieved, or how far
each topic would contribute to the goal. Examples:
• How important would it be for this technology development to be realised in
order for Europe to reach the condition described in 2030? (5 point scale from
very important through to not at all important)?
• If this came to be the case, how much of a contribution would it be making to
Europe's reaching the goal described for 2030? (answers could be in terms
of a 5 point scale from very important positive influence through to very
important negative influence)?
• What would be the influence of this state of affairs on Europe's ability to move
towards the goal described for 2030? (5 point scale from very important
positive influence through to very important negative influence)

It could be very useful to ask a second question asking about the likelihood of
development of the topic statement itself, e.g.:
• How likely do you think it is that this technology development /application will
come to characterise Europe in the next 12 years? (answers could be in
terms of a 5 point scale from very likely through to very unlikely) and
• How uncertain do you feel about this state of affairs being reached in the
next 12 years? (very certain that it will happen to very uncertain that it will
happen or a formulation along the lines of that used in EUFORIA )

A Policy Delphi could examine impacts and use ides of the sort outlined in the
previously discussed Delphi models. It would be necessary to ask about whether
the impact was positive or negative, and judgements could be elicited on 5-point
scales.

An alternative model – moving closer to the Decision Delphi model - would focus
less on where the impacts lie, and more on what needs to be done by whom.
For example, it would be feasible to ask "how much change would be required in
the following areas of policy/ or the following policy areas and agencies" and ask
how much change is required from agencies responsible for:
• Science and R&D policy,
• education policy and institutions,
• social policy (e.g. welfare, pensions, health),

6
• labour market and industrial relations policy,
• enterprise and industry policy,
• infrastructure policy (transport, communications, etc.).

It would be important here to include some opportunity for specifying the extent
to which policy integration across areas is required. Another formulation could be
along the lines of “how far would policymakers responsible for the following
areas be required to take these developments into account in their
policymaking?”. Alternatively, the formulation could be specified in terms if
different actors rather than specific policy areas, asking for instance "how much
change would be required from the following actors or agencies?", for instance:
• overseas investors,
• large firms,
• small and medium-sized enterprises,
• Universities,
• schools,
• government ministries dealing with social affairs
• etc…

And there are other possibilities, for example addressing some very specific type
of action such as shifts in private investment decisions, output of higher skilled
workers, geographical mobility of labour, and so on.
This approach provides a more direct assessment of views concerning how
much change is required for Eiuropean success, and where this change needs
to be. However, there is a danger that the statements will be by and large
specified in “motherhood” term, and for the Delphi to do more than pain a rosy
picture, or to evoke unrealistic aspirations, it will be necessary to work toward
more specified topic statements than those outlined above.

Delphi Example 4 - Multiple Scenarios Delphi

This sort of Delphi asks about what would happen under various contingencies –
for instance, it might ask what a particular feature would look like in Scenario A,
Scenario B, and so on. This sort of activity is most often pursued in a workshop
format, where participants can have had opportunity to immerse themselves in
different scenarios (and sometime different sets of participants will be specifying
each of the scenarios). But there have been some survey studies that revolve
around alternative scenarios, and in principle the approach can be valuable in
making the issue of alternative futures more concrete. Again, several submodels
are possible here, and we just focus on a few of these.

7
 (a) One approach would involve analysis of just what would happen under
only two contingencies. For each topic statement, two judgements have
to be made - what is most likely or alternatively what is the current path of
development, and what would be the most likely state of affairs were
Europe to reach a more desirable future by 2030. Thus the Delphi could
ask how far Europe could succeed in promoting a particular technology
development “if current trends or business as usual approaches persist”
and “if feasible shifts in policy and funding were implemented over the
next few years”. The precise formulation of the goals and the importance
ratings will be rather similar to those outlined in above. The above
approach essentially contrasts an extrapolative future with a more
normative one, leaving it up to respondents to determine the content of
these two poles. An alternative approach is to specify two or more
(perhaps up to four) scenarios that represent alternative paths of
development.

This approach has some of the virtues of the previous model, and additionally
opens minds concerning alternative futures – there is not one single unalterable
trajectory into the future. Compared to other approaches, it requires specific
design work on the scenarios element of the activity. Its main problem is size
and complexity. Given the focus on alternative scenarios, the number of major
questions asked concerning each topic statement is more than in the previous
Delphi models. This implies that we should minimise the requirements for
additional judgements. Some very simple version of impact judgements used in
the previous models (perhaps especially model 3) could be employed, however,
examining views of changes in policies and strategies, or their outputs,
respectively.

Delphi Approaches – Conclusion

There are evidently various sorts of Delphi that could be designed and
implemented in the context of this study.1 Different types of Delphi address
different types of policy need. It is not necessary to slavishly follow the most
common Delphi models, and indeed these may not be most appropriate for

1
Additionally, we should not forget that there are other tools for dealing with expert opinion, such
as cross-impact analysis, which could be employed here. Delphi tends to be used very widely
because it is relatively simple to administer, complete and analyse – which cannot be said of
some of these other methods. Some methods elicit a great deal of qualitative input from open-
ended questionnaires, and this takes a good deal of effort to analyse. Cross-impact analysis
explores relation between parameters, rather like a simplified computer model. But it is limited to
few variables, and even then is very demanding of respondents. Another factor in the wide use of
Delphi has probably been the availability of earlier models in TF that can be borrowed from,
making the design task – the most challenging part of the exercise – easier (though not
necessarily sufficiently well-tailored to the local requirements).

8
present purposes. There is much scope for experimenting with some of the less
common approaches to Delphi.

It looks most likely that some modification of the FISTERA or EUFORIA

approaches will be most appropriate for the present study. An early task will be
to assess these options and prepare a good case for adopting one or other
design. The following tasks will, of course, be to populate the delphi with a
manageable number of clear and relevant topic and impact statements, and to
define what additional information will need to be elicited. At the same time, we
will design an effective framework for online presentation and data collection, and
prepare out letters of invitation and lists of participants to circulate.

Refs

H. A. Linstone and M. Turoff, (eds) 2002, The Delphi Method: Techniques and
Applications (originally published 1975), web version available at:
http://www.is.njit.edu/pubs/delphibook/

D Loveridge and I Miles, with Michael Keenan, Rafael Popper & Duncan Thomas
2004, European Knowledge Society Foresight: The EUFORIA Project
Synthesis Report, European Foundation, Dublin available at:
http://www.eurofound.eu.int/publications/files/EF0404EN.pdf

R Popper and I Miles, 2005, The FISTERA Delphi Future Challenges,

Applications and Priorities for Socially Beneficial Information Society
Technologies, Report prepared for FISTERA project, available at
http://les.man.ac.uk/prest/FISTERA/delphi_results.htm and to be published on
project website

9
1
Let us illustrate this with an example suggested by Denis Loveridge. One of the possible topic
statements discussed in this section is: “(4) The skills generated by education and training institutions are
increasingly out of alignment with those required for an innovative twenty-first century economy.” In a
Decision Delphi this would take more of a form such as: “Europe’s education and training institutions should
be required to create skills and new competences in fields vital to Europe’s interests.”