See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/248497308

Delphi questionnaires versus individual and group interviews A

comparison case

Article  in  Technological Forecasting and Social Change · May 1990

DOI: 10.1016/0040-1625(90)90029-U

CITATIONS READS

75 1,362

1 author:

Jan A.G.M. Van Dijk

University of Twente
127 PUBLICATIONS   10,246 CITATIONS   

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

21st Century Digital skills, key to 21st century labor View project

democracy in a network society View project

All content following this page was uploaded by Jan A.G.M. Van Dijk on 29 August 2018.

The user has requested enhancement of the downloaded file.

TECHNOLOGICAL FORECASTING AND SOCIAL CHANGE 37, 293-304 (1990)

Delphi Questionnaires Versus Individual and

Group Interviews
A Comparison Case

JAN A. G. M. VAN DIJK

ABSTRACT

The article reports on a methodological investigation for different types of the communication device
represented by the Delphi method. Traditionally mailed or computer-mediated questionnaires are used. The
Delphi method was invented to overcome undesirable effects of group interaction while retaining the positive
effects of interactive group judgments. This article supplies evidence for the opposite: the superiority of face-
to-face techniques (individual and group interviews) as compared to mailed questionnaires in a three-round
Policy Delphi using half-structured questionnaires. Moreover, the case study proves that they might produce
different results. In a field- or quasi-experimental design the three techniques were compared for their effects.
The performance of 100 panel members on a large number of quality standards in the Delphi method is described.
A list of pros and cons of the three techniques in a (Policy) Delphi serves as a summary.

Introduction
This article reports on a (Policy) Delphi study that was a combination of a substantive
and a methodological investigation. Only a part of the results of the latter type of in-
vestigation is described here. Other results have been published in earlier articles [ 1, 21.
The present article deals with variations in the communication device represented by the
Delphi method.
In almost every definition of the Delphi method it is assumed that mailed, or perhaps
computer-mediated, questionnaires are the main instruments of communication. This is
supposed to be necessary for anonymity and in view of the physical dispersion of the
participating panelists. Another reason is the purpose of most classical Delphi studies to
collect exact quantifiable estimates (of future developments) by a panel of experts. A
written communication device with ample room for a feedback of facts and figures and
with closed questions therefore seems to be necessary. Some kind of face-to-face group
interaction (a conference, workshop, or other brainstorming session like the Nominal
Group Technique) is considered to be the only alternative [3-51. To most Delphi inves-
tigators it is a worse alternative, as there is a considerable body of research indicating
the inferiority of interactive groups in some settings, particularly if the accuracy of answers
(estimates, forecasts) is an important aim [6, 71.

JAN A. G. M. VAN DIJK is at the Faculty of Social Sciences, at the State University, Utrecht, The
Netherlands.
Address correspondence to Dr. Jan A. G. M. Van Dijk, State University at Utrecht, Faculty of Social
Sciences P.O. Box 80.140, Utrecht, The Netherlands.

Q 1990 by Elsevier Science Publishing Co., Inc. 0040-1625/90/$3.50

294 J.A.G.M. VAN DIJK

These definitions and preferences no longer meet the present situation. The classical
Delphi method has evolved and diverged in all sorts of types and applications. It has
become very difficult to construct a single definition out of them. Actually the Delphi
method is only a set of familiar questioning techniques with the common features of
some kind of interactive communication and intermediate feedback of results. The so-
called Policy Delphi [8] and the related Decision Delphi [9] have become as important
as the Conventional Delphi [lo]. All these types can be combined with group methods
of idea building, as has been shown, for instance, by Nelms and Porter [ 111. Nelms and
Porter wonder whether the classical (mailed/computer questionnaire) form of commu-
nication facilitated by the Delphi technique is not “too restricted for many problem
situations. The requirement of written feedback, editing, and distributing places a high
cost on the communication of ideas. The distance and anonymity associated with Delphi
prevent a meaningful ‘discussion’ even when such interaction would be advantageous,”
[ 11, pp. 47-481. They conclude, and they try to prove in their Delphi variant, that “there
is considerable interest in face-to-face interaction in opinion capture even though simple
interacting group decision processes have proven consistently problematic in research
over half a century,” [ 11, p. 481. Was the prime motivation of Dalkey and Helmer to
develop the Delphi technique as “an effort to overcome the undesirable effects of group
interaction while retaining the positive aspects of interacting group judgments” [ 11, p.
461 all wrong then?
The point is that in many types of Delphi method recently designed, particularly
Policy Delphi ones, qualitative opinion capture emerges. “A Policy Delphi deals largely
with statements, arguments, comments, and discussion,” according to Turoff [8, p. 891.
Now, opinions are very susceptible to question wording and questioning conditions. (This
proposition is backed by contextual approaches and theories of the research interview,
foremost the survey interview; see [ 12, 131). So there seems to be reason enough to ask
whether the answers on questions posed in mailed questionnaires differ from those given
in the conditions of individual and group interviews. Or, more generally, is the perfor-
mance of mailed questionnaires and reports as the traditional communication device in
Delphi studies really superior to individual and group interviews working with mailed
written reports (feedback) in a Policy Delphi based on statements, arguments, comments,
and discussion? What if the panel members are not professional experts used to writing
and reading difficult texts, but laymen or “lobbyists” discussing a popular or familiar
issue, as might be the case in a Policy Delphi?

Research Questions, Design, and Method

Before we proceed with a detailed exposition of the research questions, we have to
sketch the purpose and context of the investigation. The investigation was conducted at
the ABN, the largest commercial bank in Holland, among a hundred members of the
trade union of commercial workers, Dienstenbond PNV. The problematic subject matter
was the introduction of a large-scale automation of bank work by an ail-embracing
computer network called the Open Bank Project (OBP). The union members did not
know what was exactly at stake when the network was to be installed, nor did they know
what attitude they were to assume toward the project. Therefore the following research
problem was defined for the substantive investigation: what is the best strategy (goals
and means/demands) for the union with regard to the OBP on the subjects of: (I) infor-
mation, (2) quantity of work, (3) quality of work, (4) training, and (5) consultation of
employees by managers? The investigator and the representatives of the trade union were
convinced that a special type of Policy Delphi, adjusted to the problem and the participating
population, could serve as a valid research instrument.
DELPHI VERSUS INTERVIEWS 295

Most panel members were not professional experts but people engaged with their
own future work. In fact the membership was very heterogeneous socially. It contained:

l employees with low, medium, and high previous education

0 employees with low, medium, and high job level
0 female and male employees
0 non-trained and trained union members.

A systematic and explicit panel membership selection was stressed (see [l]).
Three questionnaires (for three rounds) were devised according to the following
purposes and characteristics:

0 Their character was much more qualitative than quantitative: forced choice ques-
tions followed by a request for arguments in the explanation of choices and a
feedback mainly describing these explanations with its frequencies as the only
statistics presented;
l They contained large blocks of information (and feedback): not much knowledge
presupposed;
l They were written with great care for readability and understandability;
0 They systematically promoted “adversary processes” [14, p. 541 by means of
questions with a balanced dual and polar structure (e.g., “for” and “against,”
without a middle category); these questions were designed to stimulate thinking,
discussion, and decision-making by stressing choice (see [8, 91);
l They were used under three conditions: a mailed questionnaire, an individual
interview, and a group interview.

The following research questions were posed:

I. GENERAL
What is the best communication technique (of questioning and reporting) in a Policy
Delphi among a panel of nonexperts using forced choice questions followed by expla-
nations of choices for opinion capture:

A. mailed questionnaires (with mailed reports)

B. individual interviews (with mailed reports)
C. group interviews (with mailed reports)?

II. SPECIFIC

A. Subjective Pegormance
1. Which technique is valued highest and which one lowest by the panel members
themselves, and for what reasons?
2. In which technique do panel members perform best on the high levels of appeal
to cognition, effort, involvement and self-confidence required [see I] in the
participation in Delphi studies?

B. Objective Performance
1. Are significant differences of answers recorded among the techniques? If so, in
what direction do these differences tend to go?
2. Are significant differences of opinion change between rounds recorded?
296 J.A.G.M. VAN DIJK

TABLE 1
Number of Interviews and Mailed Ouestionnnaires ComDleted/Returned in Three Delnhi Rounds

Individual Interview Group Interview Mailed Questionnaire Total

Round 1 26 27 (5 meetings) 35 88
Round 2 26 27 (5 meetings) 30 83
Round 3 30 31 (6 meetings) 19 80

Total 82 85 (16) 84 251

3. Are significant differences of dropout in participation recorded?

4. Are significant differences of no-opinion and no-answer categories recorded?
5. Are significant differences of the quality of answers recorded?
C. Other Performance
1. What is the best technique considering the time of execution and the financial
budget required?
2. What is the best technique considering the aim of anonymity of respondents in
Delphi studies?
III. CONCLUSION
Are significant differences recorded among the social categories of panel members
on all pertinent previous questions?

The answers to these questions will provide the opportunity to construct a summary
list of pros and cons of the three techniques that can be of help to the practice of Policy
Delphi research, and perhaps give clues to other types of Delphi and even opinion capture
techniques generally.
To be able to make these comparisons each panel member was subsequently ques-
tioned in three different ways, that is one way in each of three Delphi rounds (see Table
1). An attempt was made to assign them completely at random to these three conditions.
In that case it would have been possible to answer another research question we had in
mind: what is the best sequence of the three techniques mentioned in a three-round Policy
Delphi? However, we did not sufficiently succeed in that, owing to temporary dropout
of panel members (the nonresponse appearing with the mailed-questionnaire technique)
and practical difficulties (for instance, when members missed an appointed group interview
they had to be questioned in another way). The result is that two of the six possible
sequences of techniques contained too few subjects for statistical comparisons. Still, a
field- or quasi-experimental design for a single case study was left. Confined to this case
it allowed the comparison of individuals and of techniques within rounds and also rounds
and techniques as a whole, but not individuals and techniques across rounds.
Although the results of this study can not be statistically generalized to other situ-
ations, they can reveal characteristics of the techniques compared that are likely to work
in comparable projects and can be subjected to further empirical test. The study can be
called field- or quasi-experimental as there was no (successful) randomization and no
complete experimental control of groups and conditions in the ever-changing settings of
this applied social research project. Still, a pointed attempt was made to standardize all
controllable factors: the homogeneous panel (members of one union, motivated to par-
ticipate), the time period (two to three months of data collection in every round), the
place of interview (the office of the participant), the reports of intermediary results, the
interviewer (the investigator himself), the interview style (rather formal and task-oriented),
the recording apparatus (very small), the two assistants transcribing full interview reports,
DELPHI VERSUS INTERVIEWS 291

and the questions. Undoubtedly, the last factor was the most important one to standard.
The following steps were taken:

0 A question wording was chosen that would be suitable for all three techniques;
0 The request for a written explanation of an answer (category) in a mailed ques-
tionnaire was substituted by one undirective probe in an interview; the answers
following any further probe, especially needed in group interviews to stimulate
every individual’s expression, were analyzed separately;
0 The information and feedback blocks of the questionnaires were mailed to the
interviewees without the questions; they had to be read before the interview;
during the interview these blocks had to be summarized briefly before a question
was posed to allow an ongoing communication; these summaries contained exactly
the same selection of lines of the questionnaire/report in every interview;
l The group interviews were conducted in a way that made it clear to all participants
that their individual opinion was as important as the group opinion.

For a further improvement of the validity and reliability of the comparisons a large
number of methods was used (multi-method or triangulation). Along the terminology of
subjective and objective performance used before they can be arranged in an order from
more subjective to more objective methods:

1. An evaluation questionnaire containing, among others, questions about the pref-

erence of questioning techniques was distributed after the termination of the last
Delphi round.
2. Self-rating scales were completed by the participants after every round.
3. A secondary analysis of answers (answer categories) was made, first for differ-
ences between them, second for opinion change, and finally for no-opinion and
no-answer categories.
4. A dropout analysis of panel membership was made.
5. A content analysis of full mailed-questionnaire and interview transcripts scruti-
nized all explanations of answers for their “quality” (the number of arguments
or aspects contained in them).

Results

GENERAL EVALUATION BY THE PANEL MEMBERS

Of the 63 respondents that were questioned in three ways, and who returned the
evaluation form, a clear majority of 38 persons stated that their opinion appeared to the
fullest advantage in the individual interview. Minorities of 14 and eight persons preferred
the group interview and the mailed questionnaire, respectively (three persons had no
preference). The most frequent explanation for the choice of the individual interview is
the ease of oral as compared to written expression. The second explanation mentions the
free expression of one’s own view, not disturbed, interrupted, or changed by others (as
in the group interview). The preference of the group interview was motivated by the
direct, oral way of expression, checked and supported by others. Group interviewees
were able to compare their own view with that of others and develop new ideas. Those
who favored the mailed questionnaire mainly mentioned the possibility of thinking longer
and more deeply about the answer and of reading the questiomnaire) more than once.
Previous education appeared to be the best discriminating variable here. None of
298 J.A.G.M. VAN DIJK

the respondents with low previous education favored the mailed questionnaire. The mailed
questionnaire was most preferred by panel members with medium previous education,
whereas the group interview was appreciated most by members with high previous ed-
ucation. This result seems interesting enough for continued research into this relationship.

SUBJECTIVE PERFORMANCE OF PANEL MEMBERS

A number of highly correlated self-rating scales was taken together in the summary
of scales of appeal to cognition, effort, involvement, and self-confidence. They rated the
subjective performance of the participants of this Delphi study. (For complete results see
Reference 1.)
The appeal to cognition, a summary of the scales “difficulty of,” “attention to,” and
“(not)self-assuredness at answering,” was largest in the case of the mailed questionnaire
and smallest in the group interview (see Table 2). The efort ofparticipation, summary
of the scales “difficulty of,” “fatigueness of, ” “attention to,” and “tension of answering,”
is distributed in the same way across the techniques, in Rounds 2 and 3. Self-confidence,
a summary of the scales “difficulty of,” “self-assuredness at,” “reservedness at,” and
“tension of answering,” is also backed most in the group interview and least in the mailed
questionnaire, particularly in Rounds I and 2. The involvement in the Delphi discussion
process, a summary of the “attractiveness of,” “motivation at,” “reservedness at,” and
“attention to answering,” appears to be promoted best in the individual interview, directly
followed by the group interview, and least in the mailed questionnaire.
Generally these performances correspond with the above evaluation by the panel
members. The individual interview was clearly superior in stimulating the participation
in the Delphi process. However, the “good picture” of the group interview is remarkable.
It appears that participation in a group interview demands relatively less effort or cognition
and stimulates self-confidence. The elucidations of the general evaluation mentioned above
and other evidence to be supplied below make us believe that this can be explained by
the support respondents get from other group members under the condition of the group
interview. In the case of the mailed questionnaire they have to solve all problems by
themselves. In the situation of the individual interview the interviewer helps them to get
through the questionnaire, but of course he or she cannot give clues to answers-as is
done by other members in the group interview-and the interviewer remains a stranger
as well as someone who, being an expert, is supposed “to know the answer.” These
conjectures are derived from a contextual theory of the interview [ 12, 131.

OBJECTIVE PERFORMANCE OF PANEL MEMBERS

The most important thing to know would be whether significantly different answers
were recorded under the three conditions. For that purpose a large number of answer
categories was packed together in the smallest logical groups, the same that were used
to structure the feedback reports. Most of the time the analysis produced three groups
containing positive (e.g., acceptable), negative (e.g., against), and neutral (e.g., no
choice) categories. The results are staggering: 58% of the questions produce statistically
significant (below the p = 0.05 level) different answers across the three conditions. As
will be demonstrated below this could result from the greater number of no-opinion and
no-answer categories in the mailed questionnaires and the group interviews (silent per-
sons). Now, if these categories are kept out, and only the positive and negative ones are
left, the number of questions with statistically significant different answers still remains
27%! In order to analyze the direction of these differences, the subject matter of this
Delphi study and some characteristics of the panel members have to be brought forward.
The largest differences are recorded between the mailed questionnaire condition on one
DELPHI VERSUS INTERVIEWS 299

TABLE 2
Mean Scores of Summary Self-Rating Scales (l-7) by Questioning Conditions in Three Delphi Rounds
Analvsis of Variance

Round 1 Round 2 Round 3

Appeal to Cognition (n) 00 (n)

Total mean score (87) 4.20 4.04 (79) 4.24
Group interview (25) 3.55 (24) 3.01 (30) 3.67
Individual interview (25) 4.61 (25) 4.06 (30) 4.55
Mailed question (37) 4.37 (30) 4.86 (19) 4.57

P= 0.07 0.00 0.11

f= 2.65 7.08 2.25

Effort
Total mean score (84) 3.78 (64) 3.95 3.94
Group interview (24) 3.61 (16) 3.50 i29j 3.59
Individual interview (23) 3.80 (23) 4.05 (30) 4.10
Mailed question (37) 3.87 (25) 4.13 (19) 4.23

P= 0.55 0.05 0.04

f= 0.60 3.03 3.34

Involvement
Total mean score (85) 5.78 (81) 6.01 (78) 5.95
Group interview (25) 6.01 (25) 6.13 (29) 5.88
Individual interview (23) 6.06 (27) 6.24 (30) 6.10
Mailed question (37) 5.45 (29) 5.69 (19) 5.82

P= 0.01 0.03 0.52

f= 4.30 3.58 0.65

Self-confidence
Total mean score (87) 5.09 (79) 5.34 (78) 5.21
Group interview (25) 5.50 (24) 5.86 (29) 5.42
Individual interview (25) 4.89 (25) 5.35 (30) 5.13
Mailed question (37) 4.95 (30) 4.91 (19) 5.05

P= 0.11 0.00 0.45

f= 2.22 5.56 0.79

All D.F. = 2

“(N) = valid response (all cases with a missing scale value excluded).

side and the individual- and group-interview conditions on the other. However, the only
(small) tendencies that can be noticed here (with caution) are that the first one registers
relatively more answers showing satisfaction with the policies of management, pessimism
about future developments, disagreement with known union stands, and unwillingness to
undertake some action. These tendencies seem to be strongest in Round 1. If they really
exist, they can be easily explained by a well-known effect in research interviewing: social
desirability. In any case, these data strongly support a plea to use more than one ques-
tioning technique in a Delphi project!
Although the questions were not reiterative, opinion change between rounds could
be measured as a change of attitude on (sub)themes that did remain the same during the
study. This change of attitude was defined narrowly as a switch to the other side (for
 300 J.A.G.M. VAN DIJK

TABLE 3
Mean Percentage of No Opinion and No Answer Across Three Delphi Rounds

Round 1 Round 2 Round 3

Group interview 22.5 17.6 19.5

Individual interview 6.8 1.8 0.4
Mailed questionnaire 14.4 11.2 11.2

Total 15.0 10.6 1n 6

instance from positive to negative; neutral categories were left out). Such change amounted
to 26.4% between Rounds 1 and 2, and 15.3% between Rounds 2 and 3 (see Reference
1 for complete results). It might be expected that the change would be greatest in the
case of the group interview and smallest in the case of the mailed questionnaire. This
relationship was not found. In Round 2 the highest percentage change was observed in
the case of the individual interview and in Round 3 in the case of the mailed questionnaire.
(However, the numbers are too low to reveal any statistically significant relationship.)
How can this be explained? One self-rating scale filled in after every round was called
“consciousness of the opinion of others.” In Round 1 this consciousness proved to be
significantly (P = 0.0004) highest in the group interviews and lowest in the mailed
questionnaire. However, in Rounds 2 and 3 this relationship disappeared. Presumably
this is caused by the discussion process that engaged the panel members in all questioning
conditions. Anyway, it is safe to conclude that the group interviews did not influence
the individual opinion of panel members significantly more than the other questioning
techniques.
The definitive dropout in participation was low: only six out of a total maximum
panel membership of 101. Four left the project for “objective” reasons such as prolonged
sickness, two for “subjective” ones (too much effort required). None of these reasons
had any connection with questioning techniques; however, the temporary dropout (one
round) did. Seventeen members did not complete the second round and 11 did not complete
the third one. The most important reason was a missing or overtime return of mailed
questionnaires, even after three (mailed) requests. There were no failed individual inter-
views. Only three persons missed an appointed group interview and were questioned
otherwise.
The mean percentage of “no-opinion” and “no-answer” responses was distributed as
shown in Table 3.
All these differences between the three conditions are statistically significant (below
the P = 0.001 level), but actually it is no surprise that the group interview and the mailed
questionnaire produce the highest rates of no opinion and no answer. Perhaps it is
remarkable that the rate of the individual interview is so low, almost zero in Round 3.
The high rate of the group interview is largely caused by its main problem: the silent
person. However, we would like to stress that it was a relatively mild problem in this
study. As will be demonstrated below, individual arguments were expressed remarkably
well in the group interviews. The differences among the social categories of participants
with regard to silence in group interviews or no-opinion and no-answer responses under
the other conditions, that are worth mentioning (only few of them are statistically sig-
nificant), reveal the same tendencies as with other performances: these responses are
given more often by panel members with low previous education and job level, by non-
trained union members, and only slightly more by women (see Reference 1, Table 1).
They tend to diminish in Rounds 2 and 3.
The quality of answers, defined as the number of arguments and additions (details,
DELPHI VERSUS INTERVIEWS 301

aspects) supplied in the explanations, was very different among the three conditions. A
detailed content analysis was made of full answer transcripts on the same themes and
questions that were taken to analyze opinion change. One by one all sentences in the
transcripts and returned questionnaires were scrutinized for (new) arguments and additions
to arguments supplied by the respondents. A valid argument was defined as an instance
of one of the answer categories of the substantive investigation (every question received
15 to 20 of these categories), except for the no-opinion and no-answer categories, and
the categories containing an answer without any explanation. The code sheets contained
four column labels: 1) no explanation (argument or addition), 2) explanation (argument),
3) addition (of arguments in previous sentences), and 4) total number of sentences enum-
erated. So the unit of enumeration was a sentence and the recording unit was a full
explanation (of an answer) by an individual respondent (see [ 15, 161 for terminology).
This recording unit was divided into two parts, sentences before and after any second
request for explanation by the interviewer, to be able to compare the answers in interviews
and mailed questionnaires. The scores of two independently working coding assistants
(a student in social science and a student in Dutch language) were checked for reliability.
A Pearson correlation of codings was preferred as a measure of reliability. I The
mean correlation on all questions (before the second probe of interviews) appeared to be
+0.78 for arguments (in themselves), +0.77 for additions, and +0.90 for the total
number of sentences containing a valid answer. This is acceptable for a content analysis
largely based on qualitative interpretation. The results are summarized in Table 4.
All comparisons between the three questioning methods that can be made in Table
4 are strongly significant (below P = 0.005level). Most arguments and additions are
produced in the individual interview (before a second probe). They also increase during
the rounds most under this condition. The group interview is a surprisingly good second
best. Contrary to a very common belief it appears to be possible to record individual
opinions under the condition of a group interview. A comparison between the results of
the group interview and that of the most individual and anonymous technique, the mailed
questionnaire, makes this clear. A comparison of the group interview with the individual
interview reveals that with the individual interview more arguments are expressed. This
can be explained by the second arguments that were very common in this project. In a
group interview panel members had to defend their first argument against others. That
is the main reason why there were a relatively large number of additions recorded in the
group interviews, and one of the reasons why the mean individual sum of explanation-
sentences is even larger in group interviews than in individual interviews! The mean
individual speaking time of respondents in the individual interview was 32 minutes and
in the group interview 24 minutes. However, the whole communication process was much
more hectic in the group interview: the sentences were shorter, people interrupted others
and talked over each other sometimes.
The performance of the mailed questionnaire is relatively bad. This even goes for
the arguments in themselves in the explanations of answers. Frequently explanations are

‘Measures of inter-observer agreement generally are the best reliability tests for a content analysis, provided
that they correct for agreement by chance (see [ 15, 161). Correlation measures can fail to register systematic
coding errors [ 151. Yet there were three reasons to prefer a Pearson correlation measure in this content analysis:
1) a fixation of agreement by chance was not possible as the number of arguments and additions to be recorded
had no definable upper limit; 2) agreement of codings at the available ratio level (number of arguments etc.)
was observed on a finer distinction in a Pearson correlation than in a rough (dual) measure of agreement (yes
or no); 3) correlations were computed for every eight categories that had to be recorded for each unit; this
allowed a very detailed analysis for systematic coding errors comparing the coding assistants on a series of
questions (e.g., for effects of tiredness and learning).
 302 J.A.G.M. VAN DIJK

TABLE 4
Mean Number of Arguments, Additions and Sentences in Explanations
by Questioning Conditions, in Three Delphi Rounds”

Round 1 Round 2 Round 3

Arguments
(in themselves)
Total mean number 1.10 1.10 1.39
Group interview 1.10 0.82 1.23
Individual interview 1.47 1.22 1.81
Mailed questionnaire 0.84 0.93 0.97

Additions
Total mean number 0.88 1.10 1.55
Group interview 1.23 1.33 1.36
Individual interview 1.27 1.45 2.54
Mailed questionnaire 0.36 0.72 0.34

Sentences
Total mean number 2.96 3.62 5.67
Group interview 4.19 5.28 7.84
Individual interview 3.76 4.27 6.41
Mailed questionnaire 1.42 2.01 1.49

Arguments + Additions i Sentences

Total percentage 74% 62% 52%
Group interview 55% 36% 33%
Individual interview 72% 62% 67%
Mailed questionnaire 84% 82% 87%
“Interviews: before a second probe/request for explanation.

omitted and only a box is marked. This result raises doubts as to the suitability of this
technique in this type of Delphi method (Policy Delphi), this type of questions (forced
choice with explanations), and this (nonexpert) panel membership.

OTHER PERFORMANCE
If other norms are involved the mailed questionnaire performs better. Of course it
is still the cheapest, most time-saving, and (in the Delphi context) anonymous method
of all. The answers are much more “to the point,” as is shown in Table 4 when the
arguments and additions are divided by the total number of sentences. (In an individual
interview and especially in a group interview, deviations from the subject matter of the
question are much more likely.)
The financial costs are much higher in individual and group interviews (appointments,
traveling, transcription of interviews, categorizing of answers). The dispersion of panel
members made the individual interviews a bit more expensive for the investigator than
the group interviews (traveling costs).
The time budget has to be assigned in the same order. The individual interview was
most time-consuming (traveling, a mean total speaking time of 60 minutes, including the
long introduction of questions, as compared to 165 minutes in the group interview (with
a mean of 5.3 participants).
The anonymity of the responses in the feedback between rounds was complete.
However, with regard to questioning it was partly broken in the group interview (other
group members) and the individual interview (the interviewer). This might have aggra-
vated the effect of social desirability in the response.
DELPHI VERSUS INTERVIEWS 303

TABLE 5
General Performance List of Three Questioning Conditions in a Policy Delphi Study

Mailed Questionnaire Group Interview Individual Interview

General evaluation
by panel members _ 0 +
Appeal to cognition - + 0
Effort _ + 0
Involvement _ 0 +
Self-confidence - -I- O
Dropout _ 0 +
_
No opinionino answer 0 +

Quality of answers - 0 +

Anonymity + _ 0
Financial costs + 0 -

Time costs + 0
Of = highest, - = lowest, and 0 = middle

Conclusions
Now it is possible to compile a general performance list of three techniques in this
type of Delphi method. Some of these performance criteria are fairly well recognized or
obvious (such as financial costs, time budget, and anonymity). Others were revealed and
empirically established in this case study.
Clearly the subjective performance of the panel members was best in the conditions
of the individual and the group interview (see Table 5). The individual interview motivated
them best. The group interview supported self-confidence, required the least effort, and
caused the least difficulty of answering.
In the objective performance the individual interview came forth as the best technique.
The dropout and the no-opinion and no-answer response rates were lowest. The quality
of answers, defined as the number of arguments and additions in explanations, was the
highest. Here the group interview emerged as a second best. Measuring individual opinions
in the case of a group interview appears to be possible. The biggest problem remains the
number of silent persons (a mean of about 20% per question in this project).
When material and social-political concerns are drawn into the comparison the mailed
questionnaire technique scores better. Presumably this explains its popularity in the prac-
tice of Delphi research. Contrary to that, this study has called attention to the costs in
terms of the quality of participation and response of panel members that have to be paid
for these advantages in material costs and anonymity of participation. It seems likely that
this goes for all Delphi studies that do not aim at the quantitative evaluation of future or
present developments by a panel of experts. Particularly Policy Delphis need techniques
that stimulate argument and discussion. Besides that, face-to-face techniques do not always
have to be so expensive in this context. Most often it is not necessary to interview 100
persons, as in this project; a lot of Policy Delphis work with only half as many panel
members. The computer software facilitating qualitative ways of interviewing and ana-
lyzing is advancing rapidly. If face-to-face techniques are used, it remains possible to
keep the feedback completely anonymous and the interviewing partly so.
Our general conclusion would be that face-to-face techniques deserve to be better
weighed against mailed questionnaires in Delphi studies. This study suggests that they
might produce different results. In that case a combination of techniques would be
desirable. We have found some evidence (although our nonrandomized design was not
suited to prove this) that a good combination for a three-round Delphi study might be:
 304 J.A.G.M. VAN DIJK

the technique that best motivates participation for an introductory round (the individual
interview), the technique that best supports discussion and self-confidence (the group
interview) for a second round, and the most anonymous individual technique (the mailed
questionnaire) for a final voting or decision round. This would have to be weighed against
the benefits of a combination within rounds, primarily the neutralization of answer dif-
ferences .

References
1. Dijk van, J. A. G. M., Popularising Delphi Method, Developing an Instrument to Control Technological
Change for Employees, Quality and Quantity 22, 189-203 (1989).
2. Dijk van, J. A. G. M., Delphi Method as a Learning Instrument: Bank Employees Discussing an Automation
Project, Technological Forecasting and Social Change 38 (in press, 1990).
3. Delbecq, A. L., Van de Ven, A. H., and Gustafson, D. H., Group Techniquesfor Program Planning: A
Guide to Nominal Group and Delphi Processes, Scott, Forestman, Glenview, Ill., 1975.
4. Gustafson, D. H., Shukla, R., Delbecq, A., and Walster, G., A Comparative Study of Differences in
Subjective Likehood Estimates Made by Individuals, Interacting Groups, Delphi Groups, and Nominal
Groups, Organisational Behavior and Human Petformance 9 ( 1973).
5. Moore, C. M., Group Techniques for Idea Building, Sage, London, 1987.
6. Riggs, W. E., The Delphi Technique, an Experimental Evaluation, Technological Forecasting and Social
Change 23 (1983).
7. Van de Ven, A., Group Decision Making and Effectiveness, Kent State University Press, Kent, Ohio,
1974.
8. Turoff, M., The Policy Delphi, in The Delphi Method; Techniques and Applications, Harold A. Linstone
and Murray Turoff, eds., Addison-Wesley, Reading, Mass., 1975.
9. Rauch, W., The Decision Delphi, Technological Forecasting and Social Change 15 (1979).
10. Linstone, H. A., and Turoff, M. (eds.), The Delphi Method: Techniques and Applications, Addison-
Wesley, Reading, Mass., 1975.
11. Nelms, K. R., and Porter, A. L., EFTE: An Interactive Delphi Method, Technological Forecasting and
Social Change 28 (1985).
12. Dijk van, J. A. G. M., Contextual Influences on the Survey Interview, paper presented at the International
Conference on Social Science Methodology, Dubrovnik, May-June 1988.
13. Dijk van, J. A. G. M., Aanzetten tot een Contextuele Theorie van het Interview, Mens en Maatschuppij
63(3) (1988).
14. Sackman, H., Delphi Cririque: Expert Opinions, Forecasting, and Group Process, Lexington Books,
Lexington, Mass., 1975.
15. Krippcndorff, K., Content Analysis, an Introduction to Its Methodology, Sage, London, 1980.
16. Holsti, 0. R., ContentAnalysisfor the Social Sciences and Humanities, Addison-Wesley, Reading, Mass.,
1969.

Received 3 October 1988: revised 19 May 1989

View publication stats