TFS-17336; No of Pages 8

Technological Forecasting & Social Change xxx (2010) xxx–xxx

Contents lists available at ScienceDirect

Technological Forecasting & Social Change

Delphi: A brief look backward and forward

Harold A. Linstone a,⁎, Murray Turoff b
a
Portland State University, P.O. Box 751, Portland, OR 97207, United States
b
New Jersey Institute of Technology, Newark, NJ 07102, United States

a r t i c l e i n f o a b s t r a c t

Article history: In response to a request by the guest editors, we have set down our thoughts regarding the
Received 29 September 2010 evolution of Delphi, beginning with our immersion in the subject in the late 1960s and
Accepted 29 September 2010 concluding with some rumination about its future. Our focus is on the changing roles of Delphi.
Available online xxxx Most importantly, with the profound impact of the internet on organizational and community
planning systems, it will foster a new age of participation through communication,
Keywords: coordination, and collaboration.
Delphi © 2010 Published by Elsevier Inc.
Social networks
Participative planning
Communities of practice
Policy Delphi
Delphi conferencing

1. The Past

Delphi was developed at the RAND Corporation in the 1950s by Olaf Helmer, Norman Dalkey, Ted Gordon and associates under
the auspices of the U.S. Air Force as a technique to apply expert input in a systematic manner using a series of questionnaires with
controlled opinion feedback. Key features were preservation of anonymity in the expert panel's responses and iteration of the
questionnaires. A key benefit of participation was the ability of individuals to participate in a group communication process
asynchronously at times and places convenient to them. The initial applications were in the area of national defense, with the first
unclassified use a RAND report [1] on long-range forecasting. An offspring was the Trend Delphi, in which a group extrapolates a
trend curve into the future.
We both encountered Delphi first in 1968: Turoff in the work for the Office of Emergency Preparedness (OEP) and Linstone in
the preparation of the very first issue as editor of the new journal Technological Forecasting and Social Change (TFSC). That issue
already included an article on a U.S. Navy technological forecasting procedure entitled “SEER: a Delphic Approach Applied to
Information Processing” by G. Bernstein and M. Cetron. Our two paths soon crossed as the second and third TFSC volumes (1970–
1972) featured Turoff's articles on “Design of a Policy Delphi” [2] and “Delphi Conferencing” [3], respectively. A Policy Delphi seeks
to expose different policy options and the most important pro and con evidence or arguments to support each policy resolution.
Delphi Conferencing refers to an online computer-mediated asynchronous conference system with anonymity.
Our Delphi book [4] featuring many articles from TFSC was published in 1975. In a foreword Olaf Helmer wrote “Delphi has
come a long way in its brief history, and it has a long way to go”. He called for the solidification of the Delphi technique and foresaw
its potential importance in supplying ‘soft’ data in the social sciences. Now a centenarian, he can view his comments more than
thirty years ago with eminent satisfaction.

⁎ Corresponding author.
E-mail addresses: linstoneh@aol.com (H.A. Linstone), turoff@njit.edu (M. Turoff).
URL: http://is.njit.edu/turoff (M. Turoff).

0040-1625/$ – see front matter © 2010 Published by Elsevier Inc.

doi:10.1016/j.techfore.2010.09.011

Please cite this article as: H.A. Linstone, M. Turoff, Delphi: A brief look backward and forward, Technol. Forecast. Soc. Change
(2010), doi:10.1016/j.techfore.2010.09.011
2 H.A. Linstone, M. Turoff / Technological Forecasting & Social Change xxx (2010) xxx–xxx

A recent development is the Problem Solving Delphi, a system for providing collaborative judgment [5]. It collects participants’
rankings or paired comparisons. Using Thurstone's Law of Comparative Judgment the votes of the N participants are converted to a
single group interval scale where the distance between objects in the scale is proportional to the amount of agreement on
preference order. Two solutions or components of solutions at the same point on the scale would be due to a 50–50% vote on
whether A was preferred to B or B was preferred to A. Participants are encouraged to focus their discussion on resolving the major
disagreements [6,7].
Over the past four decades more articles in TFSC have been devoted to Delphi than to any other technique in the domain
covered by the Journal. A recent featured article is “RT Delphi: an efficient “round-less” almost real time Delphi method” by Ted
Gordon and Adam Pease [8]. It exemplifies how computers and the Internet have enhanced the original concept by allowing for
computer-mediated asynchronous communication that is now accessible globally by any group member. This flexibility makes it
possible for the user to participate in any phase of a decision process at any time.
The popularity and proliferation of Delphi literature over half a century inevitably means that more than one generation of
analysts is involved. The Linstone–Turoff book [4] already included a bibliography of 670 Delphi-related items and by now the
number of papers must be in the thousands. Since the book was made available free online, the number of citations has more than
doubled to over 2200 (Google scholar: 8/15/2010). As a new generation enters the field it is oblivious of some of the work done by
a previous generation with the result that it “reinvents the wheel.” Computer Mediated Communications (CMC) is particularly
subject to this problem. The computer field is driven more by industry than by academics and every company seeks a new name
for its “new” system to reflect its alleged uniqueness.
This is a continuing problem in the areas of Computer and Information Systems where every vendor wants to make his system
sound new and rarely points out the evolution based upon earlier systems. Derivatives of Delphi have emerged under names such
as prediction markets, collaborative tagging, recommender systems (like Netflix), and social networks that usually serve a
commercial objective [7]. But a new name does not necessarily imply a new field.
In the Delphi area the specific topics of Group Decision Support Systems (GDSS), Collaborative Systems, and Collective
Intelligence, and CMC are areas employing many of the ideas from the Delphi area [9,10]. These papers categorize hundreds of
experiments and field studies in Group Decision Support Systems based in part upon common Delphi factors such as anonymity,
feedback structures, pen names, communication process structures, and voting. In the 1978 and 1993 editions of The Network
Nation [11], Hiltz and Turoff extrapolated the concept of Delphi in the form of Computer Mediated Communications as attempts to
produce Collective or Collaborative Intelligence among groups. This is the seemingly "simple" concept that an effective group
communication process should allow the group to reach a "better" result than any member of the group would have determined
acting alone.
One of the most important aspects of Delphi we attempted to illustrate in the Delphi book [4], by the wide variety of examples
we chose, was that every communication structure presented in the different applications was specifically tailored to the “nature
of the application and the nature of the participating group”. We focused on the philosophy C. W. Churchman in his book on The
Design of Inquiry Systems [12] and that of Heidegger's “negotiated reality” as not only templates for the design of information
systems but also for the design of Delphi processes [4 (Chap. 2)]. This concept has been extended in a number of ways in that the
later work by Dr. Turoff applied the same approach to Computer Mediated Communications and to online versions of Delphi [11,
(Chap. 14); 13,14].
Today when “collaborative structures or knowledge structures” are referred to in the literature , they are another set of names
for exactly what was meant in the 1975 book by us as Delphi or group communication structures. A major concern in designing a
Delphi, particularly online versions, is that the structure allows the participants to classify easily their inputs, particularly
qualitative ones, in a morphological structure they can mutually understand. The resulting knowledge structure and its evolving
content can prevent information overload [15]. In the future, we hope that participating groups will be able to collaboratively build
and evolve the knowledge structure that they desire for their ongoing deliberations. Evolving structures with user participation
was an essential part of the EIES (Electronic Information Exchange System) from 1976 to the mid-nineties [11,16].
Recent experiments and field trials have confirmed the efficacy of using graduate online learning groups in upper division
courses oriented to collaborative learning for the total class [17,18]. In the 1975 book we specifically recommended research into
the use of Multi-Dimensional Scaling (MDS) as a feedback structure by including an article on that topic. To date there has not been
any meaningful use of that method except for the one dimensional approach of Thurstone to expose the subjective group linear
scale from individual ranking or paired comparison input. The fact that MDS can now be run on one's personal computer might
change that in the future. Also MDS is a perfect method for trying to aid a group to find the underlying dimensions or factors they
are using to represent knowledge in their topic.
The “Tower of Babel” we often find in the professional literature when crossing disciplinary lines is by no means unique to
Delphi, as demonstrated in a special TFSC issue on Strategic Foresight developed by Coates, Durance, and Godet [19]. There is much
discussion of Technology Foresight as a new concept developed in the UK in the 1980s by SPRU (Science and Technology Policy
Research at the University of Sussex, UK) and others. Actually work that is now defined as technology foresight was done in the
1960s but not under that name. Not surprisingly, the new generation of researchers is often unfamiliar with their predecessors’
work, which may have been done in a different institution or country or in the context of classified national security projects.
An historical appreciation of the evolution of a field, as well as codification of terminology, usually comes at a later stage, when
maturity sets in and textbooks are compiled.
Before proceeding, we should address a persistent misperception about Delphi. Over the years as a reviewer and editor-in-chief
of TFSC it has often been necessary to correct the mistaken impression that the aim of Delphi is consensus. Our 1975 book clearly

Please cite this article as: H.A. Linstone, M. Turoff, Delphi: A brief look backward and forward, Technol. Forecast. Soc. Change
(2010), doi:10.1016/j.techfore.2010.09.011
 H.A. Linstone, M. Turoff / Technological Forecasting & Social Change xxx (2010) xxx–xxx 3

states that Delphi is “a method for structuring a group communication process”, not a method aimed to produce consensus. The
number of rounds should be based on when stability in the responses is attained, not when consensus is achieved. In fact, a bipolar
distribution may be a result and a very significant one indeed. This shows a crucial difference between Delphi and a traditional
panel, where consensus is desired and may even be forced. Futurist Joe Coates [20] put the aim of the forecast Delphi as follows:

“The value of the Delphi is not in reporting high reliability consensus data, but rather in alerting the participants to the
complexity of issues, by forcing, cajoling, urging, luring them to think, by having them challenge their assumptions… [O]ne
deficiency I see characteristic of many Delphis is the failure to push hard enough on the challenge to concepts and
underlying assumptions. More attention should go into the basis of divergence rather than the basis of convergence.”

In the late 1960s the director of OEP, General Lincoln (Eisenhower's logistic general in World War II and a West Point
Professor), having heard about ongoing work in using the Delphi Method to examine the future of the U.S. stockpile of strategic
materials expressed concern about the fact that bureaucracy often tried to guess what his policy decision would be and to bring
him evidence to support that particular view. He explained that what he really wanted to know as a policy maker was: What were
the strongest arguments against whatever decision he was likely to choose? He believed this would allow him to better defend his
final decision. The result of this was Dr. Turoff designing a Policy Delphi structure to produce the strongest opposing arguments
about resolutions to a policy issue, using the properties of a Hegelian Inquiry process as defined by C. W. Churchman [12].
There are three types of measurements of human judgments that are treated as an academic undertaking:

1. Measurements of judgments that are based upon static unchanging psychological variables so that one can understand what
the variables are.
2. Measurements of judgments to understand how to influence the marketing of products and services.
3. Measurements of human understanding and judgment to try to determine how to promote a group understanding of mental
cognitions by the individuals in the group.

These are very different undertakings and objectives and one cannot say that the knowledge and findings of one of these areas
automatically apply to the other areas. This has often led to confusion about Delphi and unwarranted criticisms of it. The third
objective is clearly what one tries to measure as part of the Delphi process and the most relevant work is in the areas of
visualization and information representation design. We are not trying to get a quick subconscious response to a question we ask
but to stimulate thought and consideration about what is usually a very complex question. We are also trying to make people
aware of the pooling of knowledge and different viewpoints among the participants.

2. The future

The Delphi concept with its various derivatives continues to evolve. Online collaborative learning systems constitute an
emerging area that can make effective use of Delphi [17,18]. They can be done with a class as a whole using any Computer
Mediated Communication System where there is a discussion thread conference system that allows the instructor to assign pen
names to each student to participate in the Delphi process. This might also include role-playing “games” as well as enactments of
the views of stakeholders represented by the students’ pen names for any multi-sided situation. The instructor, or even one of the
students, can collect individual items to be voted on and put them into an online survey system to allow the class to vote every few
days on differing options they are considering.
Recently some discussion board systems have started to add voting options that would present a more integrated process for
carrying out a limited Delphi process. However, the growing number of facilities to carry out a Delphi on the Web has definitely led
to the start of a new growth curve in the use of these methods for the support of large collaborative groups. The fact that most
professionals and just about all college students do have Web access and their own computers makes possible many applications
of Delphi where the computer can handle the organization of the material into a meaningful structure in a dynamic manner
without the round structures that characterized the paper and pencil Delphi exercises. Large numbers of proprietary Delphis in
organizations have been done with the merger of a discussion based bulletin board system and a survey system.
An outstanding challenge is to extend the Delphi concepts to allow large groups to build collaborative group models. An
approach that is promising for this is the incorporation of the concept of “structural modeling” [21–23] to allow the users and the
group to:

1. Reach consistency in specifying the inputs to a complex modeling situation.

2. Produce outputs from each individual's model, once consistent, which allows the results to be placed into the formulation of a
group model.
3. Highlight disagreements or differences of views about the model and use that to ask participants to consider disagreements
about such things as the direction of influence between two variables in a relationship within the model.

The concept of structural modeling is to provide the user with the ability to make subjective estimates about a problem that can
be used in a computer program to build a working model of the person's cognitive model. Equally a group can express comparative
inputs about a problem and build a collaborative model. This Delphi process for collaborative model building can be used to evolve
the model on a continuous basis for the update of the model as better data becomes available [24]. As with other online Delphis any

Please cite this article as: H.A. Linstone, M. Turoff, Delphi: A brief look backward and forward, Technol. Forecast. Soc. Change
(2010), doi:10.1016/j.techfore.2010.09.011
4 H.A. Linstone, M. Turoff / Technological Forecasting & Social Change xxx (2010) xxx–xxx

one at any time can input new material or change earlier views to any part of the knowledge structure that defines the scope of the
group activity and the process may be integrated into a continuous planning process rather than single plan for X years.
It is amusing that one of the first classified Delphis at RAND in the 1960's was specifically designed to replace a computer based
simulation model. At the time the U.S. census tapes providing factory floor space (i.e., square feet) as the only measure of
production potential by industry type were used to determine the hypothesized targeting of U.S. locations based upon the Russian
nuclear threat. A computer calculated the reduction of floor space in a given U.S. industry to arrive at a picture of how bad the
results would be. It was felt by some at RAND that this approach was very misleading given examples of, say, only one plant in the
U.S. that made huge landing gears for large transport planes. Only one target was needed to make all that manufacturing floor
space for large transport planes around the country useless. The famous expression of “garbage in, garbage out” came from many
examples of computer models that ignored the knowledge of professionals or their subjective inputs and instead built models
around quantitative data sources even when the assumptions behind the available data were not really relevant to the problem
being addressed.
The original RAND effort above sent a Delphi to 5 to 15 experts in a given industry and asked them to rank order the most
critical top ten targets in their industry and explain why each was so important. This turned out much more insightful results than
the use of the Census tapes providing only floor space data and much less expensive in those days (e.g. computer processing costs
versus the time of people). In most areas of foresight and future forecasting what is important are the details of conditional (IF x
THEN y) forecasts which clarify the potential relationships among future events. This yields the information necessary for real
planning and the integration of many aspects of planning into one system that provides relationship modeling to determine the
most important options for the future [23,24].
The design of the Steel and Ferroalloy Delphi application (4, (chapter III.C.3)) illustrated the challenge of model building. Three
experts in this industry were asked to create a commodity flow model of materials. This led to a network of nodes having 45 links.
Of these fundamental variables, only nine were collected on a yearly basis. Finding data to make meaningful models is harder than
most model builders realize. So we decided to ask the other forty-one industrial experts to take this flow model and fill in the
missing estimates for the prior year. Twenty-five of the experts actually made independent changes to the flow model as they did
not agree with the three that built the original simplified model of the industry.
Dr. Linstone has spent much time since 1977 on the concept of multiple perspectives in addressing complex systems. Two
books published in 1971, Churchman's The Design of Inquiring Systems [12] and Allison's Essence of Decision: Explaining the Cuban
Missile Crisis [25] as well as personal experience in corporate planning, suggested a means to bridge the glaring gap between
traditional systems analysis and the real world. Churchman developed what he called a Singerian Inquiring System.1 Allison
examined the Cuban missile crisis from three points of view: rational actor, organizational process, and bureaucratic politics. Each
yielded unique insights.
In the corporate environment it was found exceedingly valuable to introduce two perspectives to augment the systems analysis
approach in examining a complex system [26,27]. A system can be viewed through distinct lenses: the engineer “sees” a proposed
new venture very differently than does the CEO in the same corporation. The former focuses on the technical aspects while the
latter must be concerned with organizational questions, such as the rivalry between departments when the inevitable reallocation
of budgets to support the new system impinges on current activities. Then there are also considerations that reflect the personal
career plans for those involved with the new system—who will gain and who will lose?
We label the traditional approach the technical or T perspective; the added ones are the organizational/institutional (O) and
the personal/individual (P). Each uses distinct paradigms and sweeps in insights not attainable with the others. (For typical
perspective characteristics see Table 1)2 Multiple perspectives provide a much more meaningful basis for planning than does the T
perspective alone. They are particularly valuable in identifying the differences in assumptions underlying forecasts and that
frequently bias them.
The categories in Table 1 are general aspects of what you would like participants in a Delphi to be aware of as triggers to
stimulate new contributions. Attention to the O perspective will also make it much more likely that the institutional changes that
must often accompany technological changes to make them effective will not be ignored. The T-O-P concept underscores an
insight derived in complexity science about adaptive systems, namely, that each system element does not see the whole picture
but must rely on its own internal models, that is, its own perspective. In other words, ‘rational’ organizational (O) or individual (P)
behavior is by no means equivalent to the rational technical perception (T) of the system (and its optimization). Below we discuss
two significant considerations that relate this concept to Delphi, experts’ T-focused bias and creating a shared reality:
a. Experts’ T-focused bias
Engineers and scientists are prone to be too optimistic in the short term and too pessimistic in the long term. This bias is
ingrained in their T-focused training. In the short term they feel a new technical solution or concept is feasible, therefore it will
be implemented. In other words, if it can be done it will be done. They tend to underestimate organizational or other
nontechnical difficulties (funding, political obstruction, etc.) that may impede any realization of the technological concept. In
the long term they may not be able to envision a technological solution and assume therefore that there will not be one,
reflecting a lack of imagination. The same applies to software developers trying to estimate efforts needed to produce a piece of

1
It is in essence a pragmatic meta-inquiring system that considers the system designer a fundamental part of the system and takes holistic thinking seriously
by constantly sweeping in new components.
2
Later a fourth perspective, the religious/mythological (R), was added.

Please cite this article as: H.A. Linstone, M. Turoff, Delphi: A brief look backward and forward, Technol. Forecast. Soc. Change
(2010), doi:10.1016/j.techfore.2010.09.011
 H.A. Linstone, M. Turoff / Technological Forecasting & Social Change xxx (2010) xxx–xxx 5

Table 1
Typical characteristics of multiple perspective types.

Technical (T) Organizational (O) Personal (P)

Worldview Science-technology Unique group or institutional view Individual, the self

Objective Problem solving, product Action, process, stability Power, influence, prestige
System focus Artificial construct Social Genetic, psychological
Mode of inquiry Observation, analysis: Consensual, adversary bargaining and Intuition, learning, experience
data and models compromise
Ethical basis Logic, rationality Justice, fairness Morality
Planning horizon Far (low discounting) Intermediate (moderate discounting) Short for most (high discounting for most)
Other descriptors Cause and effect Agenda (problem of the moment) Challenge and response, leaders and followers
Optimization, cost-benefit analysis Satisficing Ability to cope with only a few alternatives
Quantification, trade-offs Incremental change Fear of change
Use of probabilities, averages, Reliance on experts, internal training Need for beliefs, illusions, misperception
statistical analysis, expected value of practitioners of probabilities
Problem simplified, idealized Problem delegated and factored, issues Hierarchy of individual needs (survival to
and crisis management self-fulfillment)
Need for validation, replicability Need for standard operating procedures, Need to filter out inconsistent images
routinization
Conceptualization, theories Reasonableness Creativity and vision by the few, improvisation
Uncertainties noted Uncertainty used for organizational Need for certainty
self-preservation
Criteria for “acceptable risk” Logical soundness, openness to Institutional compatibility, political Risk aversion
evaluation acceptability, practicality
Scenario typology Probable Preferable Possible
• Criterion analysis (reproducible) value (explicative) image (plausible)
• Orientation exploratory (extrapolative) normative (prescriptive) visionary
• Mode structural participative perceptual
• Creator think-tank teams stakeholders individuals
Communications Technical report, briefing Insider language, outsiders’ assumptions Personality, charisma desirable
often misperceived

software. Most experienced managers know this and they then increase the effort needed for short term jobs and leave the long
term ones alone! In theory, there is a crossover point for every individual where they predict the effort with complete accuracy.
However, it is very likely the location of the crossover point, at which this occurs, may be different for each individual.
b. Creating a shared reality
T-focused scientists and engineers and their forecasts are not the only justification for drawing on multiple perspectives to
assure realism. Policy and decision Delphis must deal with organizations and individuals. A look at Table 1 suggests that the O
perspective recognizes factors quite different from those that concern T-oriented analysts, such as an organization's agenda,
preference for satisficing, standard operating procedures, political acceptability, problem delegation, and more discounting
than is the norm with the T perspective. The P perspective in turn focuses on power, influence, a need for certainty, risk
aversion, the counterintuitive nature of probabilities, and long-held beliefs. These perspectives sweep in considerations that
underlie Delphi responses. Personal interviews are often the only way to elicit these considerations. They require unique
subtlety and sensitivity not generally found in computer processes or T-trained persons. The detailed recommendations for
such interviewing in Linstone [27. pp. 290-300] make this very clear. Some sample questions for an astute interviewer:

For organizational actors –What are the unstated aims of the group? When and how are standard operating procedures
circumvented? Who are the organization's strongest friends and strongest enemies?
For individual actors –What are the career objectives of each key actor? Who is a risk taker? Who is the most effective action
implementer not shown on the company's organization chart?

Answers to such questions may prove exceedingly valuable in determining the makeup of participants in policy or decision
Delphis. An example is the case of Perinatal Regionalization in Los Angeles County hospitals, where the multiple perspective
analysis was a precursor to a Delphi [27, pp. 284–289]. The aim was to design a regional network organization based on the
different perspectives through participative action. Some Delphis have broken down the voting by the participants to show how
different stakeholders have voted on the given issues. This can prove to be very significant when dealing with potential actions,
decisions, and policies. This is also a useful approach to recommender systems based upon Delphi concepts [7].
In recent years one area of extreme future concerns has become that of Emergency Preparedness and Management. Our success
in this area is directly dependent on the behavior of organizations and the individuals in all the parts and levels of the
organizations. It cannot be accomplished by a small segment of the organization working alone and it has become apparent that
many current assumptions about organizational behavior are not up to current threats [28,29]. In recent years the growing scale
and frequency of crisis and emergency situations have shown a complete lack of adequate foresight and planning by all sorts
of organizations that should have known better. There are fundamental changes needed to allow approaches like multiple
perspective theory, creative risk assessment, future planning, emergency preparedness, and resilient emergency response to take

Please cite this article as: H.A. Linstone, M. Turoff, Delphi: A brief look backward and forward, Technol. Forecast. Soc. Change
(2010), doi:10.1016/j.techfore.2010.09.011
6 H.A. Linstone, M. Turoff / Technological Forecasting & Social Change xxx (2010) xxx–xxx

place. In one of many recent examples at evacuation civic employees did not show up to drive buses because they could not take
their families on the same buses and most highway services were closed leading to clogged highways because there were no
provisions made to keep them open.
In emergency management the concept of High Reliability Organizations was the subject of LaPorte's memorandum in the
Department of Political Science at UC Berkeley in 1989 [27, pp. 184–188). It was also developed in quite some detail by Weick's
work with the Navy and popularized in his book [30]. This is the sort of behavioral process that must not only work in
organizations but also across organizations. We have not overcome the fact that crises and emergencies do not recognize any
organizational, governmental, or geographical boundaries. The response to any large scale situation must cut across all such
boundaries and take on the property of Virtuality [14]. This is where the real world changes as a response to what is happening in
the virtual world. Social networks are a good example of a system becoming as real to the participants online as it once was to
them offline.
Even though physical command and control centers failed in 9/11 and in Katrina, we have not learned about the need for
integrated systems across all the organizations and individuals that need to be involved. Current disaster real time command and
control systems as well as planning systems do not involve most community organizations, volunteer individuals, the press, and
volunteer organizations [31]. Conservatively, thousands cooperating and/or collaborating in a Katrina type of crisis in a very timely
and urgent manner are what is needed. Current efforts of social networks trying to respond to a disaster situation illustrate the
need and the problems that could benefit from the Delphi-like concepts we have been talking about for many years. Social
networks only recently introduced groups as a formal entity in their systems but they seem completely ignorant of what large
groups need to really collaborate effectively.
It is also ironic that in this area we stumble across the same sort of phenomena of rediscovery of what has been understood in
the past. The principles of High Reliability Organizations or High Reliability Theory were made very explicit in the 1959 and 1979
papers [32,33] by Lindblom on "The Science of Muddling Through." In the 1979 paper [33], he details the components of one form
of muddling through as “disjointed incrementalism.” This matches up point by point with what was later termed as HRO or HRT.
He further pointed out in that paper:

“A fast-moving sequence of small changes can more speedily accomplish a drastic alteration of the status quo than can an
only infrequent major policy change.” (Page 520)

What Lindblom believed was that it is fundamental to discover from understanding the past and current situation to determine
anything that is wrong. The organization should have a principal goal to understand even the smallest mistakes and to improve the
situation. If this is not done, such difficulties would grow and compound one another to bring about bigger difficulties that will no
longer be so easy to correct. This also is one of the most fundamental expressions of High Reliability Theory. Both views believe
that to accomplish this, determining and expressing the risks and current mistakes should be accomplished by every employee at
any level. Their viewpoints have to be taken seriously both for reporting problems and suggesting solutions.
Unfortunately for Lindblom the expression “the science of muddling through” was not as acceptable to sponsors from
corporations and government as the term “High Reliability Organizations” and represents another factor in the many new names
for Delphi that have evolved over time. As with many new paradigms there are always strong negative reactions from those
objecting or a paradigm shift. It is still worth reviewing two earlier defenses of Delphi against earlier attacks [20,34].
Today there are many inhibitions in organizations against a free flow of information, especially where bad news is involved.
The suggestion box never works because there is no ability of one participant to support the views of other participants and they
are hardly ever considered by those that have real authority to make changes. This is why identifying risks and current errors are
not done well today in most organizations [29]. There is a real role to be played by a modern online and continuous Delphi system
that allows anonymous contributions (or a meaningful choice to be anonymous or use a real name) by anyone in the organizations
and the ability for all to reinforce via both voting and comments the views of others. This should feed into a process to improve
both current operations and evolving plans for the organization. Ultimately, this will encourage resiliency at all levels in the
organization and a far better ability to deal with the unexpected by the organization and its members [7].
For example, there was a story told over drinks by employees of a major computer firm in the early days of message systems.
Top management issued a white paper explaining the choice of the technology to be used in their next generation products. Many
professionals throughout the organization strongly disagreed with this and a subgroup used the company's internal worldwide
message system to write collaboratively a counter document which they circulated to everyone who was issued the original policy
document. Top management was so outraged they started to demand that the message system be removed from the network. It
took months to convince management that this would completely disrupt many current dispersed efforts. About six months later
the original white paper was withdrawn and a new plan incorporating of the ideas in the counter document was issued. There was
at least one more company where a where similar example occurred that Dr. Turoff is aware of. However, one can be certain this
occurred in many companies after the first introduction of organizational e-mail systems.
In the early days electronic message systems were first installed in organizations by the computing services department for use
by the computer professionals. Such free software came with the early networking software for ARPA net. Other professionals and
managers in other parts of the company learned about this new way of sending memos and messages and some started to use it. It
spread by a process of technological diffusion of a useful service within the organization. There was never any really awareness
or concern by higher level of management nor was there any consideration of many possible implications and policy issues. The
above problem example occurred in slightly different ways in many companies and usually went undocumented. By the time

Please cite this article as: H.A. Linstone, M. Turoff, Delphi: A brief look backward and forward, Technol. Forecast. Soc. Change
(2010), doi:10.1016/j.techfore.2010.09.011
 H.A. Linstone, M. Turoff / Technological Forecasting & Social Change xxx (2010) xxx–xxx 7

top management discovered the problems, the electronic message systems had become too valuable to remove. Some early
commercial products limited message sending privileges to reflect the organization structure and eliminated lateral
communication sending options. Fortunately, these software versions never became very popular and died out relatively fast
[11]. It was the "good old days" when any technological advance was considered good.

3. Conclusions and summary observations

In conclusion, the future of Delphi will be in collaborative organizational and community planning systems that are continuous,
dispersed, and asynchronous. It will replace the impact of controlled surveys as a mechanism of influencing various organizational
and community decision processes. These will allow many thousands to participate or observe an ongoing planning process.
However, given the history of this field it is unlikely these systems will be referred to as Delphi systems. One would hope those that
develop these systems will pay attention to what we already know about large scale collaboration and will not be relearning and
stumbling over the same problems many of us have had to deal with in the past.
The greatest impact the Web is having is the fostering a new age: The Age of Participation. Social networks and their growth are
one manifestation of that age where the public that has adapted to the internet and the younger generations want to voice their
views and feelings about all the matters that concern them. Increasingly, they want to provide the content rather than passively
receive content from select organizations and an imposed leadership. Opinion leaders will evolve as part of the recognition of
their contributions to the community. The ability to communicate, coordinate, collaborate and fully participate with others is the
new online order.
Internet-based collaboration is now occurring even in the most abstract areas of science and mathematics. A recent case is the
proposed proof of a classic mathematical conjecture, known as “P versus NP”. It states that the set of problems that can be easily
solved (P) does not equal NP, the problems that are impossible for computers to solve, but for which solutions are easily
recognizable.3 This is of practical importance as modern cryptography is based on the assumption that P does not equal NP. In
August 2010 a Hewlett-Packard mathematician circulated his “proof” to a number of complexity theorists. A discussion via blogs
and a wiki was quickly set up and discussion and collective analysis was carried out in real time. Web-connected software
programs make such collaborative work now feasible among top researchers on a continuous basis. [35]. Today, the use of the Web
by professional groups is commonly referred to as "Communities of Practice."
Even the academic process of peer review is being challenged. It has been suggested that the internet can serve to extend the
review process to a much broader interested audience. In a recent test case, the Shakespeare Quarterly posted four essays not yet
accepted on a scholarly web site and 41 people offered more than 350 comments which were transmitted to the authors. The
resulting revision was subsequently published. [36]
It is our hope that large groups of participants, in the range from social networks to communities of practice, will be able to
negotiate and evolve their own virtual reality that they find satisfies their requirements and needs. What they will demonstrate
will gradually penetrate more formal organizations. The evolution of both Wikis and Blogs appear to be an early demonstration or
trend indicator of what is possible on a much wider scale. The contrast between early articles in the Wall Street Journal on why
managers should never be seen using a keyboard, to most managers (and school children) today using their thumbs to key things
into their phone almost anywhere, at anytime, gives us some hope we are on the way to the future age of participation.

References

[1] RAND Report P-2982 (Sept.1964) Report on a Long-Range Forecasting Study by Ted Gordon and Olaf Helmer. (For a sample of the results described in this
report, see Linstone (1999), pp. 286-287.}
[2] M. Turoff, The design of a policy Delphi, Technological Forecasting Social Change vol. 2 (1970) 149–171.
[3] M. Turoff, Delphi conferencing: computer-based conferencing with anonymity, Technological Forecasting Social Change vol. 3 (1971-1972) 159–204.
[4] H.A. Linstone, M. Turoff (Eds.), The Delphi method: techniques and applications, Addison-Wesley, Reading, Mass, 1975, (available online at http://is.njit.edu/
turoff).
[5] C. White, et al., A dynamic Delphi process utilizing a modified Thurstone scaling method: collaborative judgment in emergency response, Proceedings of
ISCRAM 2007, 4th International Conference on Information Systems for Crisis Response and Management., Delft, the Netherlands, Brussels University Press,
May 13-16 2007.
[6] Murray Turoff, The past, present, and future of Delphi, FUTURA, The Quarterly Journal of the Finnish Society for Futures Studies, Helsinki, 2009, pp. 32–44,
0785-5494.
[7] M. Turoff, S.R. Hiltz, The future of professional communities of practice, in: C. Weinhardt, S. Luckner, J. Stößer (Eds.), WeB 2008. LNBIP, vol. 22, Springer-
Verlag, Berlin Heidelberg, 2009, pp. 144–158.
[8] T. Gordon, A. Pease, An efficient, “round-less” almost real-time Delphi method, Technological Forecasting Social Change vol. 73 (2006) 321–333.
[9] J. Fjermestad, S.R. Hiltz, An assessment of group support systems experimental research: methodology and results, Journal of Management Information
Systems (JMIS) 15 (3) (Winter 1998-99) 7–150.
[10] J. Fjermestad, S.R. Hiltz, Group support systems: a descriptive evaluation of case and field studies, Journal of Management Information Systems 17 (3)
(Winter 2000) 112–157.
[11] S.R. Hiltz, Murray Turoff, The Network Nation: Human Communication via Computer, Revised EditionMIT Press, 1993 (original edition 1978, Addison
Wesley).
[12] C.W. Churchman, The Design of Inquiring Systems, Basic Books, New York, NY, 1971.
[13] Murray Turoff, Computer mediated communication requirements for group support, Journal of Organizational Computing vol. 1 (1) (1990).
[14] Murray Turoff, Virtuality, CACM vol. 40 (9) (September 1997) 38–43.

3
It is one of the “Millennium Problems” chosen by the Clay Mathematics Institute in 2000 as the seven greatest unsolved mathematics problems.

Please cite this article as: H.A. Linstone, M. Turoff, Delphi: A brief look backward and forward, Technol. Forecast. Soc. Change
(2010), doi:10.1016/j.techfore.2010.09.011
8 H.A. Linstone, M. Turoff / Technological Forecasting & Social Change xxx (2010) xxx–xxx

[15] S.R. Hiltz, Murray Turoff, Structuring computer-mediated communication systems to avoid information overload, Communications of the ACM July 28 (7)
(1985) 680–689.
[16] Murray Turoff, S.R. Hiltz, A.N.F. Bahgat, Ajaz Rana, Distributed group support systems, Management Information Systems Quarterly (MISQ) (December 1993)
399–417.
[17] Turoff, M., Hiltz, S.R., Li, Z., Wang, Y., & Cho, H., “The Delphi Process as a Collaborative Learning Method,” in Moore, J.C. (ed.): (edited by J. C. Moore) Elements
of Quality Online Education: Into the Mainstream: Wisdom from the Sloan Consortium, 121-134. Needham, MA: Sloan-C, Needham, MA September 2004,
pp 121-134.
[18] M. Turoff, Hiltz S.R. Hiltz, Yao X. Yao, Li.Z. Li, Wang Y. Wang, H.K. Cho, Online Collaborative Learning Enhancement Through the Delphi Method, 2006
available at:, http://tojde.anadolu.edu.tr/index.htm.
[19] J.F. Coates, P. Durance, Godet, “Strategic Foresight” issue, Technological Forecasting Social Change vol. 77 (9) (2010).
[20] J.F. Coates, “In Defense of Delphi: A Review of Delphi Assessment: Expert Opinion, Forecasting, and Group Process” by H. Sackman, Technological Forecasting
Social Change 7 (1975) 193–194.
[21] H.A. Linstone, et al., The use of structural modeling for technology assessment, Technological Forecasting Social Change vol. 14 (1979) 291–327.
[22] G.G. Lendaris, Structural modeling—a tutorial guide, IEEE Transactions on Systems, Man, and Cybernetics SMC-10 (12) (1980) 34.
[23] Murray Turoff, Delphi and its potential impact on information systems, AFIPS Conference Proceedings, vol. 39, 1971.
[24] Bañuls, V. A., Turoff, M., and Lopez, J., Clustering Scenarios using Cross-Impact Analysis, Published in the proceedings of ISCRAM 2010, Seattle, May (available
at iscram.org).
[25] G. Allison, The Essence of Decision: Explaining the Cuban Missile Crisis, Little, Brown & Co., Boston, MA, 1971.
[26] H.A. Linstone, Multiple Perspectives for Decision Making, , 1984 New York: North-Holland.
[27] H.A. Linstone, Decision Making for Technology Executives: Using Multiple Perspectives to Improve Performance, Artech House, Norwood, MA, 1999.
[28] M. Turoff, S.R. Hiltz, C. White, L. Plotnick, A. Hendela, Y. Xiang, The past as the future of emergency preparedness and management, Journal of Information
Systems for Crisis Response and Management 1 (1) (January-March 2009) 12–28.
[29] B. Van de Walle, M. Turoff, Decision support for emergency situations, in: F. Burstein, C. Holsapple (Eds.), Handbook on Decision Support Systems,
International Handbook on Information Systems Series, Springer-Verlag, 2008, (This chapter is open on the Web).
[30] K. Weick, K. Sutcliffe, Managing the Unexpected: Assuring High Performance in an Age of Complexity, Jossey-Bass Publishers, 2001.
[31] Information systems for emergency management, in: B. Van de Walle, M. Turoff, S.R. Hiltz (Eds.), Advances in Management Information Systems monograph
series (Editor-in-Chief: Vladimir Zwass), M.E. Sharpe Inc, Armonk, NY, 2010, Chapter 16.
[32] C. Lindblom, The science of muddling through, Public Administration Review (1959) 79–88.
[33] C. Lindblom, Still muddling, not yet through, Public Administration Review (November/December 1979) 517–526.
[34] P.G. Goldschmidt, Scientific Inquiry or Political Critique: Review of the Sackman Report, Technological Forecasting and Social Change 7 (1972) 195–213.
[35] New York Times, August 17, 2010, p. D-2.
[36] New York Times, August 24, 2010.

Harold Linstone is University Professor Emeritus at Portland State University and founding editor-in-chief of this Journal. His most recent book is Decision Making
for Technology Executives: Using Multiple Perspectives to Improve Performance (Artech House, 1999). Prior to his academic position he was Associate Director of
Corporate Planning—Systems Analysis at the Lockheed Corporation. His M.A. and Ph.D. degrees are in Mathematics.

Murray Turoff is a Distinguished Professor Emeritus at the New Jersey Institute of Technology. He is a co editor of a recent book on Emergency Management
Information Systems (M.E. SharpE 2010). Besides his early and continuing work with the Delphi Method, he spent most of academic research career in the design
and evaluation of Computer Mediated Communication systems. After 9/11 he turned his attention back to his early work in Emergency Management and in 2004,
he was a cofounder of the international organization ISCRAM (Information Systems for Crisis Response and Management).

Please cite this article as: H.A. Linstone, M. Turoff, Delphi: A brief look backward and forward, Technol. Forecast. Soc. Change
(2010), doi:10.1016/j.techfore.2010.09.011