The current issue and full text archive of this journal is available on Emerald Insight at:

https://www.emerald.com/insight/0022-0418.htm

Review of the Delphi method in Review of the

Delphi method
library and information
science research
Brady D. Lund 929
Emporia State University, Emporia, Kansas, USA
Received 12 September 2019
Revised 8 November 2019
Abstract 21 January 2020
26 January 2020
Purpose – This article presents an introduction to the Delphi method and review of Delphi studies published in Accepted 30 January 2020
the literature of library and information science (LIS).
Design/methodology/approach – A review of Delphi studies published between the years of 1971 and 2019
is performed, using studies retrieved from the Library and Information Science Source database. A total of 122
articles were retrieved and evaluated based on the population studied, means of identifying experts, number of
participants for each study round, type of Delphi, and type of findings.
Findings – General librarians (any type), academic librarians, and information science researchers are the
most common populations in LIS Delphi studies. On average (middle 50 percent of studies), 14–36 experts are
used in the first round of LIS Delphi studies (median n 5 23). Employment in a specific role and publications in
scholarly journals are the most common means of identifying experts. Variants of the e-Delphi (online survey/
email) method are increasingly common, particularly in LIS Delphi studies that focus on general information
science, rather than library, topics. Though LIS Delphi studies are relatively few in number, they have a
consistent record of being published in some of the most prestigious LIS journals.
Originality/value – This paper provides an introduction to the Delphi method for LIS research and presents
an overview of existing literature in LIS that utilizes the research method. No overview of this extent exists in
the LIS literature, and, thus, this paper may serve as an important information source about the method for LIS
researchers.
Keywords Review, Research methods, Delphi, Delphi method, Information studies, Information research,
Library and information science
Paper type General review

Though the Delphi method has existed as a legitimate research method for over five decades,
there is limited knowledge of this method among library and information science (LIS)
researchers and few LIS scholarly publications that employ the method. Little is known of
how the method has historically been used in LIS publications and how this knowledge may
guide future LIS Delphi studies. The purpose of the following review article is to present an
overview of the major features of LIS Delphi studies – populations used, types of Delphi used,
types of findings retrieved – in order to both encourage and provide precedent and guidance
for future LIS Delphi research.

Literature review
History of the Delphi method
The Delphi method was developed by the RAND Corporation in the 1950s and 1960s as a
means to prepare for national security threats during the Cold War (Woudenberg, 1991).
RAND gathered several top foreign policy and national security experts across the United
States and utilized a new methodology of consensus-gathering, which they believed would
mitigate the challenges presented from a single member of a focus group dominating the
Journal of Documentation
Vol. 76 No. 4, 2020
pp. 929-960
Author would like to acknowledge the support of the Research Faculty at Emporia State University's © Emerald Publishing Limited
0022-0418
School of Library and Information Management. DOI 10.1108/JD-09-2019-0178
JD conversation and the vast disparity in responses received during individual interviews
76,4 (Dalkey and Helmer, 1963; Helmer, 1967). The researchers quickly recognized that this
method held potential value for the study of consensus among other social groups, like
improving the diffusion of educational innovations (Helmer, 1966), students (Dalkey, 1969),
and marketing (Jolson and Rossow, 1971). At this point in time, however, the method was still
only used within the RAND Corporation; few academics had adopted it as a legitimate
research method (Pill, 1971).
930 Adoption of the Delphi method as a legitimate research method in academia began during
the early 1970s, primarily in the fields of management and marketing, where collecting the
focused opinions of a large group of respondents was shown to have potential benefit for
developing advertising campaigns (Jolson and Rossow, 1971). Other areas where researchers
began to experiment with the method include medical science (Milholland et al., 1973; Travis,
1973), information systems (Turoff, 1971; Grabbe and Pyke, 1972), and computer and
information science (Firschein et al., 1973; Nanus et al., 1973). One interesting aspect of the
Delphi method – being a method that is often used to forecast the future – is that its usefulness
can, in a sense, be examined by reevaluating these early studies conducted over four decades
ago. Generally, these studies show mixed results. For instance, in Firschein et al.’s study of
artificial intelligence, there were correct predictions about the development of search engines,
industrial robotics, and diagnostics for machinery during the 1980s and 1990s, but overly
optimistic responses in regard to when robot tutor (1988), automatic language translator
(1995), computer psychiatrist (2000), robot chauffeur (2000), and animal symbionts (2010)
would be commercially available.
A quintessential methodological book for Delphi is Linstone and Turoff’s (1975) The
Delphi method: Techniques and applications. This book was a major step in the legitimization
of Delphi as a scholarly research method. It outlined the philosophy, aim, and steps in the
Delphi method. Following the publication of this book (which has been cited nearly 10,000
times, according to the Web of Science database), research involving the Delphi method has
continued to grow substantially (Rowe and Wright, 2011). Today, the method is used in
virtually all disciplines, and is considered to have an acceptable level of validity for
performing scholarly research (Landeta, 2006; Lilja et al., 2011; Tomasik, 2010; Worrell et al.,
2013). The basic elements of the Delphi method, as described by Dalkey and Helmer during its
early development (1963, p. 458), include (1) repeated individual questioning of a panel of
experts, (2) anonymity among the panel of experts, and (3) interspersed opinion feedback. As
the method emerged from the RAND Corporation and was adopted by an increasingly large
group of researchers, the method became more formalized, and variations began to emerge.

Description and philosophy of the Delphi method

As noted by Aichholzer (2009, p. 253), “a major justification for the validity of the Delphi
method derives from the ‘theory of errors,’ according to which the aggregated group
responses can be expected to represent a statement that is superior to the majority of the
individual experts’ ones.” Another way to state this theory is Aristotle’s adage, “the whole is
greater than the sum of its parts.” Even experts may misjudge what aspects of a topic are
important, or have ideas that are not “the best,” but when considered as a collective, the best
ideas tend to be elevated.
According to Linstone and Turoff (1975), the Delphi method can be viewed as a structured
communication process from which the intended outcome is a consensus to a complex (even
ambiguous) problem (p. 3). Participants in a Delphi study are traditionally selected based on
some level of expertise relevant to the research questions of the study. The Delphi method
itself is “an iterative process to collect and distill the anonymous judgements of experts using
a series of data collection and analysis techniques interspersed with feedback” (Skulmoski
et al., 2007, p. 1). In the ideal administration of the Delphi method, this process results in a Review of the
consensus among the respondents, whereby in the last iteration, only very precise and Delphi method
popular feedback is collected, giving a clear, expert response to the research questions. A full
depiction of the Delphi process is shown in Figure 1.
As noted by Hasson and Keeney (2011), there are at least ten commonly used types of
Delphi designs: classical, used to forecast and gather opinion; modified, similar to classical, but
generally fewer rounds; decision, used to inform immediate decision-making; policy, used to
guide policy development; real-time, similar to focus group, but with tabulation of results after 931
each round; e-Delphi, a Delphi study facilitated by email or online survey; technological, real-
time online Delphi, similar to real-time; online, Delphi performed in the setting of a chat room or
discussion board; argument, where participants are purposely selected to represent opposite
sides of an issue; and disaggregative policy, where participants are asked to speculate on
future scenarios. Many other less-common types and variations of the Delphi method exist.

Selection of experts
Careful selection of experts for the Delphi study is immensely important. These individuals
should not be selected based on relationships to the researcher (Avella, 2016; Hasson et al., 2000).
Nor should they necessarily be the most visible members of a particular field (e.g. in a study
about library instruction, the former presidents of ALA or CILIP are not necessarily experts,
even though they are visible figures within the field of librarianship). Rather, the experts in a
Delphi study should be those individuals with the most intimate knowledge and experience with
the topic (Baker et al., 2006; Welty, 1972). In a study of library instruction, this likely means
actual library instructors. These individuals, even though they may not conduct substantial
research or publicly speak about or represent library instruction, work as instructional
librarians every day, which gives them an insider’s perspective on the topic. Often, these
individuals are identified based on their contributions to the literature within a field (Gordon,
1994). For a study of instructional librarianship topics, the researcher may conduct a literature
search for contributions on this topic and refer to top journals in this field (e.g. College and
Research Libraries, Journal of Academic Librarianship), and identify a pool of potential
participants to contact based on the authors making significant contributions to the field.

Assumptions for Delphi research

Assumptions made by researchers who use the Delphi method include that the ideas
generated by experts are necessarily the best ideas, that the researcher can conduct this
research without introducing problematic bias into the iterative process, and that sufficient
response rate can be attained for this type of study (Hsu and Sandford, 2007). There is some
reason to believe experts may not necessarily produce the best new ideas (Welty, 1972). As
found in industry and entertainment, often an outsider’s perspective can actually produce the
most useful new ideas. However, identifying potential outsiders that would be capable of

1. Identify Research Problem

2. Develop Research Questions
3. Select the “Experts” (Participants in the Study)
4. Administer the First Round of the Study (Questionnaire)
5. Synthesize Responses from First Round of the Study
6. Present Respondents with the Synthesized Responses
7. Administer the Second Round of the Study
8. Synthesize Responses from Second Round of the Study
9. Present Respondents with the Synthesized Responses Figure 1.
10. Continue this Process Until a Pre-Prescribed Number of Rounds Has Steps in the classical
Been Reached, or the Respondents Appear to Have Reached a Consensus Delphi method
11. Analyze the Responses from Each Round; Disseminate Findings!
JD providing useful feedback is much more difficult than identifying the experts in a given area.
76,4 The reader of a study that utilizes Delphi should be comfortable that the researcher faithfully
collated the responses in an anonymous and random order so as to reduce bias. The number
of research participants for the first round of the Delphi should be quite large (particularly if
the questionnaire will be distributed online or via mail). This is due to a high level of attrition
that often occurs between iterations of the method as respondents lose interest in continually
completing the same questionnaire.
932
Strengths and weaknesses of the Delphi method
Those who elect to use the Delphi method in their research are generally seeking to identify
some ideal (as in, policy formation) or prediction, in other words, blaze new territory rather
than work within an area that is already well-defined. Delphi mitigates major weaknesses in
survey, interview, and focus groups methods: no one research subject is able to dominate the
conversation, as could be done in a focus group, and (in the ideal Delphi) responses that are less
popular (not shared by many members of the group, are likely to be eliminated in the iterative
process, as opposed to receiving equal weight in a survey or interview format). The limitations
of Delphi arise from that same iterative process. There are many instances where researchers
may not want to eliminate responses that are less popular. These less popular responses can be
important pieces of data that, while less commonly occurring, still represent the responses/
feelings of some of the respondents. For this reason, Delphi may be less advisable for creating
inventories of respondents’ thoughts on a topic; investigating personal experiences and
emotions; exploring the relationship between individuals and their cultures; producing new
research on a topic/idea that has already been thoroughly investigated in the literature; and so
forth (Donohoe and Needham, 2009; Keeney et al., 2001). It is not impossible to use Delphi for
these research purposes, but other methods may produce more robust results.

Prior investigations of the Delphi method in LIS literature

Several LIS publications have provided some introduction or overview of the Delph method
in LIS research; however, these publications each have several limitations. Fischer (1978)
provided a general overview to the Delphi method (when it was still a relatively young
method). At the time, the author was critical toward the potential of the method for the study
of LIS topics. Buckley (1995) provided another effort to philosophize about the use of the
Delphi method in LIS research. His article presents an introduction to the method similar to in
the preceding sections of this article, but does not review prior LIS research using the Delphi
method, instead simply offering the author’s opinion of how the Delphi could best be used in
LIS (“better for collecting opinion than for forecasts”).
Recently, several studies have provided short reviews of the Delphi method as part of a
publication proposing a new variant/type of the method. This is the case in the studies of Ju
and Jin (2013) and Poirer and Robinson (2014). In the Ju and Jin (2013) article, a short review of
LIS Delphi studies (published between the years of 1971 and 2011) is presented, which
includes a discussion of typical sample size and study purpose. A total of 87 studies were
examined. This review, while useful at the time of its publication, is outdated and has several
gaps (e.g. it does not report in any detail what types of Delphi were used in these studies).
Poirer and Robinson (2014) provide a much shorter and limited review of Delphi studies while
introducing their variant, known as the Slow Delphi.
Chu (2015) reviewed research methods in three major LIS journals (Journal of
Documentation, Journal of the Association for Information Science and Technology, and
Library and Information Science Research) over the years 2001–2010. Among these three
journals and in ten years, the Delphi method was used as a research method in only three
published studies. This suggests that, as of the year 2010, the Delphi method was a relatively
uncommon method in major LIS research publications. As such, a detailed introduction of Review of the
how the method has been and may be used in LIS research may be beneficial to future Delphi method
expansion in the use of the method.

Research purpose and questions

The purpose of this study is to review the use of the Delphi method in LIS. Although a few
studies in the past have discussed the Delphi method and some of the seminal LIS studies to 933
use it (Ju and Jin, 2013; Poirer and Robinson, 2014), this review is more comprehensive in that
it evaluates study populations, how “experts” are identified, the number of experts used in
these studies, the type of Delphi used, and the type of findings retrieved for all LIS Delphi
studies published since the advent of the method. No existing review of the Delphi method in
LIS provides this current and extensive of a review of use of this method. This review offers
detailed guidance for future Delphi studies in LIS based on the structure of past studies.
There is one central research question that guides this review:
(1) What are the common attributes of LIS studies that employ the Delphi method?
This central question is supported by several sub-questions:
(1) What are the most common populations used in LIS Delphi studies?
(2) What are the most common methods for selecting “experts” for LIS Delphi studies?
(3) What are the typical numbers of experts participating in LIS Delphi studies?
(4) What are the most common types/variations of Delphi method used in LIS Delphi
studies?
(5) What are the most common types of findings reported in LIS Delphi studies?

Method
A review of LIS research studies that have utilized the Delphi method was performed. The
population studied, means of identifying experts, number of participants for each study
round, type of Delphi, and type of findings were identified by the researcher in consultation
with an experienced Delphi researcher. A total of 122 studies were identified for the years of
1971–2019, by querying the Library and Information Science Source database for “Delphi
studies,” “Delphi method,” and “Delphi” AND “Research,” for peer-reviewed academic journal
articles and then vetting these articles for relevance to LIS research and actual usage (rather
than mention only in the literature review) of the Delphi method.
The findings for each LIS Delphi study are included in the appendix to this article. As
shown in Figure 2, the number of LIS Delphi studies published has increased significantly
since 2000, and particularly in the last decade. This finding supports the notion that Ju and
Jin’s (2013) article, which reported only through the year 2011, is outdated, and a new review
of the Delphi method is warranted.

Findings
As shown in Table I, the top population for the LIS Delphi studies are general librarians.
This population name indicates that librarians were selected from a variety of library types
(e.g. academic and public). Often, this population is selected when researchers are
examining concerns or trends in a particularly geographic region; for instance, Koskiala
and Huhtanen’s (1989) forecast for Finnish libraries. Just behind general librarians are
academic librarians and information science researchers. Information science researchers
JD
76,4
120

100

934 80

60

40

20
Figure 2.
Distribution of LIS
Delphi studies 0
over time 1970 1980 1990 2000 2010 2020
Year

Population Frequency

Librarians–general 19
Information science researchers 18
Librarians–academic 18
Information technology managers 5
Medical professionals 5
Librarians–school 5
Librarians–public 5
Table I. Librarians–medical 5
Top populations for Information technology professionals 5
LIS Delphi studies LIS Educators 4

were often the population of choice when examining general topics (not contextualized) in
information science, for instance, Poirier and Robinson’s (2014) study of slow movement
principles in information behavior research. These three top populations were far more
common than any others.
Table II displays the top methods through which participants are selected/experts are
identified for LIS Delphi studies. The most frequent method is through employment, that is,
employment in a specific occupation (e.g. cataloging librarian) in a particular institution (e.g.
academic library selected by the researchers for inclusion in the study) signifies “expertise” in
the subject of the study (e.g. challenges for academic cataloging librarians). Scholarly
publications is the second most common method for the identification of experts (e.g.
publication of a scholarly article on FRBR signifies expertise in FRBR), followed by
membership in professional organizations (e.g. a committee role in CILIP or ALA indicates a
level of expertise in general librarianship topics). Other methods of identifying experts
include through the researchers’ professional network (assuming the researchers themselves
are experts, this may be an effective means for identifying other experts), award winners, and
editorial board members for journals in the relevant field of study. Less common, and
potentially more problematic, methods for selecting experts include posting on ListServ/
mailing lists, posting or selecting based on profiles on social media (e.g. Twitter/LinkedIn), Review of the
and selecting based on proximity (e.g. individuals employed at nearby relevant institutions). Delphi method
In Table III the methods for identifying experts are broken down by whether the study
focuses primarily on library-related topics or general information science topics. Among
library literature, it is much more common for employment or membership/leadership role in
a professional organization to represent “expertise,” while, among information science
literature, scholarly publications are a much more common method of identifying “expertise.”
Information science studies were also less likely to use the potentially problematic methods of 935
recruiting experts (proximity, ListServ/Social Media posting) than library studies.
Tables IV and V display findings regarding the most common types/variations of the
Delphi method used in LIS studies. Table III shows the results for all studies (all-time), while
Table IV shows results only since 2000 (i.e. after the emergence of the e-Delphi) divided by
studies that mostly examine library-related topics versus studies that mostly examine
general information science topics. Evident among both tables is that the e-Delphi is more
popular than the traditional (mail or in-person) Delphi study. The classical method (full three
rounds, used to forecast or gather opinion) is the most common, while the modified method

Population Frequency

Employment 48
Scholarly publications 34
Professional organization 19
Network 7
Listserv 5
Proximity 5
Award winner 3 Table II.
Editorial board members 2 Top participant
Student 2 selection methods for
Social network 2 LIS Delphi studies

Library literature Frequency Info science literature Frequency

Table III.
Employment 25 Employment 20 Comparison library
Scholarly publications 13 Scholarly Ppublications 19 and information
Professional organizations 11 Network 3 literature - how
Proximity 5 Professional organizations 2 participants were
Listserv 4 Award winner 2 selected

Population Frequency

Classical e-Delphi 30
Modified e-Delphi 23
Classical Delphi 17
Policy e-Delphi 9
Modified Delphi 9
Critical e-Delphi 7
Policy Delphi 4
Critical Delphi 2 Table IV.
Grounded Delphi 2 Top Delphi type for LIS
Online Delphi 2 Delphi studies
JD (two rounds and/or unique forms of analysis/distribution of Delphi) is the second most
76,4 common. Among general information science studies, the variations in Delphi type is much
greater (15 unique variations since 2000) than library studies (10 unique variations); greater
innovation among variations of the Delphi method is observed in general information science
studies, such as the use of the Critical Delphi, Grounded Delphi, and Slow Delphi.
Top types of findings for LIS Delphi studies are shown in Table VI. By far the most
common type of finding was a model/framework/action plan of a topic. This is true both
936 among library and general information studies. The second most frequent type of finding
was a forecast; however, a forecast was largely the primary finding in only library studies,
while the development of a policy or taxonomy was much more common among general
information studies.
Shown in Table VII are the top journals for the publication of LIS Delphi studies. These
studies were published in 65 journals, all but three of which were journals dedicated to LIS
topics. Of the top ten studies with the most Delphi publications, seven are included in the
Social Science Citation Index; these seven journals have an average Impact Factor of 2.4.
Finally, Table VIII and Figure 3 display findings related to the number of participants/
experts recruited for LIS Delphi studies. In table, a number of descriptive statistics are
presented, including the mean, median, first and third quartiles, and minimum and maximum.
Based on the variance between the mean and median values, there is an evident right skew in
the data (there are a few Delphi studies that used an extremely large number of participants,
relative to the typical study). In general, the attrition from round one to three appears to be
relatively minimal; as such, this may indicate a problem with a study should a large amount of
attrition occur in a study. Of note for the minimum values (three for all rounds), there were two
of this type, one which the researchers completed all rounds and reported the results and one
which the Delphi was deemed “Failed” and discontinued after the first round.
Figure 3 displays the trend in median number of Delphi experts in LIS studies over time.
Two trends may be readily observed from this figure. First, the amount of attrition over the
course of the Delphi study is much smaller/more stable as time has progressed. In the 2010s
decade, virtually no attrition is observed over the course of the three rounds. Second, the

Type of Delphi (libraries) Frequency Type of Delphi (info) Frequency

Classical e Delphi 17 Classical e-Delphi 12

Table V. Modified e-Delphi 14 Modified e-Delphi 9
Top Delphi types since Classical Delphi 5 Policy e-Delphi 6
2000, for library vs info Modified Delphi 4 Critical e-Delphi 5
studies Policy e-Delphi 3 Classical Delphi 3

Population Frequency Frequency libraries Frequency info

Framework 45 24 21
Forecast 14 12 2
Taxonomy 10 4 6
Opinion 10 6 4
Policy 10 3 7
Opportunities 10 7 3
Concerns 8 4 4
Table VI. Curriculum 6 5 1
Top types of findings Themes 5 3 2
for LIS Delphi studies Tool 4 1 3
Population Frequency
Review of the
Delphi method
Journal of the Association for Information Science and Technology 9
Journal of Medical Internet Research 8
Electronic Library 7
Information Systems Management 6
Information Research 5
Journal of Education for Library and Information Science 5 937
Library Management 4
International Journal of Information Management 4 Table VII.
Library and Information Research 3 Top journals for LIS
Journal of Librarianship and Information Science 3 Delphi studies

Round Mean Median 1st quartile 3rd quartile Minimum Maximum

1 38 23 14 36 3 431 Table VIII.

2 32 20 13 32 3 270 Number of participants
3 27 20 13 32 3 115 for LIS Delphi studies

90

80

70

60

50

40

30

20
Figure 3.
10 Median number of
experts used in LIS
0 Delphi studies by
1970 1980 1990 2000 2010
decade and round
First Round Second Round Third Round

number of experts used in Delphi studies has decreased over time, though it has remained
relatively stable since 1990. Early Delphi studies averaged 45–80 experts and were quite
unstable (often additional experts were invited between the second and third round to
account for attrition). Over the past three decades, the average number of experts has fallen
between 20 and 35.

Discussion
Among the top populations, part of the reason for the high frequency of general and academic
librarians is the dominance of these populations in early LIS Delphi studies. In fact, 17 of the
earliest 25 LIS Delphi studies utilized a librarians/LIS educator population, including the
earliest study, Belzer et.al (1971) study of LIS curriculum. The second LIS Delphi study to be
published, Simsova (1975) utilized the general librarian population in a study of comparative
librarianship practices. Among the 25 most recent LIS Delphi studies, only 11 have librarians
as the study population. These include seven of the eight LIS Delphi studies published in 2019.
While the ideal number of experts for a Delphi study cannot be identified from this
examination alone, the fact that all of the studies included in this examination were accepted
JD for publication indicates that, at least for certain circumstances and certain journals, any of
76,4 these numbers may be appropriate. However, the optimal range for number of exerts likely
falls between the first and thirds quartiles – perhaps the median value of 23 for the first round
would be an optimum minimum for which to aim.
Variations of the e-Delphi are the most common types of Delphi in LIS studies. Though the
first e-Delphi in LIS was not published until the mid-1990s, with Westbrook (1997), these
studies have been published at a rate of over three per year since that time. Among the unique
938 variants of the Delphi introduced within LIS literature are the Ground Delphi (Howard, 2018),
the Slow Delphi (Poirer and Robinson, 2014), a Fuzzy (Statistical) Delphi (B€ uy€
uk€ozkan, 2004),
and the Serious Leisure Delphi (Price and Robinson, 2017). These studies extend the use of the
Delphi into new territory, particularly the Slow Delphi and Serious Leisure Delphi, which
utilize Delphi to investigate information behavior. This may be an important innovation for
that field of study, which has traditionally utilized standard surveys or interviews as a
research method (Julien et al., 2011). As noted in the results section, LIS appears to be a
receptive discipline for these Delphi variants, with these studies published in some of the
discipline’s top journals.
A common concern early in the history of the Delphi was its validity and acceptance by
major journal publishers. This does not appear to be a legitimate concern for modern LIS
Delphi studies. The majority of journals that have published Delphi articles are included in
the Social Science Citation Index, and several others, like the Journal for Education in Library
and Information Science, are highly regarded by researchers. All of the journals that
published the most (top-ten) Delphi articles are prominent publications that would be known
by most LIS researchers.
Future LIS research and the Delphi method
There are many potentially excellent uses of the Delphi method in LIS research today.
Capitalizing on the strengths of the Delphi method to form consensus and develop policy
concepts, Delphi may be used to investigate areas where significant ambiguity or deviation
occurs. Terminology to describe concepts in LIS varies widely based on nationality,
geographic region, where one receives their LIS education, and the type of organization in
which they work, among other factors. Delphi may be used to study the extent to which a
more consistent lexicon could be developed for the discipline.
Library policy, on both micro and macro levels, could also be investigated using Delphi
methods. Library policies, particularly on issues such as privacy, tend to vary significantly
from organization to organization. Such studies, however, would likely need to be conducted
with care, as some library policies vary significantly because this is what is needed for each
library’s specific context.
There is also ample opportunity to expand upon prior Delphi studies within LIS and
related disciplines. For instance, the Vance et al. (1977) study, now 45 years old, could
be re-administered in much the same way as the original to investigate the outlook of
current LIS educators toward the future of the discipline and profession; given the
removal of geographic barriers via the internet, it would also be much easier to
administer this study to a global population of LIS educators. The same could be said
of Feret and Marcinek’s (1999) study, which only forecasted five years into the future.
This could be expanded to a larger time-span, such as 15 or 25 years, to provide more
relevant findings. These are but a few examples of areas in which Delphi could be used
to investigate important issues in LIS.

Limitations of Delphi
A summary of positives and negatives for the use of the Delphi method in LIS research is
displayed in Figure 4.
 The Delphi method is powerful in terms of consensus formation for policy development or Review of the
future forecasting. It overcomes some of the weaknesses of other research methods, like the Delphi method
potential for conversation dominance/power differential in focus groups and equal weighting
of all ideas in surveys and interviews. By using the iterative process, participants in the
Delphi process are able to recognize whether their ideas align with that with the larger group
and contemplate adjusting their responses.
However, there are several limitations to the Delphi method. Participant attrition is a
major challenge with Delphi. For instance, in the Vance et al. (1977) study, an initial panel of 939
300 individuals was identified; of this panel of 300, a total of 180 participated in the first round
of the Delphi, 118 in the second round, and only 90 in the third round. So less than one-third of
the individuals initially identified for participation completed the entire Delphi process. In
Schmidt et al.’s (2001) study, which has been cited over 1,000 times in the last two decades, the
initial participant group was only 36, and only 19 continued to the ranking (second phase)
portion of the study. In general, this attrition is not considered to significantly harm the
validity of the research study (as indicated by the popularity of the Schmidt study), but it can
be problematic if the researchers would like to perform a large number of iterations and the
respondent group becomes significantly lower than the initial respondent group. As occurred
in the Townsend et al. (2016) study, the researchers in a Delphi study may realize that an
additional iteration is needed only long (three months in the case of Townsend) after the
previous iterations, resulting in fatigue and attrition of participants.
Another major limitation of Delphi is tied to the method’s assumptions. The method makes
three potentially problematic assumptions: that it is possible to identify individuals who can be
considered “experts” on a subject, that “experts” always have the best ideas, and that the most
popular ideas are the best ideas. Arguably, experts are not the likely sources of the best ideas.
For instance, the most remarkable ideas to come from great social thinkers like Everett Rogers
and Thomas Kuhn came during their early career, before they would have necessarily been
considered “experts.” Many of the modern information infrastructures and services (Apple,
Facebook, Google) were created by individuals while in their early 20s – not experts in
computer and information technology. However, it is much more difficult to identify a Kuhn or
Zuckerberg before they emerge than it is to recruit them after they have put their good ideas
into action. Similarly, the best ideas are often not the most popular ideas. The ideas of the

Positives:

Time-tested method used across virtually all disciplines

Strong method for consensus building, policy formation, and
forecasting
Flexible method, like survey, that does not require presence of
all participants at the same time or same physical location
Can be used to build practical strategies as well as theory-
building

Negatives:

Lesser-known method that may be difficult to explain to

participants
Participant attrition of the course of several iterations (takes
longer time commitment than survey/interview methods)
Researcher Bias/Error Figure 4.
Positives and
Problematic selection of participants
negatives of the Delphi
Problematic collation of responses by the researcher(s) method in LIS research
Poorly developed questions
JD researchers and innovators mentioned above were highly criticized when first disseminated.
76,4 However, it is again difficult to determine which few unpopular ideas out of the thousands that
are proposed will become “the best.” It is much easier to focus on popular ideas. This is not
necessarily problematic from an analytical standpoint, so long as the researcher does not make
the claim that the ideas retrieved from the Delphi are the best ideas.

Conclusion
940 While this study cannot make definitive statements about the ideal number of experts for a
Delphi study or what type of Delphi is best, it does indicate several trends about what published
studies in high-quality peer-reviewed LIS journals suggest are appropriate. For instance, based
on these studies, it is likely that a selection of 14–36 experts for the first round of the Delphi will
be appropriate for the production of publishable research, with the ideal group laying around
the median value of 23. Based on the large number of both classical and modified e-Delphi
studies published in LIS literature, it is likely that either type may be appropriate, given
sufficient justification by the researchers; additionally, LIS journals – and particularly IS
journals – appear particularly receptive to unique adaptations/variations of the Delphi method
(again, given sufficient justification, such as that in Poirer and Robinson’s 2014 study).
Given the findings for LIS Delphi studies, two methods for “expert” identification seem
most appropriate: employment in a relevant position in a relevant institution to the research
questions, and scholarly publications on topics relevant to the research questions. Some less
common, and more questionable, methods include recruiting through a ListServ or social
media, or recruiting locally available individuals (both of which should probably be
discouraged). As for the populations of the experts represent, it does not appear that one is
more appropriate than any other, though the most common populations for LIS Delphi
studies are general librarians, information science research, and academic librarians.
Successfully executed Delphi studies may be published in a number of high-quality LIS
journals, particularly for those studies that examine general information science topics. While
the Delphi method is not a common research method for LIS studies (only about 2 studies on
average per year since the inception of the method), these findings indicate several promising
avenues for Delphi studies, granted that the method is executed properly. LIS researchers may
benefit from investigating and executing this method more frequently, particularly when
developing frameworks, taxonomies, or forecasting the future of professions or academic fields.

References
Aichholzer, G. (2009), “The Delphi method: eliciting experts’ knowledge in technology foresight”, in
Bogner, A., Littig, B. and Menz, W., Interviewing Experts, Palgrave MacMillan, London,
pp. 252-274.
Avella, J.R. (2016), “Delphi panels: research design, procedures, advantages, and challenges”,
International Journal of Doctoral Studies, Vol. 11, pp. 305-321.
Baker, J., Lovell, K. and Harris, N. (2006), “How expert are the experts? An exploration of ‘expert’
within Delphi panel techniques”, Nurse Researcher, Vol. 14 No. 1, pp. 59-71.
Belzer, J., Issac, A., Finkelstein, E. and Williams, J. (1971), “Curricula in information science: Analysis
and development”, Journal of the American Society for Information Science, Vol. 22 No. 3,
pp. 193-223.
Buckley, C. (1995), “Delphi: a methodology for preferences more than predictions”, Library
Management, Vol. 16 No. 7, pp. 16-19.
Chu, H. (2015), “Research methods in library and information science: a content analysis”, Library &
Information Science Research, Vol. 37, pp. 36-41.
Dalkey, N.C. (1969), The Delphi Method: An Experimental Study of Group Opinion, RAND Corporation,
Santa Monica, CA.
Dalkey, N.C. and Helmer, O. (1963), “An experimental application of the Delphi method to the use of Review of the
experts”, Management Science, Vol. 9 No. 3, pp. 458-467.
Delphi method
Donohoe, H.M. and Needham, R.D. (2009), “Moving best practice forward: Delphi characteristics,
advantages, potential problems, and solutions”, International Journal of Tourism Research,
Vol. 11 No. 5, pp. 415-437.
Feret, B. and Marcinek, M. (1999), “The future of the academic library and the academic librarian: a
Delphi study”, Librarian Career Development, Vol. 7 No. 10, pp. 91-107.
941
Firschein, O., Fischler, M.A., Coles, L.S. and Tenenbaum, J.M. (1973), “Forecasting and assessing the
impact of artificial intelligence on society”, Proceedings of the International Joint Conference on
Artificial Intelligence, Morgan Kaufman, San Francisco, CA, pp. 105-120.
Fischer, R.G. (1978), “The Delphi method: a description, review and criticism”, The Journal of
Academic Librarianship, Vol. 4 No. 2, pp. 64-70.
Gordon, T.J. (1994), “The Delphi method”, Futures Research Methodology, Vol. 2 No. 3, pp. 1-30.
Grabbe, E.M. and Pyke, D.L. (1972), “An evaluation of the forecasting of information processing
technology and applications”, Technology Forecasting and Social Change, Vol. 4 No. 2, pp. 143-150.
Hasson, F. and Keeney, S. (2011), “Enhancing rigour in the Delphi technique research”, Technological
Forecasting and Social Change, Vol. 78 No. 9, pp. 1695-1704.
Hasson, F., Keeney, S. and McKenna, H. (2000), “Research guidelines for the Delphi survey technique”,
Journal of Advanced Nursing, Vol. 32 No. 4, pp. 1008-1015.
Helmer, O. (1966), The Use of the Delphi Technique in Problems of Educational Innovations, RAND
Corporation, Santa Monica, CA.
Helmer, O. (1967), Analysis of the Future: The Delphi Method, RAND Corporation, Santa Monica, CA.
Howard, K.J. (2018), “Emergence of a new method: The grounded Delphi method”, Library and
Information Research, Vol. 42 No. 126, pp. 5-31.
Hsu, C. and Sandford, B.A. (2007), “The Delphi technique: making sense of consensus”, Practical
Assessment, Research and Evaluation, Vol. 12 No. 10, pp. 1-8.
Jolson, M.A. and Rossow, G.L. (1971), “The Delphi process in marketing decision making”, Journal of
Marketing Research, Vol. 8 No. 4, pp. 443-448.
Ju, B. and Jin, T. (2013), “Incorporating nonparametric statistics into Delphi studies in library and
information science”, Information Research, Vol. 18 No. 3, available at: http://informationr.net/ir/
18-3/paper589.html.
Julien, H., Pecoskie, J.L. and Reed, K. (2011), “Trends in information behavior research, 1999-2008: a
content analysis”, Library & Information Science Research, Vol. 33, pp. 19-24.
Keeney, S., Hasson, F. and McKenna, H.P. (2001), “A critical review of the Delphi technique as a research
methodology for nursing”, International Journal of Nursing Studies, Vol. 38 No. 2, pp. 195-200.
Koskiala, S. and Huhtanen, A. (1989), “The Finnish Delphi study: Forecasting the extent of
information technology use in libraries in 1996 and 2010”, The Electronic Library, Vol. 7 No. 3,
pp. 170-175.
Landeta, J. (2006), “Current validity of the Delphi method in social sciences”, Technological Forecasting
and Social Change, Vol. 73 No. 5, pp. 467-482.
Lilja, K.K., Laakso, K. and Palomaki, J. (2011), “Using the Delphi method”, 2011 Proceedings of
PICMET: Technology Management in the Energy Smart World, Institute of Electrical and
Electronics Engineers, Piscataway, NJ.
Linstone, H.A. and Turoff, M. (1975), The Delphi Method: Techniques and Applications, Addison-
Wesley, Reading, MA.
Milholland, A.V., Wheeler, S.G. and Heieck, J.J. (1973), “Medical assessment by a Delphi group opinion
technic”, New England Journal of Medicine, Vol. 288, pp. 1272-1275.
JD Nanus, B., Wooton, M. and Borko, H. (1973), “The social implications of the use of computers across
national boundaries”, Proceedings of the 1973 American Federation of Information Processing
76,4 Societies Conference, Association for Computing Machinery, New York, NY, pp. 735-745.
Pill, J. (1971), “The Delphi method: substance, context, a critique and an annotated bibliography”,
Socio-Economic Planning Sciences, Vol. 5 No. 1, pp. 57-71.
Poirier, E. and Robinson, L. (2014), “Slow Delphi: an investigation into information behavior and the
slow movement”, Journal of Information Science, Vol. 40 No. 1, pp. 88-96.
942
Price, L. and Robinson, L. (2017), “‘Being in a knowledge space’: information behavior of cult media
fan communities”, Journal of Information Science, Vol. 43 No. 5, pp. 649-664.
Rowe, G. and Wright, G. (2011), “The Delphi technique: past, present and future prospects”,
Technological Forecasting and Social Change, Vol. 78 No. 9, pp. 1487-1490.
Schmidt, R., Lyytinen, K., Keil, M. and Cule, P. (2001), “Identifying software project risks: an
international Delphi study”, Journal of Management Information Systems, Vol. 17 No. 4, pp. 5-36.
Simsova, S. (1975), “A Delphi survey of comparative librarianship”, International Library Review,
Vol. 7 No. 4, pp. 417-426.
Skulmoski, G.J., Hartman, F.T. and Krahn, J. (2007), “The Delphi method for graduate research”,
Journal of Information Technology Education: Research, Vol. 6 No. 1, pp. 1-21.
Tomasik, T. (2010), “Reliability and validity of the Delphi method in guideline development for family
physicians”, Quality in Primary Care, Vol. 18 No. 5, pp. 317-326.
Townsend, L., Hofer, A.R., Hanick, S.L. and Brunetti, K. (2016), “Identifying threshold concepts for
information literacy: a Delphi study”, Communication in Information Literacy, Vol. 10 No. 1,
pp. 23-49.
Travis, H.R. (1973), Identification of the Role of the Physician’s Assistant in Oregon Utilizing the Delphi
Technique (Doctoral Dissertation), Oregon State University, Corvallis, OR.
Turoff, M. (1971), “Delphi and its potential impact on information systems”, Proceedings of the 1971
American Federation of Information Processing Societies Conference, Association for
Computing Machinery, New York, NY, pp. 317-326.
Vance, K.E., Magrill, R.M. and Downen, T.W. (1977), “Future of library education: 1975 Delphi study”,
Journal of Education for Librarianship, Vol. 18 No. 1, pp. 3-17.
Welty, G. (1972), “Problems of selecting experts for Delphi exercises”, Academy of Management
Journal, Vol. 15 No. 1, pp. 121-124.
Westbrook, L. (1997), “Information access issues for interdisciplinary scholars: results of a Delphi
study on women’s studies research”, The Journal of Academic Librarianship, Vol. 23 No. 3,
pp. 211-216.
Worrell, J.L., Gangi, P.M. and Bush, A.A. (2013), “Exploring the use of the Delphi method in
accounting information systems research”, International Journal of Accounting Information
Systems, Vol. 14 No. 3, pp. 193-208.
Woudenberg, F. (1991), “An evaluation of Delphi”, Technological Forecasting and Social Change,
Vol. 40, pp. 131-150.

Further reading
Abukhader, S (2019), “A taxonomy of the expected roles of librarians towards knowledge management:
An eight-layer perspective for practice”, Library Management, Vol. 40 Nos 1/2, pp. 34-44.
Addison, T. (2003), “E-commerce project development risks: evidence from a Delphi survey”,
International Journal of Information Management, Vol. 23 No. 1, pp. 25-40.
Adhiarna, N., Hwang, Y.M., Park, M.J. and Rho, J.J. (2013), “An integrated framework for RFID
adoption and diffusion with a stage-scale-scope cubicle model: A case of Indonesia”,
International Journal of Information Management, Vol. 332, pp. 378-389.
Ali Al-Busaidi, K. (2014), “SWOT of social networking sites for group work in government Review of the
organizations: An exploratory Delphi study from IT managers’ perspective”, VINE: The journal
of information and knowledge management systems, Vol. 44 No. 1, pp. 121-139. Delphi method
Alexander, G.L., Powell, K., Deroche, C.B., Popejoy, L., Mosa, A.S.M., Koopman, R., . . . and Dougherty,
M. (2019), “Building consensus toward a national nursing home information technology maturity
model”, Journal of the American Medical Informatics Association, Vol. 26 No. 6, pp. 495-505.
Ambrosiadou, B.V. and Goulis, D.G. (1999), “The Delphi method as a consensus and knowledge
acquisition tool for the evaluation of the DIABETES system for insulin administration”, 943
Medical informatics and the internet in medicine, Vol. 24 No. 4, p. 257.
Anthopoulos, L.G. and Reddick, C.G. (2016), “Understanding electronic government research and
smart city: A framework and empirical evidence”, Information Polity, Vol. 21 No. 1,
pp. 99-117.
Arazy, O., Yeo, L. and Nov, O. (2013), “Stay on the Wikipedia task: When task-related disagreements
slip into personal and procedural conflicts”, Journal of the American Society for Information
Science and Technology, Vol. 64 No. 8, pp. 1634-1648.
Asonitis, S. and Kostagiolas, P.A. (2010), “An analytic hierarchy approach for intellectual capital:
Evidence for the Greek central public libraries”, Library Management, Vol. 31 No. 3,
pp. 145-161.
Baker, D. (2006), “Digital library futures: a UK HE and FE perspective”, Interlending and Document
Supply, Vol. 34 No. 1, pp. 4-8.
Bartlett, G., Vedel, I., Hong, Q.N., Pluye, P., Rousseau, M.-C., Fabregues, S., . . . and O’Cathain, A.
(2018), “The mixed methods Appraisal Tool (MMAT) version 2018 for information
professionals and researchers”, Education for Information, Vol. 34 No. 4, pp. 285-291.
Baruchson-Arbib, S. and Bronstein, J. (2002), “A view to the future of the library and information
science profession: A Delphi study”, Journal of the American society for Information Science and
Technology, Vol. 53 No. 5, pp. 397-408.
Battistella, C.C.B.U.I. and Nonino, F.F.N.U.I. (2012), “What drives collective innovation? Exploring the
system of drivers for motivations in open innovation, Web-based platforms, Information
Research, Vol. 17 No. 1, pp. 7-7, available at: https://emporiastate.idm.oclc.org/login?url5http://
search.ebscohost.com/login.aspx?direct5true&db5lls&AN576620584&site5ehost-live.
Boaz, M. (1978), “The future of library and information science education”, Journal of Education for
Librarianship, Vol. 18 No. 4, pp. 315-323.
Brancheau, J.C. and Wetherbe, J.C. (1987), “Key issues in information systems management”, MIS
Quarterly, Vol. 11 No. 1, pp. 23-45.
Bronstein, J. and Aharony, N. (2009), “Views and dreams: A Delphi investigation into Library 2.0
Applications”, Journal of Web Librarianship, Vol. 3 No. 2, pp. 89-109.
Brouwer, W., Oenema, A., Crutzen, R., De Nooijer, J., De Vries, N. and Brug, J. (2008), “An exploration
of factors related to dissemination of and exposure to internet-delivered behavior change
interventions aimed at adults: a Delphi study approach”, Journal of medical Internet research,
Vol. 10 No. 2, pp. e10.
Brunner, M., McGregor, D., Keep, M., Janssen, A., Spallek, H., Quinn, D., . . . and Solman, A. (2018), “An
eHealth capabilities framework for graduates and health professsionals: Mixed-methods study”,
Journal of medical Internet research, Vol. 20 No. 5, e10229.
Bush, G. and Jones, J.L. (2010),“Exploration to identify professional dispositions of school librarians: A
Delphi study”, School Library Media Research, Vol. 13, pp. 1-29.
B€ uk€ozkan, G. (2004), “Multi-criteria decision making for e-marketplace selection”, Internet Research,
uy€
Vol. 14 No. 2, pp. 139-154.
Cape, B. (2004), “Gathering opinion and initiating debate: the success of the Delphi Method in purely
qualitative research”, Library and Information Research, Vol. 28 No. 89, pp. 35-44.
JD Casselden, B., Pickard, A.J. and McLeod, J. (2015), “The challenges facing public libraries in the Big
Society: The role of volunteers, and the issues that surround their use in England”, Journal of
76,4 librarianship and information science, Vol. 47 No. 3, pp. 187-203.
Cegielski, C.G., Bourrie, D. and Hazen, B.T. (2013), “Evaluating adoption of emerging IT for corporate
IT strategy: Developing a model using a qualitative method”, Information Systems
Management, Vol. 30 No. 3, pp. 235-249.
Chavez-Hernandez, M.T. (1996), “The establishment of a library networking model for the Caribbean
944 region: a Delphi study”, Journal of Interlibrary Loan, Document Delivery and Electronic Reserves,
Vol. 7 No. 2, pp. 51-75.
Cherinet, Y.M. (2018), “Blended skills and future roles of librarians”, Library Management, Vol. 39
No. 1, pp. 93-105.
Chisa, K. and Hoskins, R. (2015), “Decolonizing indigenous intellectual and cultural rights in
heritage institutions: a survey of policy and protocol in South Africa”, Mousaion, Vol. 33
No. 3, pp. 55-72.
 and O’Shea, S. (2013), “BILI: Building Information Literacy in
Connolly, A., Curran, L.I., Lynch, A.
Ireland”, Library and Information Research, Vol. 37 No. 114, pp. 37-54.
Cunha, M.B.F., Barbosa, A., Cardoso, J. and Borbinha, J. (2018), “Risk Factors Identification in the
Domain of Scientific Data Management: A generic assessment supported by a Delphi Study”,
World Digital Libraries-An international journal, Vol. 11 No. 1, pp. 1-11.
Du Toit, M. and Stilwell, C. (2012), “The KwaZulu-Natal School Library Policy and its feasibility for
implementation in the Province”, South African Journal of Libraries and Information Science,
Vol. 78 No. 2, pp. 120-131.
Du, Y. (2009), “Librarians’ responses to “reading at risk”: A Delphi study”, Library and Information
Science Research, Vol. 31 No. 1, pp. 46-53.
Dwyer, M. (1999), “A Delphi survey of research priorities and identified areas for collaborative
research in health sector library and information services UK”, Health Libraries Review, Vol. 16
No. 3, pp. 174-191.
Eldredge, J.D., Ascher, M.T., Holmes, H.N. and Harris, M.R. (2012), “The new Medical Library
Association research agenda: final results from a three-phase Delphi study”, Journal of the
Medical Library Association: JMLA, Vol. 100 No. 3, p. 214.
Feret, B. and Marcinek, M. (2005), “The future of the academic library and the academic librarian: A
Delphi study reloaded”, New Review of Information Networking, Vol. 11 No. 1, pp. 37-63.
Elsayed, A.M (2017), “Web content strategy in higher education institutions: The case of King
Abdulaziz University”, Information Development, Vol. 33 No. 5, pp. 479-494.
Farzandipour, M., Meidani, Z., Riazi, H. and Sadeqi Jabali, M. (2018), “Task-specific usability
requirements of electronic medical records systems: Lessons learned from a national survey of
end-users”, Informatics for Health and Social Care, Vol. 43 No. 3, pp. 280-299.
Feret, B. and Marcinek, M. (1999), “The future of the academic library and the academic librarian: a
Delphi study”, Librarian Career Development, Vol. 7 No. 10, pp. 91-107.
Frank, E.P. and Pharo, N. (2016), “Academic Librarians in Data Information Literacy Instruction: A
case Study in Meteorology”, College and Research Libraries, Vol. 77 No. 4, pp. 536-552.
Garcia-Crespo, A., Colomo-Palacios, R., Soto-Acosta, P. and Ruano-Mayoral, M. (2010), “A qualitative
study of hard decision making in managing global software development teams”, Information
Systems Management, Vol. 27 No. 3, pp. 247-252.
Garnett, C., Crane, D., West, R., Brown, J. and Michie, S. (2015), “Identification of behavior change
techniques and engagement strategies to design a smartphone app to reduce alcohol consumption
using a formal consensus method”, JMIR mHealth and uHealth, Vol. 3 No. 2, pp. e73.
Garvin, L.A. and Simon, S.R. (2017), “Prioritizing Measures of Digital Patient Engagement: A Delphi
Expert Panel Study”, Journal of medical Internet research, Vol. 19 No. 5, pp. e182.
Gonzalez, R., Gasco, J. and Liopis, J. (2006), “Information systems managers’ view about outsourcing in Review of the
Spain”, Information Management and Computer Security, Vol. 14 No. 4, pp. 312-326, doi:10.
1108/09685220610690790. Delphi method
Goswami, P.R. (2004), “The application of leadership and management principles and strategies to a
library and information service”, Science and Technology Libraries, Vol. 23 Nos 2-3, pp. 35-42.
Hales, C. (1985), “How should the information needs of the aging be met? A Delphi response”, The
Gerontologist, Vol. 25 No. 2, pp. 172-176.
945
Halpin, E., Rankin, C., Chapman, E.L. and Walker, C. (2015), “Measuring the value of public libraries in
the digital age: What the power people need to know”, Journal of Librarianship and Information
Science, Vol. 47 No. 1, pp. 30-42.
Haqaf, H. and Koyuncu, M. (2018), “Understanding key skills for information security managers”,
International Journal of Information Management, Vol. 43, pp. 165-172.
Harer, J.B. and Cole, B.R. (2005), “The importance of the stakeholder in performance measurement:
critical processes and performance measures for assessing and improving academic library
services and programs”, College and Research Libraries, Vol. 66 No. 2, pp. 149-170.
Holsapple, C.W. and Joshi, K.D. (2002), “Knowledge manipulation activities: results of a Delphi study”,
Information and Management, Vol. 39 No. 6, pp. 477-490.
Howard, K., Partridge, H., Hughes, H. and Oliver, G. (2016), “Passion trumps pay: a study of the future
skills requirements of information professionals in galleries, libraries, archives and museums in
Australia”, Information Research: An International Electronic Journal, Vol. 21 No. 2, pp. n2.
Howze, P.C. and Dalrymple, C. (2004), “Consensus without all the meetings: using the Delphi method
to determine course content for library instruction”, Reference Services Review, Vol. 32 No. 2,
pp. 174-184.
Hsieh, L.F., Chin, J.B. and Wu, M.C. (2006), “Performance evaluation for university electronic libraries
in Taiwan”, The Electronic Library, Vol. 24 No. 2, pp. 212-224.
Hsu, M.J. (2014),“The study of critically essential competences for digital publishing editors”,
Publishing research quarterly, Vol. 30 No. 1, pp. 11-22.
Iden, J. and Langeland, L. (2010), “Setting the stage for a successful ITIL adoption: A Delphi study of
IT experts in the Norwegian armed forces”, Information systems management, Vol. 27 No. 2,
pp. 103-112.
Irvine, K.Y., Jessiman, W. and Felce, A. (2012), “Prioritizing research and dissemination: a Delphi study
of NHS Highland midwives”, Library and Information Research, Vol. 36 No. 113, pp. 32-47.
Joia, L.A. and Sily de Assis, M.F. (2019),“Motivations for the IT professional turnaway intention: A
Delphi approach”, Information Systems Management, Vol. 36 No. 3, pp. 228-242.
Jungwirth, D. and Haluza, D. (2019), “Information and communication technology and the future of
healthcare: Results of a multi-scenario Delphi survey”, Health informatics journal, Vol. 25 No. 1,
pp. 161-173.
Keller, A. (2001), “Future development of electronic journals: a Delphi survey”, The electronic library,
Vol. 19 No. 6, pp. 383-396.
Khosrow-Pour, M. and Herman, N. (2001), “Critical issues of Web-enabled technologies in modern
organizations”, The Electronic Library, Vol. 19 No. 4, pp. 208-220.
Erm, A., Arst, H., Trei, T., Reinart, A. and Hussainov, M. (2001), “Optical and biological properties of
€
Lake Ulemiste, a water reservoir of the city of Tallinn I: Water transparency and optically active
substances in the water lakes and reservoirs”, Research and Management, Vol. 6 No. 1, pp. 63-74.
Kochtanek, T.R. and Hein, K.K. (1999), “Delphi study of digital libraries”, Information Processing and
Management, Vol. 35 No. 3, pp. 245-254.
Lamothe, J. (2013), “What keeps CARL directors awake at night?”, Library Management, Vol. 34 Nos
6/7, pp. 512-520.
JD de Leeuw, R.A., Walsh, K., Westerman, M. and Scheele, F. (2018), “Consensus on quality indicators of
postgraduate medical e-learning: Delphi study”, JMIR Medical Education, Vol. 4 No. 1, p. e13.
76,4
Ludwig, L. and Starr, S. (2005), “Library as place: results of a Delphi study”, Journal of the Medical
Library Association, Vol. 93 No. 3, p. 315.
Luo, L. (2012), “Professional preparation for “text a librarian”: what are the requisite competencies?”,
Reference and User Services Quarterly, Vol. 52 No. 1, p. 44.
946 Maceviciute, E. and Wilson, T.D. (2009), “A Delphi investigation into the research needs in Swedish
librarianship”, Information Research: An International Electronic Journal, Vol. 14 No. 4, pp. n4.
Madrid, M.M. (2013), “A Study of Digital Curator Competences: A survey of experts”, The
International Information and Library Review, Vol. 45 Nos 3-4, pp. 149-156.
Abd Manaf, Z. (2008), “Establishing the national digital cultural heritage repository in Malaysia”,
Library Review, Vol. 57 No. 7, pp. 537-548.
Martın-Alcazar, F., Ruiz-Martınez, M. and Sanchez-Gardey, G. (2019), “Assessing social capital in
academic research teams: a measurement instrument proposal”, Scientometrics, Vol. 121 No. 2,
pp. 917-935.
Matar, J.L., Maybery, D.J., McLean, L.A. and Reupert, A. (2018), “Web-Based Health Intervention for
Young People Who Have a Parent with a Mental Illness: Delphi Study Among Potential Future
Users”, Journal of medical Internet research, Vol. 20 No. 10, e10158.
Matheson, N.W. (1982).“Perspectives on academic health sciences libraries in the 1980s: indicators
from a Delphi study”, Bulletin of the Medical Library Association, Vol. 70 No. 1, pp. 28.
McFadzean, E., Ezingeard, J.N. and Birchall, D. (2011), “Information assurance and corporate strategy:
a Delphi study of choices, challenges, and developments for the future”, Information Systems
Management, Vol. 28 No. 2, pp. 102-129.
McLeod, J. and Childs, S. (2007), “Consulting records management oracles—a Delphi in practice”,
Archival Science, Vol. 7 No. 2, pp. 147-166.
Medeiros, N. (2001), “Peering over the fortress walls: the metadata invasion begins Oclc systems and
services”, International digital library perspectives, Vol. 17 No. 4, pp. 154-156.
Millican, R. and Wallace, D.P. (1992), “Research needs in academic law libraries”, Law Library Journal,
Vol. 84, p. 421.
Missingham, R. (2011), “Parliamentary library and research services in the 21st century: A Delphi
study”, IFLA Journal, Vol. 37 No. 1, pp. 52-61.
uller, R.M., Linders, S. and Pires, L.F. (2010), “Business intelligence and service-oriented architecture:
M€
a Delphi study”, Information Systems Management, Vol. 27 No. 2, pp. 168-187.
Soleimanzade, N., Asemi, A., CheshmehSohrabi, M. and Shabani, A. (2019), “The scientific information
exchange general model at digital library context: Internet of things”, Library Philosophy and
Practice (e-journal), Vol. 2019, pp. 21-50, available at: http://digitalcommons.unl.edu/
libphilprac/2150.
Neuman, D (1995), “High school students’ use of databases: results of a national Delphi study”, Journal
of the American Society for Information Science, Vol. 46 No. 4, pp. 284-298.
Niehaves, B. (2011), “Iceberg ahead: On electronic government research and societal aging”,
Government Information Quarterly, Vol. 28 No. 3, pp. 310-319.
Noh, Y. (2010a), “A Korean study on verifying evaluation indicators and developing evaluation scales
for electronic resources using a pilot evaluation”, Libri, Vol. 60 No. 1, pp. 38-56.
Noh, Y. (2010b), “A study on developing evaluation criteria for electronic resources in evaluation
indicators of libraries”, The Journal of Academic Librarianship, Vol. 36 No. 1, pp. 41-52.
Pinto, M., Ponjuan, G., Fernandez, M. and Sales, D. (2017), “Information literacy life cycle and its
standards and models: A view from Ibero-America”, Journal of Librarianship and Information
Science, Vol. 49 No. 4, pp. 409-423.
Poirier, E. and Robinson, L. (2014a), “Slow Delphi: An investigation into information behaviour and Review of the
the Slow Movement”, Journal of Information Science, Vol. 40 No. 1, pp. 88-96.
Delphi method
Poirier, L. and Robinson, L. (2014b), “Informational balance: slow principles in the theory and practice
of information behaviour”, Journal of documentation, Vol. 70 No. 4, pp. 687-707.
Pournaghi, R. and Babalhavaeji, F. (2015), “The factors and criteria for prioritization of GIS utilization
by libraries”, The Electronic Library, Vol. 33 No. 2, pp. 181-195.
Price, L. and Robinson, L. (2017), “‘Being in a knowledge space’: Information behaviour of cult media 947
fan communities”, Journal of Information Science, Vol. 43 No. 5, pp. 649-664.
Rabina, D. (2013), “Reference Materials in LIS Instruction: A Delphi Study”, Journal of Education for
Library and Information Science, Vol. 54 No. 2, pp. 108.
Richey, J. (2012), “Motivators and Barriers to Sexual-Health Information Provision in High School
Libraries: Perspectives from District-Level Library Coordinators and High School Principals”,
School Library Research, Vol. 15, pp. 1-17.
Riedmann, D., Jung, M., Hackl, W.O. and Ammenwerth, E. (2011), “How to improve the delivery of
medication alerts within computerized physician order entry systems: an international
Delphi study”, Journal of the American Medical Informatics Association, Vol. 18 No. 6,
pp. 760-766.
Sanchez-Tarrago, N., Fernandez-Molina, J.C. and Rivero, A.C. (2012), “An open access policy for the
scientific output of Cuba’s National Health System”, Libri, Vol. 62 No. 3, pp. 211-221.
Saunders, L (2009), “The future of information literacy in academic libraries: a Delphi study”, Portal:
Libraries and the Academy, Vol. 9 No. 1, pp. 99-114.
Schneider, F., van Osch, L. and de Vries, H. (2012), “Identifying factors for optimal development of
health-related websites: a Delphi study among experts and potential future users”, Journal of
Medical Internet Research, Vol. 14 No. 1, pp. e18.
Seakins, A. and Dillon, J. (2013), “Exploring research themes in public engagement within a Natural
History Museum: A modified Delphi approach”, International Journal of Science Education,
Part B, Vol. 3 No. 1, pp. 52-76.
Sheikhshoaei, F., Naghshineh, N., Alidousti, S. and Nakhoda, M. (2018), “Design of a digital library
maturity model (DLMM)”, The Electronic Library, Vol. 36 No. 4, pp. 607-619.
Shirish, A., Boughzala, I. and Srivastava, S.C. (2016), “Adaptive use of social networking applications
in contemporary organizations: Examining the motivations of Gen Y cohorts”, International
Journal of Information Management, Vol. 36 No. 6, pp. 1111-1123.
Silva, A.G., Sim~oes, P., Santos, R., Queiros, A., Rocha, N.P. and Rodrigues, M. (2019), “A Scale to
Assess the Methodological Quality of Studies Assessing Usability of Electronic Health
Products and Services: Delphi Study Followed by Validity and Reliability Testing”, Journal of
medical Internet research, Vol. 21 No. 11, e14829.
Simsova, S. (1975), “A Delphi survey of comparative librarianship”, International library review, Vol. 7
No. 4, pp. 417-426.
Smiraglia, R.P. (2010), “A research agenda for cataloging: The CCQ Editorial Board responds to the Year
of Cataloging Research”, Cataloging and Classification Quarterly, Vol. 48 No. 8, pp. 645-651.
Streatfield, D., Andrade Blanco, P., Chiranov, M., Dryzaite, I., Kochanowicz, M., Liubyva, T. and
Tkachuk, Y. (2015), “Innovative impact planning and assessment through Global Libraries”,
Performance Measurement and Metrics, Vol. 16 No. 2, pp. 177-192.
Tanloet, P. and Tuamsuk, K. (2011), “Core competencies for information professionals of Thai
academic libraries in the next decade (AD, 2010-2019)”, The International Information &
Library Review, Vol. 43 No. 3, pp. 122-129.
Torrecilla-Salinas, C.J., De Troyer, O., Escalona, M.J. and Mejıas, M. (2019), “A Delphi-based expert
judgment method applied to the validation of a mature Agile framework for Web development
projects”, Information Technology and Management, Vol. 20 No. 1, pp. 9-40.
JD Townsend, L., Hofer, A., Lin Hanick, S. and Brunetti, K. (2016), “Identifying threshold concepts for
information literacy: A Delphi study”, Communications in Information Literacy, Vol. 10
76,4 No. 1, pp. 1.
Vance, K.E., Magrill, R.M. and Downen, T.W. (1977), “Future of library education: 1975 Delphi study”,
Journal of education for Librarianship, Vol. 18 No. 1, pp. 3-17.
Waller Casstevens, S. (2016), “A Delphi Study”, Teacher Librarian, Vol. 44 No. 1, pp. 16-19.
948 Weingand, D.E. (1989), “Wisconsin Continuing Education Profile: A Delphi Projection of Needs
(1986)”, Journal of Education for Library and Information Science, Vol. 30 No. 1, pp. 39-50.
Welland, S. (2017), ““Us and them”: expert and practitioner viewpoints on small New Zealand
community archives”, Information Research, Vol. 22 No. 4, pp. 1-12.
Westbrook, L (1997), “Information access issues for interdisciplinary scholars: Results of a Delphi
study on women’s studies research”, The Journal of Academic Librarianship, Vol. 23 No. 3,
pp. 211-216.
Wijayaratne, A.L. and Singh, D. (2015), “Developing an academic library website model, a designer’s
checklist, and an evaluative instrument: a Delphi method approach”, The Electronic Library,
Vol. 33 No. 1, pp. 35-51.
Willard, P. (1992), “Public librarians’ views of the impact of personal computers on public libraries”,
The Australian Library Journal, Vol. 41 No. 1, pp. 47.
Wright, A., Sittig, D.F., Ash, J.S., Feblowitz, J., Meltzer, S., McMullen, C., . . . and Evans, R.S. (2011),
“Development and evaluation of a comprehensive clinical decision support taxonomy:
comparison of front-end tools in commercial and internally developed electronic health
record systems”, Journal of the American Medical Informatics Association, Vol. 18 No. 3,
pp. 232-242.
Yuvaraj, M. (2016), “Ascertaining the factors that influence the acceptance and purposeful use of
cloud computing in medical libraries in India”, New Library World, Vol. 117 Nos 9/10,
pp. 644-658.
Zeng, X., Ping, S., Mengli, W. and Du Weichun, C. (2008), “Delphi research on information literacy
competency standards for higher education in Beijing, China”, Chinese Librarianship,
Vol. 25, pp. 1-9.
Zhang, Y. and Salaba, A. (2009), “What is next for functional requirements for bibliographic records?
A Delphi study”, The Library Quarterly, Vol. 79 No. 2, pp. 233-255.
Zhu, Q., Du, J.T., Meng, F., Wu, K. and Sun, X. (2011), “Using a Delphi method and the analytic
hierarchy process to evaluate Chinese search engines: A case study on Chinese search engines”,
Online Information Review, Vol. 35 No. 6, pp. 942-956.
Zins, C. (2007a), “Classification Schemes of Information Science: Twenty-Eight Scholars Map the
Field”, Journal of the American Society for Information Science & Technology, Vol. 58 No. 5,
pp. 645-672.
Zins, C. (2007b), “Conceptions of Information Science”, Journal of the American Society for Information
Science & Technology, Vol. 58 No. 3, pp. 335-350.
Zins, C. (2007c), “Conceptual Approaches for Defining Data, Information, and Knowledge”, Journal of
the American Society for Information Science & Technology, Vol. 58 No. 4, pp. 479-493.
Zins, C. (2007d), “Knowledge Map of Information Science”, Journal of the American Society for
Information Science and Technology, Vol. 58 No. 4, pp. 526-535.
Zins, C. and Santos, P. L. (2017), “Content Selection in Undergraduate LIS Education”, Journal of
Education for Library and Information Science, Vol. 58 No. 3, pp. 120-140.

Corresponding author
Brady D. Lund can be contacted at: blund2@g.emporia.edu
Article Population How experts were identified Number of participants Delphi type Findings

Abukhader (2019) LIS professional Posting on Listserv 3 (This Delphi "failed" e-Delphi A taxonomy of expected roles
associations due to an unsatisfactory of librarians related to Appendix
response rate) knowledge management
Addison (2003) Project managers of South Employment in software 32, all rounds Classical e-Delphi Identified potential risks in the
African software houses houses development of e-commerce
projects
Adhiarna et al (2013) Korean RFID experts Employment at specific 14, both rounds Policy Delphi Development of a framework
institution for RFID adoption
Ali Al-Busaidi (2014) IT managers in government Employment as IT manager 10, all rounds Classical e-Delphi Attitudes of IT managers
organizations in Oman in Oman toward the use of social
networking sites for
organizational purposes
Alexander et al. (2013) Nursing home Employment and 31, all rounds Policy e-Delphi Development of an
administrators in the U.S. experience as nursing home information technology
administrator maturity model for nursing
homes in U.S.
Ambrosiadou and Goulis Diabetologists Experience in area of 5, all rounds Decision Delphi Development of a rule-based
(1999) employment decision support system for
individuals with diabetes, used
for real-world subjects
Anthopoulos and Smart City experts Scholarly Publications; 16, all rounds Disaggregative Policy e- Action areas and measures for
Reddick (2016) highly ranked projects Delphi future e-government research
contribution to the smart city
Arazy et al. (2013) Senior librarians at U.S. The researchers 3, all rounds Modified Delphi An analysis of how group
university member disagreements/
conflict affect group
performance in authoring
Wikipedia articles
Asonitis and Kostagiolas Academic staff and library Employment in Greek 11, first round; 9, second Modified Delphi Framework for the
(2010) directors affiliated with public libraries round identification of human capital
library services for Greek in Greek public libraries
public libraries
Baker (2006) Academic library Senior employees of higher Not Specified Not Specified Forecast the future of digital
professionals in U.K. education libraries libraries in U.K.
Bartlett et al. (2018). Library and information Scholarly publications 73, all rounds Modified e-Delphi Development of MMAT
professionals and representing mastery of mixed-method appraisal tool
researchers research methods

(continued )
Delphi method

949
Review of the

retrieved for years

1971–2019
Table AI.
LIS Delphi studies
 JD
76,4

950

Table AI.
Article Population How experts were identified Number of participants Delphi type Findings

Baruchson-Arbib and LIS professionals Senior employees in 40, first round; 26, Modified e-Delphi This Delphi identified trends
Bronstein (2002) libraries and LIS second round in the future of the LIS
departments discipline
Battistella and Nonino Web experts, innovation Scholarly Publications 14, all rounds Critical Delphi Identified potential drivers for
(2012) experts, web developers innovation in the development
of web-based platforms
Belzer et al. (1971) Information science Committee membership 19 members of the ASIS Classical Delphi This Delphi study identified
educators education for important aspects for
information science inclusion in information
committee science curriculum. The Delphi
and questionnaire identified
the most important topics in
LIS education as being
"introduction to information
science," "systems theory and
application," "mathematical
methods in information
science," "computer
organization and
programming," "abstracting,
indexing, and cataloging," and
"Research methods," while the
Delphi also identified
"communication, information,
and coding theory" as an
important area of the
curriculum.
Boaz (1978) LIS school deans and Employment as LIS school Not specified (sent to Classical Delphi Concepts concerning the
professors dean or professor one professor and dean future of library and
for each school, but information science education
response rate not
reported)
Brancheau and Chief information systems Position in professional 90, first round; 54, Classical Delphi Top issues in information
Wetherbe (1987) executives organizations second round; 70, third systems management
round
Bronstein and Aharony Librarians involved with Posting on Listserv 39, first round; 9, second Modified e-Delphi Opinions about the adoption of
(2009) Internet technologies round web 2.0 technologies and the
changing role of libraries

(continued )
Article Population
Brouwer et al. (2008) Experts in Internet
intervention research and
practice
Brunneret al.(2018) Experts in eHealth
education, practice, or policy
Bush and Jones (2010) School librarians
B€
uy€
uk€ozkan (2004) E-marketplace users
Cape (2004) Tutors of higher education
courses
Casselden et al. (2015) U.K. library managers
Cegielski et al. (2013) IT managers
Chavez-Hernandez Library and information
(1996) professionals in the
Caribbean region
Cherinet (2018) Public librarians
Chisa and Hoskins (2015) Knowledge preservation
experts in South Africa
Connolly et al. (2013) Information literacy experts
(Academic librarians)
Cunha et al. (2018) Scientific data specialists
Table AI.
How experts were identified
Scholarly publications
Not specified
Editorial board members
Employment at specific
institution
Position in professional
organizations
Position in professional
organizations
Employment at specific
institution
Not Specified
Students at a specific
university
Location near the
researchers
Not specified
Scholarly publications;
employment
Number of participants Delphi type Findings
30, first round; 93, Classical e-Delphi Important factors related to
second round; 59, third the dissemination of and
round exposure to Internet
interventions aimed at adults
> 12, all rounds Policy Delphi Development of an eHealth
capabilities framework
33, first round; 17, Classical e-Delphi Identified professional
second round; 21, third dispositions, including
round pedagogies and assessments,
for school librarians
Not Specified Fuzzy (modified) Delphi System for identifying the
ideal e-marketplace
10, first round; 6, second Classical Delphi List of criteria to evaluate
round Holocaust films for libraries
15, all rounds Classical e-Delphi Identification of opportunities
and challenges for libraries
utilizing volunteer service
Not Specified Classical e-Delphi Adoption strategy for
emerging IT in corporate
settings
Not Specified Not specified Design of a library network
model
Not Specified (less than Classical e-Delphi Identified future roles and
60) skills desired among
professional librarians
4, all rounds Classical Delphi Framework for the
preservation of indigenous
knowledge in South Africa
16, all rounds Modified Delphi New approach to information
literacy development in
Ireland focused on community
of practice and enhanced
advocacy
11, all rounds Modified e-Delphi Identified risk factors for the
security, storage, and
dissemination of scientific data
(continued )
Delphi method
951
Review of the

Table AI.
Article Population
Du Toit and Stilwell School librarians
(2012)
Du (2009) Library and information
professionals and
researchers
Dwyer (1999) Library practitioners
involved in professional
organizations
Eldredge et al. (2012) Leaders within national,
section, or chapter levels of
the Medical Library
Association; authors of
articles in top medical
library journals
Leaders within the Position in professional
Medical Library organizations
Association
Elsayed (2017) Higher education website
coordinators at specific
institution
Farzandipour et al. (2018) Electronic medical records
management specialists
Feret and Marcinek Academic library
(1999) professionals
Academic library Employment in tenured/
professionals leadership positions in
libraries or LIS schools
How experts were identified
Location near the
researchers
Position in professional
organizations
Position in professional
organizations
Position in professional
organizations
54, first round; 15, other
rounds
Employment as web
coordinator at Specific
Institution
Employment and
experience as an electronic
medical records
management specialist
Employment in tenured/
leadership positions in
libraries or LIS schools
36, all rounds
Number of participants Delphi type
5, all rounds Policy e-Delphi
11, both rounds Modified e-Delphi
24, first round; not Classical Delphi
specified, other rounds
141, first round; 108, Policy Delphi
second round; 115, third
round
Modified e-Delphi Identification of the 12
highest priority questions
facing medical librarianship/
medical library association
35, all rounds Classical e-Delphi
28, all rounds Modified e-Delphi
23, all rounds Classical e-Delphi
Classical e-Delphi Replication of the 1999 study
(above), identifies alignment
between 1999 and 2005
study and forecasts the next
decade of librarianship
JD
76,4
952
Findings
Guidelines for developing
policy intervention for school
librarianship programs in
South Africa
Expert opinions regarding
findings in the National
Endowment for the Arts’
report, Reading at Risk
Set of 20 research priorities for
health libraries in the U.K.
Agenda for the future of the
Medical Library Association
Strategy for the development
of web content at Abdulaziz
University
Usability requirements of
electronic medical records
systems
Major topics in the future of
academic librarianship
(continued )
Article Population
Frank and Pharo (2016) Instructional librarians
(academic)
Garcia-Crespo et al. Members of software
(2010) development teams
Garnett et al. (2015) Experts in the field of
alcohol and behavioral
change
Garvin and Simon (2017) National (U.S.) eHealth
experts
Gonzalez et al. (2006) Information systems
managers for large
corporations in Spain
Goswami (2004) Government document
librarians
Hales (1985) Elderly care providers,
policy administrators, and
library and information
professionals
Halpin et al. (2015) U.K. public library leaders
Haqaf and Koyuncu Information security
(2018) managers
Harer and Cole (2005) Academic librarians (U.S.)
Table AI.
How experts were identified
Association with the
research institutions in the
discipline of interest
Employment as software
developers on selected
teams
Scholarly publications;
networks
Scholarly publications
Employment as information
systems manager for large
corporation
Employment as Gov Doc
Librarian
Position in professional
organizations; employment
in specific institutions
Employment in
administration of public
library
Professional experience as
indicated on LinkedIn
Scholarly publications
Number of participants Delphi type Findings
10, all rounds Modified e-Delphi Perceptions of data
information literacy and
attitudes toward info literacy
instruction for meteorology
graduate students
25, all rounds Classical Delphi Guidance for hard decision-
making in global software
development teams
7, all rounds Classical e-Delphi Strategies to engage
behavioral change in alcohol
consumption through a
smartphone app
12, all rounds Modified e-Delphi Measures to assess digital
patient engagement with a
Veterans’ Affairs eHealth
portal
357, first round; 15, Modified Delphi Summarizes information
second round systems managers views
about outsourcing
Not Specified Classical Delphi Opinions about management
principles in library services
114, first round; 102, Policy Delphi Action plan for better
second round; 93, third satisfying the information
round needs of the elderly
19, all rounds Classical Delphi Identified areas of knowledge
that need to be communicated
to key library stakeholders
19, all rounds Classical e-Delphi Identification of key skills for
information security
managers and training
programs to best acquire these
skills
10, first round; 9, second Critical e-Delphi Framework for the
round measurement of quality in
academic library services and
programs
(continued )
Delphi method
953
Review of the

Table AI.
Article Population
Holsapple and Joshi Knowledge management
(2002) researchers and
professionals
Howard (2018) LIS professionals
Howard et al. (2016) LIS professionals in
Australia
Howze and Dalrymple Academic librarians (U.S.)
(2004)
Hsieh et al. (2006) Libraries in U.K., U.S.,
Germany, and Taiwan
Hsu (2014) Digital publishers
Iden and Langeland IT experts in the Norwegian
(2010) Armed Forces
Irvine et al. (2012) Midwives working in the
NHS Highland
Joia and Sily de Assis IT Professionals
(2019)
Jungwirth and Haluza Austrian healthcare experts
(2019)
Keller (2001) Researchers, publishers,
journal agents, and
librarians involved with e-
publishing
Khosrow-Pour and Researchers of web-based
Herman (2001) technologies
How experts were identified
Scholarly publications;
Employment
Scholarly publications;
position in professional
organization
Professional networks;
snowball sampling
Employment at specific
institution
Geographic distribution
Employment in
administrative position in
digital publisher
Employment in Norwegian
Armed Forces
Employment as midwives in
this region
Not specified
Employment
Employment in position
involving e-publishing
Scholarly publications
JD
76,4
954
Number of participants Delphi type Findings
31, first round; 17, Modified e-Delphi List of important knowledge
second round manipulation activities
NA Grounded Delphi Introduces the grounded
Delphi to LIS
18, all rounds Grounded Delphi Important skills for future LIS
professionals in Australia
34, first round; 25, Modified e-Delphi Determined standardized
second round learning content for library
instruction
Not specified (“six Modified e-Delphi Role of e-libraries in the
universities”) knowledge society
18, all rounds Critical Delphi Development of core
competencies for digital
publishing editors
15, all rounds Policy e-Delphi Plan for successful adoption of
the Information Technology
Infrastructure Library
64, first round; 53, Classical Delphi Research priorities related to
second round; 50, third midwifery in NHS Highlands,
round U.K.
35, all rounds Modified e-Delphi Identification of the main
motivations that lead IT
professionals to move to
another field or profession
73, all rounds modified e-Delphi Perceptions of the
implementation of telehealth in
medical practice in Australia
41, first round; 42, Online Delphi Forecast of future role of
second round; 39, third electronic journals in research
round
17, all rounds Critical e-Delphi Identified issues related to the
management of web-enabled
technologies in organizational
settings
(continued )
Article Population
Erm et al. (2001) Nurses in U.K.
Kochtanek and Hein Digital library experts
(1999)
Koskiala and Huhtanen Finnish librarians
(1989)
Lamothe (2013) Canadian academic library
directors
de Leeuw et al. (2018) International e-learning
experts
Ludwig and Starr (2005) Health science library
stakeholders
Luo (2012) Reference librarians
Maceviciute and Wilson Swedish librarians
(2009)
Madrid (2013) Digital resource curators
Abd Manaf (2008) Individuals involved with
digitization of cultural
heritage in Malaysia
Martın-Alcazar et al. Members of Spanish
(2019) research teams from variety
of disciplines
Matar et al. (2018) Young people with parents
with mental illness
Matheson (1982) Health science library
stakeholders
Table AI.
How experts were identified
Employment in specific
institutions
Scholarly publications
Employment in libraries in
Finland
Library directors within the
CARL network
Scholarly publications;
position in professional
organization
Professional Networks;
ListServ Posts
ListServ; scholarly
publications
Employment in libraries in
Sweden
Scholarly publications
Location near the
researchers
Scholarly publications
Location near the
researchers
Employment in relevant
stakeholder positions
Number of participants Delphi type Findings
431, first round; not Classical e-Delphi Nursing research priorities in
specified, other rounds the U.K.
6, first and second Online Delphi Identification of key issues/
rounds; 4, third round concerns related to digital
libraries
11, all rounds Classical Delphi Forecast for use of information
technology in Finnish libraries
(for years 1996–2010)
22, first round; 13, other Modified e-Delphi Themes and concerns for
rounds Canadian academic libraries
23, all rounds Real-Time Delphi Development of an evaluation
tool for postgraduate medical
e-learning
30, all rounds Classical e-Delphi Forecast of the future of health
science library spaces
39, first round; 20, Classical e-Delphi Competencies for the
second round; 10, third successful delivery of text
round reference services
78, first round; 64, Modified e-Delphi Research needs/priorities for
second round; 16, third Swedish librarians
round
16, first round; 14, Modified Delphi Competencies for digital
second round; 15, third curation in information
round organizations
20, all rounds Classical e-Delphi Identification of challenges
and successful approaches to
digitization of cultural heritage
in Malaysia
62, first round; 45, Modified e-Delphi Model for the assessment of
second round social capital among academic
research teams in Spain
14, first round; 268, Modified e-Delphi Perceptions toward web-based
second round intervention for mental health
79, first round; 133, Classical Delphi Perceptions/forecasting for
second round; 112, third health science libraries in the
round 1980s
(continued )
Delphi method
955
Review of the

Table AI.
Article Population
McFadzeanet al.(2011) Corporate information
systems managers
McLeod and Childs Records management
(2007) professionals in U.K.
Medeiros (2001) Metadata librarians
Millican and Wallace Law librarians
(1992)
Missingham (2011) Leading innovators and
thinkers in library services
for parliaments
M€
uller et al. (2010) Business intelligence
professionals
Soleimanzade et al. (2019) Digital library and Internet
of Things experts
Neuman (1995) School librarians (U.S.)
Niehaves (2011) e-Government researchers
Noh (2010a) Electronic resources
librarians
Noh (2010b) Academic library
professionals
How experts were identified
Employment in senior
management position of
selected firms
Position in professional
organizations
Not Specified
Scholarly publications
Position in professional
organizations
Employment at specific
institutions; scholarly
publications
Scholarly publications
Award winners;
recommendations from
professional network
Scholarly publications
Employment at specific
institution
Not Specified
JD
76,4
956
Number of participants Delphi type Findings
36, all rounds Policy e-Delphi Plan to successfully address
legal and regulatory
challenges among information
assurance, information
systems, and corporate
strategies
8, all rounds Policy e-Delphi Proposed standards for
records management
Not Specified Not Specified Identification of aspects of
metadata to be covered in LIS
curricula; research topics of
priority to the metadata
community
138, first round; not Modified Delphi Research priorities for law
specified, other rounds libraries
9, all rounds Real-time Delphi Participants anticipated great
changes in services offered by
parliamentary libraries in
coming years
16, first round; 13, Classical e-Delphi Identification of how service-
second round; 11, third oriented architecture could
round support the use of business
intelligence
15, all rounds Real-Time Policy Delphi Development of a policy model
for automated library services
25, all rounds Classical Delphi Issues/Concerns with use of
online/CD-ROM databases in
school libraries
12, first round; 24, Classical e-Delphi Concerns related to
second round eGovernment research and
societal aging
10, all rounds Classical e-Delphi Evaluation standards for
electronic resources in
libraries
Not specified Modified e-Delphi Development of evaluation
criteria for the use of electronic
resources in academic libraries
(continued )
Article Population
Pinto et al. (2017) International information
literacy instructors
Poirier and Robinson Information behavior
(2014a) researchers
Poirier and Robinson Information behavior
(2014b) researchers
Pournaghi and Librarians and GIS experts
Babalhavaeji (2015)
Price and Robinson Cult media fans
(2017)
Rabina (2013) Reference services
professionals in public and
academic libraries
Richey (2012) High school library
coordinators and principals
Riedmann et al. (2011) Researchers with expertise
in computerized physician
order entry systems
Table AI.
How experts were identified
Scholarly publications;
employment-related
accomplishments
Scholarly publications
Scholarly publications
Employment as GIS
librarian
Professional networks;
social media post
Scholarly publications;
position in professional
organization; professional
or teaching experience
Employment in select Texas
school libraries
Scholarly publications
Number of participants Delphi type Findings
43 - only second round Classical Delphi–article Mapping of the information
reports only on round two literacy life cycle
25, all rounds Slow Delphi This study evaluated the slow
Delphi method within the
context of a study of slow
movement in the information
disciplines and professions
25, all rounds Slow Delphi This is the complete study,
which evaluates the
integration of slow movement
principles in the information
disciplines and professions
Not specified Classical e-Delphi Identification of factors
influencing the adoption of
Geographic Information
System technology in libraries
and LIS research
31, first round; 30, Classical e-Delphi Identified aspects of fan
second and third information behavior (e.g.,
rounds gatekeeping, classifying and
tagging, and
entrepreneurship).
31, all rounds Classical e-Delphi A set of seminal reference
sources for library
professionals
29, first round; 25, Classical Delphi Motivations and barriers for
second and thirds provision of sexual health
rounds information in high school
libraries’
73, first round; 69, Modified e-Delphi Identification of context
second round information items to help
prioritize medication alerts
(continued )
Delphi method
957
Review of the

Table AI.
Article Population
Sanchez-Tarrago et al. Members of the Cuban
(2012) sector of health, Higher
education and science,
technology, and the
environment
Saunders (2009) Information literacy experts
(academic librarians)
Schneider et al. (2012) Experts in health-related
information online
Seakins and Dillon (2013) Staff of a natural history
museum
Sheikhshoaei et al. (2018) Scholarly publications;
employment with relevant
specialization
Shirish et al. (2016) French business school
students
Silva et al. (2019) Usability study experts
Simsova (1975) Members of the Association
of British Library Schools
directory, International and
Comparative Librarianship
committee members, and
contributors to Focus
Journal
Smiraglia (2010) Cataloging and
classification quarterly
editorial board members
Streatfield et al. (2015) Library managers in
Lithuania
How experts were identified
Employments in Cuban
sector of health, higher
education, and science,
technology, and the
environment
Scholarly publications
Scholarly publications
Employment in specific
institution
Referral; snowball sampling
Enrolled at a specific
institution
Scholarly publications
Position in professional
organizations
Membership on journal
board
Employment as library
manager
JD
76,4
958
Number of participants Delphi type Findings
30, all rounds Policy e-Delphi Development of a policy for the
open access publication of
scientific contribution from the
Cuban National Health System
13, all rounds Classical Delphi Forecast for information
literacy education and
applications in academic
libraries
20, first round; 270, Classical e-Delphi Factors that should be
second round; 32, third accounted for in the design of
round health websites
9, all rounds Classical e-Delphi Research themes in public
engagement in museums
20, all rounds Modified e-Delphi Identification of the maturity
features of a digital library
maturity model
24, all rounds Technological Delphi Indication of the use of social
networking applications
among Generation Y students/
professionals
22, first round; 23, Policy e-Delphi Scale to assess the
second and third methodological quality of
rounds studies assessing usability of
electronic health products and
services
45, first round; 12, Modified Delphi Several findings about the
second round need for interdisciplinary
solutions in comparative
librarianship, to challenges
with theory and practice in
librarianship, to policy and
support for developing nations
Not specified Not specified Research agenda for
cataloging and classification
56, all rounds Modified Delphi Model for library evaluation
(continued )
Article Population
Tanloet and Tuamsuk Thai university library
(2011) professionals and educators
Torrecilla-Salinas Experts in agile, capability
et al.(2019) maturity model integration,
and web engineering
Townsend et al. (2016) Academic instructional
librarians
Vance et al. (1977) LIS professionals (academic)
and LIS educators
Waller Casstevens (2016) Texas school and public
librarians
Weingand (1989) LIS professionals and
educators
Welland (2017) Archival management
employees
Westbrook (1997) LIS researchers, members of
University Presses, and
members of the Academic
Women’s Caucus (U.S.)
Wijayaratne and Singh Academic library
(2015) professionals
Willard (1992) Public librarians (Australia)
Wright et al. (2011) National experts in
healthcare IT
Yuvaraj (2016) Medical librarians
Table AI.
How experts were identified
Not specified
Scholarly publications
Scholarly publications;
position in professional
organization
Employment in specific
institutions
Employment as library
professional in Texas
Employment in libraries
and library schools in
Wisconsin
Position in professional
organizations
Employment in specific
institution; position in
professional organization
Scholarly publications;
personal contacts; posting
on Listserv
Employment in public
libraries accessible to the
researcher
Membership in a research
group
Connection with medical
libraries and experience
with library technology
Number of participants Delphi type Findings
21, all rounds Classical e-Delphi Core competencies for Thai
academic library professionals
20, first round; 19, Policy e-Delphi Development of a framework
second round; 18, third for the use of agile in web
round development projects
19, first round; Classical e-Delphi Core competencies for
unknown, other rounds information literacy
instruction
150, first round; 118, Classical e-Delphi Fundamental elements (core
second round; 90, third curriculum concepts) of library
round education
29, all rounds Classical e-Delphi Concepts of joint-use of
libraries in Texas
38, first round; 25, other Classical Delphi Forecast of future changes in
rounds library education in Wisconsin
(through the year 2000)
15, all rounds Classical e-Delphi Perceptions of the value and
management of small
community archives in New
Zealand
19, first round; 19, Decision e-Delphi Set of priorities for providing
second round; 20, third access to women’s studies
round research
15, first round; 10, Modified e-Delphi Designer’s checklist for
second round academic library websites
25, all rounds Classical Delphi Forecasts the effect that
personal computers may have
on public libraries
11, all rounds Modified Delphi Taxonomy for clinical decision
support in healthcare
organizations
32, all rounds Classical e-Delphi Factors that influence the
adoption and use of cloud
computing technology in
medical libraries in India
(continued )
Delphi method
959
Review of the
 JD
76,4

960

Table AI.
Article Population How experts were identified Number of participants Delphi type Findings

Zeng et al. (2008) University librarians in
China
Zhang and Salaba (2009) FRBR experts
Zhu et al. (2011) LIS professionals with
research expertise involving
search engines
Zins (2007a) Scholars of information
science
Zins (2007b) Scholars of information
science
Zins (2007c) Scholars of information
science
Zins (2007d) Scholars of information
science
Zins and Santos (2017) Brazilian information
science experts
Employment in university
libraries
Scholarly publications;
FRBR review group
membership
Scholarly Publications
“Leading scholars”
“Leading scholars”
“Leading scholars”
“Leading scholars”
Research databases
13, all rounds Modified e-Delphi Development of information
literacy competency standards
for university libraries in
China
33, all rounds Classical e-Delphi Key issues and challenges
facing FRBR research and
practice
12, all rounds Policy e-Delphi Core criteria for evaluating
search engines
57, first round; 39 s and Critical e-Delphi Knowledge map of the
third round discipline of information
science
57, first round; 39 s and Critical e-Delphi Definition/conceptions of the
third round discipline of information
science
57, first round; 39 s and Critical e-Delphi Definitions and relationships
third round between data, information,
and knowledge
57, first round; 39 s and Critical e-Delphi Knowledge map of the
third round discipline of information
science
21, first round; 18 s and Critical e-Delphi Design of content for
third round undergraduate LIS education