Introduction and Overview: Visualization, Retrieval,
and Knowledge†
This Perspectives issue is assembled to provide an his-
torical background to visualization in information retrieval.
It is a review of the assumptions and technology configu-
rations by which the current literature may be interpreted.
The techniques of the authors of this issue differ, but all
treat their techniques as manuals of description flowing
from a history of common mathematical and technical in-
fluences.
All technologies have histories of development. The
historical forces of visualization frame the current efforts
and comprise the field in which new problem dimensions
are addressed. No field of scientific inquiry emerges without
a background. This issue adds to the depth necessary for the
study of visualization by new students and new scholars.
Moreover, these articles also reflect developments by
some of the world’s most prestigious research institutions.
Not only are two American national laboratories repre-
sented, but also the National Aeronautics and Space Ad-
ministration (NASA), the Institute for Scientific Informa-
tion, the Digital Libraries Initiative in the Alexandria
Project, the Canadian National Archives, and the Founda-
tions of Advanced Information Visualization (FADIVA)
European group. This issue is far from the last word on this
topic, but surely it is among the most authoritative ones.
In the Beginning Was the Word
The history of information retrieval is mostly the history
of word retrieval. Very early in this history, word frequency
in document collections was used to convey distinctions
among classes of documents. Much followed from this basic
insight and today the notion of combining like documents
by common words is universal in practical information
retrieval engineering. This concept has reached so refined a
state, however, that discoveries now occur only in small
increments.
There seems little amiss in the notion that knowledge of
the world about us should be encapsulated by words. Yet,
words may be false, or their meaning misunderstood. Take
the notion of words which uniquely define knowledge of
nature, for example. One might universally define “spring”
as the period between the winter solstice and the vernal
equinox, or the word “sun” as the brightest of all objects in
†
This Perspectives issue is dedicated to the memory of Robert R.
Korfhage: teacher, scholar, friend.
© 1999 John Wiley & Sons, Inc.
JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE. 50(9):790 –793, 1999
the sky. But the first definition will elude the untutored and
the second remain unverifiable for half of each planetary
revolution.
In many circumstances, descriptive knowledge of the
world must be limited to closed vocabularies such as math-
ematics or predicate logic. These special languages remove
ambiguity, but frequently limit the power of expressiveness
to the trivial, or distort common observations into the realm
of the bizarre. The common sense in which words are used
to encapsulate knowledge is the only sense in which words
interest us. We seek in description to elucidate those prop-
erties of a phenomenon which most differentiate it from
others, or which best provide the boundaries by which
phenomena may be treated in common as a class. It may
well be that in the future visualization in retrieval will be
construed as such a special language, but one vastly more
expressive and universal.
The digital world technology has introduced bequeaths
us with ever more complex problems of knowledge differ-
entiation and classification. No one would argue that images
do not encapsulate knowledge, but how images encapsulat-
ing knowledge may be described from their primitive forms
is still in the resolution stage. Images are not usually parsed
into words as documents are so treated. On an even more
intricate plane of examples, the methodologies by which
sounds may be retrieved by images is a problem so con-
founded that no literature exists yet to address it.
In the early literature of information science, reference is
often made to a paradigm known as “the answer document.”
In this formalism, the question is immutable, and the world
image is that of an “answer document” flying like an arrow
to intercept the question and bring it to ground. This char-
acterization is considered antique. Yet, in the test collec-
tions that we must use to determine the efficacy of varying
approaches to retrieval, we essentially resort to the same
formalism. Although these static methodologies may be
altered by interactive experimentation, the quality of knowl-
edge is of a different order when we do so.
The authors in this issue present an alternative vision. It
is a vision which relies on user presentation of entire answer
sets resting in a visual field. It is not an “answer document”
upon which this vision relies, but rather an “answer set.” Is
this vision of greater power than the ordering of retrieved
objects by lists? All the authors in this issue claim that it is
a more powerful vision. Moreover, this claim is made on the
face validation presented by our human senses and the
pre-linguistic evolutionary properties of the human visual
neocortex. The visual processing skills which permitted our
species early triumph over much stronger and better adapted
CCC 0002-8231/99/090790-04
ones provide the prima facie rationale for the use of visu-
alization in information retrieval. As one observer put it late
at night at an ASIS conference, “Whatever we were doing to
survive in the brush 50,000 years ago, you can bet that it
wasn’t perfecting the skill of taking notes.”
No one claims that words are not useful in capturing
knowledge. Indeed, for detailed control and description of
complex phenomena, they represent our most valued cul-
tural legacy. The illiterate cannot expect to profit much from
visualization technology. Rather, the claim made in this
issue is that the wholistic expression of the interrelation of
knowledge objects is more powerful than any other compi-
lation of such objects. In this paradigm, a single Linnean
taxonomic tree is more revelatory than all the nodes of
description in the tree individually presented. And in this
proposition, these authors propose new visual grammars for
interpreting the manifold relations among words and im-
ages.
Such propositions do not resolve the issues of knowing
by any user. Nor do they resolve the issues of classification
boundaries by which like phenomena may be known. They
do, on the other hand, offer a paradigm of alternative
engineering strategies for user apprehension of a multime-
dia world. Alternatives are priceless. Even a rough and
uneven road is preferable to a dead end. And although it is
quite unfair to characterize advances in word retrieval as
negligible, such advances have surely been limited in recent
years.
The First Visual Interface
The first visual interface to a collection was designed and
implemented at the Johnson Space Center of NASA in the
years 1988 to 1992. In this interface, described in the article
entitled, “The NASA Image Collection Visual Thesaurus,”
the authors assumed that the task of inferring images from
terms and terms from images would introduce invariance in
image indexing. The system remained in use for two years,
but eventually failed because no automatic method to assign
terms to images could be discovered, and the manual cost of
such term assignment was too great to be supported.
The authors of this article attempted to use image de-
scriptions clustered by cosine vector methods to identify a
unique image for every thesaurus term. The candidate im-
ages suggested by this method were often heartbreakingly
close to the mark. But close was not good enough. These
developments were described in detail by Seloff (1990).
Although the Seloff article has been widely cited, the initial
article which specified the design parameters for the system
of his report has remained unpublished. It appears in this
article in the form originally presented at the ASIS midyear
conference of 1988.
The article is significant because it represents the first
identification of the components of a visual interface. Its
heritage is reflected in the article “A Collection of Visual
Thesauri for Browsing Large Collections of Geographic
Images,” by Ramsay et al., in this issue. These authors were
JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE—July 1999
more successful than Rorvig et al., due to their use of
Kohenen feature maps, a far more robust technique than the
simple cosine vector intersection of terms. They were also
able to take advantage of the homogeneity of their earth
observation data. Their system is more than merely “prom-
ising.” It actually works.
Theoretical Foundations
The article by Henry Small of the Institute for Scientific
Information contributes the most clearly defined statement
of visualization techniques to appear in print. This is a rather
bold statement, yet it is true. The article addresses the two
decade-long historical use of visualization techniques in
calculating the relationships among scientific fields by their
patterns of co-citation.
Researchers who seek “cookbook” renditions of algo-
rithms as they developed over time will find this article to be
a precise guide to alternatives. Small, in his article entitled
“Visualizing Science by Citation Mapping,” begins with the
simplest of algorithms as conceived within the computa-
tional limitations of the 1970s and ends with the most
ambitious ones presently available through Sandia National
Laboratories (SNL). In this article, students and scholars
will find algorithms applicable to many different aspects of
the co-citation problem, as Small frankly describes the
research paths that were successful and led to further en-
hancements as well as the ones that were eventually dis-
carded either because of their inefficiency in computation,
or their failure to yield truthful insights validated by earlier
techniques. Many of these algorithms may be transplanted
to address similar problems with data that may be encoun-
tered by researchers who require some intermediate pro-
cessing alternatives.
Exemplar Applications
This issue would surely be guilty of intellectual hubris
without providing a few examples of applications of re-
trieval visualization to common search problems. Although
the bibliography provided by Robert Korfhage in the online
site illustrates the widespread trials of visualization technol-
ogy, these two applications illustrate what can be done on a
completely practical level. Neither of the two systems pre-
sented in this section is a demonstration system, rather both
are part of larger efforts to control massive collections. The
first of these two articles, by Ramsay et al., “A Collection of
Visual Thesauri for Browsing Large Collections of Geo-
graphic Images,” describes the efforts to provide some
meaningful access to the more than eight million earth-
observing images made available in this decade. The second
article by Brooks and Campbell, “Interactive Graphical
Queries for Bibliographic Search,” describes the use of a
Canadian government-sponsored National Archives product
designed to permit visual searching of text materials in a
traditional boolean logic-based environment.
791
 In Ramsay et al., earth-observing images are parsed as
texts are parsed. These authors use Gabor filters to combine
like terrains. No clearer description of this process is avail-
able in the present literature. A Gabor filter yields textures.
By segmenting images into component texture boundaries,
search classes may be derived without resorting to textual
description. This technology thus succeeds where the
NASA effort by Rorvig et al. failed. The results reported in
this article are concrete and verifiable; indeed, anyone who
has ever traveled over Arizona highways can authenticate
these data. The authors acknowledge the contributions of
the Alexandria Digital Libraries Project, particularly the
work of Manjunath and Ma (1996), but claim their own
extensions to this work as well.
Brooks and Campbell, on the other hand, describe the
translation of interactive boolean interfaces with data to a
visual display. The “Islands” interface which they illustrate
harnesses the power of visualization to the process of com-
mercial text retrieval. It is a fact that students are still
mystified by these processes. One need only while away a
few minutes on any college campus to realize that most
persons still don’t have a clue about the meaning of term
conjunction and its impact on search results. Anyone who
has ever performed a boolean search will be able to examine
the effect of visualization on this process, and this article is
offered to permit a broad view of practice changes which
can be expected in future systems. “Islands” may not be the
ideal interface, but something like it, to paraphrase an
advertising slogan, will be “. . . coming soon to a computer
near you.”
Conference Notes and Bibliographic References
One of the landmark developments in visual retrieval
occurred at a workshop held in Zurich in the summer of
1996 in conjunction with the Association for Computing
Machinery’s Special Interest Group on Information Re-
trieval Annual Meeting. For the first time, both European
and North American interests were represented in the de-
velopment of criteria for evaluation of visual information
retrieval. Among the Europeans, the newly formed
FADIVA group played the dominant role. The workshop
report reproduced in this issue has been widely circulated,
but never before published. This conference led to the first
visualization of native TREC/Tipster data as a prelude to
formal visual information retrieval evaluation strategies
(Rorvig & Fitzpatrick, 1998; Rorvig, 1998).
For practical use in permitting users to copy this bibli-
ography, it is available through the ASIS SIGVIS website
,http://www.asis.org/SIG/SIGVIS/references.html., where
future editions may be conveniently updated.
Mark Rorvig
Lois F. Lunin
792 JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE—July 1999
References
Brooks, M. & Campbell, J. (1999). Interactive graphical queries for bib-
liographic search. Journal of the American Society for Information
Science, 50, 814 – 825.
Manjunath, B.S. & Ma, W.Y. (1996). Texture features for browsing and
retrieval of image data. IEEE Transactions on Pattern Analysis and
Machine Intelligence, 18(8), 837– 841.
Ramsey, M.C., Chen, H., Zhu, B., & Schatz, B.R. (1999). A collection of
visual thesauri for browsing large collections of geographic images.
Journal of the American Society for Information Science, 50, 826 – 834.
Rorvig, M. (in press). Scaled and visualized TREC data and query feed-
back. Information Processing and Management.
Rorvig, M. & Fitzpatrick, S. (in press). Scaled and visualized structure in
TREC IR test collection documents. Information Processing and Man-
agement.
Rorvig, M.E., Turner, C.H., & Moncada, J. (1999). The NASA image
collection visual thesaurus. Journal of the American Society for Infor-
mation Science, 50, 794 –798.
Seloff, G.A. (1990). Automated access to the NASA-JSC image archives.
Library Trends, 38, 682– 696.
Small, H. (1999). Visualizing science by citation mapping. Journal of the
American Society for Information Science, 50, 799 – 813.
About the Authors
Jennifer Campbell leads strategic projects at the Can-
ada Institute for Scientific and Technical Information
(CISTI), at the National Research Council of Canada. Ms.
Campbell has extensive experience with information man-
agement and retrieval systems, having managed a national
current awareness service and having worked intensively
with clients searching for scientific and technical informa-
tion requirements. Ms. Campbell has a B.Sc. (Hons.) in
microbiology and a master of library sciences. (At the time
of publication, Ms. Campbell is with Nortel’s International
Optical Networks, London.) Martin Brooks leads the In-
teractive Information Group at the Institute for Information
Technology, at the National Research Council of Canada.
Dr. Brooks’s research interest is development of new algo-
rithms and software architectures for modeling numerical
data, with applications to content-based multimedia re-
trieval, pattern recognition, and robotics. Dr. Brooks has a
B.Sc. in mathematics and a Ph.D. in computer science.
Marshall Ramsey is a Ph.D. student at the University of
Arizona’s Department of Management Information Systems
and a member of the UA/MIS Artificial Intelligence Group.
He received his B.S., B.A., and M.S. in MIS from the
University of Arizona in 1993 and 1997. Ramsey was
awarded a graduate fellowship from the National Library of
Medicine for a semantic analysis and retrieval project from
1996 to 1997. His research interests include semantic, cross-
media, and translingual retrieval and intelligent agents.
Hsinchun Chen is a professor of Management Information
Systems at the University of Arizona and head of the
UA/MIS Artificial Intelligence Group. He is also a visiting
senior research scientist at the National Center for Super-
computing Applications (NCSA). He received an NSF Re-
search Initiation Award in 1992, the Hawaii International
Conference on System Sciences (HICSS) Best Paper
Award, and an AT&T Foundation Award in Science and
Engineering in 1994 and 1995. He received a Ph.D. degree
in information systems from New York University in 1989.
Chen has published more than 40 journal articles covering
semantic retrieval, search algorithms, knowledge discovery,
and collaborative computing in many respected journals. He
is a PI of the Illinois Digital Library Initiative project,
funded by NSF/ARPA/NASA, 1994 –1998, and has re-
ceived several grants from NSF, DARPA, NASA, NIH, and
NCSA. He is the guest editor of IEEE Computer special
issue on “Building Large-Scale Digital Libraries” and the
Journal of the American Society for Information Science
special issue on “Artificial Intelligence Techniques for
Emerging Information Systems Applications.” Bin Zhu is a
Ph.D. student in management information systems at the
University of Arizona and a member of the UA/MIS Arti-
ficial Intelligence Group. Her research interests include
information analysis, information visualization, and human–
computer interaction. Zhu received an undergraduate degree
in meteorology in 1989 from Beijing University and an
M.S. degree in atmospheric science in 1997 from the Uni-
versity of Arizona. Bruce R. Schatz is professor in the
Graduate School of Library and Information Science at the
University of Illinois at Urbana-Champaign. He holds joint
appointments in computer science, neuroscience, and health
information sciences. He is senior research scientist at the
National Center for Supercomputing Applications (NCSA),
and at the PACI Partner for Digital Libraries he is head of
the Digital Library Research Group in the information tech-
nologies division. Schatz is the founding Director of the
CANIS (Community Architectures for Network Informa-
tion Systems) Laboratory, a unique facility which develops
fundamentally new technology for the Net and deploys it in
large-scale testbeds. Previous flagship projects have in-
cluded the Worm Community System in the NSF National
Collaboratory program and the NSF/DARPA/NASA Digital
Libraries initiative (DLI) project that built a production
testbed of federated SGML for scientific journals. Schatz is
currently the principal investigator of the flagship project in
the DARPA Information Management program, which
JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE—July 1999
funded the work discussed in this article, and is developing
the Interspace Prototype, the first large-scale analysis envi-
ronment utilizing scalable semantics.
Mark Rorvig is an associate professor at the School of
Library and Information Sciences, University of North
Texas. He has conducted research into the measurement of
information retrieval, visualization generally, and image
retrieval for the last 12 years. Charles Hudson Turner and
Jesus Moncada were students of Dr. Rorvig during his
period as an assistant professor at the University of Texas at
Austin.
Matthias Hemmje is presently a research associate at
the Integrated Publication and Information Systems Institute
(IPSI) of the German National Research Center for Infor-
mation Technology located in Darmstadt, FRG. He has been
active in the field of visual interface systems since the early
1990s, and is the principal architect of Lyberworld. His
most recent publication is “Virgilio: A Non-Immersive VR
System To Browse Multimedia Databases,” in Proceedings
of the IEEE International Conference on Multimedia Com-
puting and Systems, 1997. Dr. Hemmje has recently re-
turned to the academic world from the private sector, where
he performed considerable work on the commercialization
of visual information retrieval interfaces.
Henry Small received a joint Ph.D. in chemistry and the
history of science from the University of Wisconsin in 1971.
After a brief career as an historian of science at the Amer-
ican Institute of Physics’s Center for History and Philoso-
phy of Physics, he joined the staff of the Institute for
Scientific Information in 1972, where he is currently Direc-
tor of Contract Research. His 1973 paper in JASIS on
co-citation in the scientific literature led to numerous papers
on citation analysis and the mapping of science. His current
research centers on delineating document trails through
science. He has served on the JASIS editorial board since
1986, and in 1987 received the JASIS Best Paper Award. He
is recipient of the Derek de Solla Price Medal from the
journal Scientometrics, and is a fellow of the AAAS.
793