Empirical Studies of Information Visualization

Chapter 6

The farther back you can look,the farther forward you are likely to see.

Winston ChurchillThe purpose ofan empirical study is to discover and explain facts and factual rela-tionships.The dictionary defines the term “empirical”as meaning “relying on orderived from observation or experiment”.In philosophy ofscience,inductivismrefers to the beliefthat scientific knowledge is derived from the facts ofexperienceacquired by observation and experiment,and that science is objective (Chalmers,1982).This view first became popular due to the Scientific Revolution in the 17thcentury,especially the work ofgreat pioneering scientists such as Galileo andNewton.The foundation ofinductivism – the nature ofobservation – has beenstudied and challenged by philosophers over the past few hundred years.Chalmersdescribes the problem with inductivism as:“two normal observers viewing the sameobject from the same place under the same physical circumstances do not necessa-rily have identical visual experiences,even though the images on their respectiveretinas may be virtually identical.”In other words,our visual experience is influencedby our past experience,our knowledge,and our expectation.This in part explainswhy Galileo’s rivals could not see through Galileo’s telescope what Galileo could see,and why an X-ray expert could tell much more than a novice student from the sameX-ray photograph ofa patient.Modern philosophy ofscience has advanced muchfurther beyond inductivism in understanding the nature ofscientific knowledgeand scientific methods.Interested readers are recommended to plunge into the vastliterature ocean ofphilosophy ofscience.In this chapter,we will restrain ourselvesand focus on empirical studies ofinformation visualization.A study is countedas an empirical study ifit explores and analyzes data associated with the use of information visualization functions as well as systems.Until recently,the numberofempirical studies ofinformation visualization techniques and applications hasbeen rather small in comparison to the overall growth ofthe field,but the situationis changing.

6.1Introduction

The 1990s witnessed leaps and bounds in the ﬁeld ofinformation visualization withincreasingly powerful techniques and visually appealing information visualizationartifacts (Card et al.,1999;Chen,1999a;Ware,2000;Spence,2001).Research in infor-mation visualization has been traditionally dominated by sophisticated and eye-popping innovations ofvisual representations and technical mechanisms.Incontrast,empirical evaluations ofinformation visualizations are often overshadowedby the enthusiasms for what can be done rather than for what should be done.

George Robertson,the inventor ofthe classic Cone Trees (Robertson et al.,1991),raised the issue in his keynote speech at the 1998 IEEE InformationVisualization Symposium.Evaluations ofinformation visualization techniquesbegan to emerge,but the pace needs to be faster and the probe needs to be deeper.In 2000,Iand Mary Czerwinski guest edited a nine-article special issue on empiricalevaluations ofinformation visualizations in

International Journal ofHuman

–

Computer Studies

,featuring two thematic streams:(1) evaluations ofclassic infor-mation visualization techniques and (2) applications ofinformation visualizationtechniques in practical settings.Also in 2000,we guest edited another special issuein

Journal ofthe American Society for Information Science

on individual differencesin virtual environments.For a fast-advancing ﬁeld like information visualization,empirical evaluationsare particularly important.Empirical evidence is an integral part ofour domainknowledge,especially on what works,what failed,or what remains unknown.Thisis also a good opportunity for a fast-expanding ﬁeld to incorporate valuable experi-ences and adapt established methodologies from such relevant disciplines asHuman–Computer Interaction (HCI) and psychology.Indeed,a substantial pro-portion ofempirical studies have appeared in HCI-related journals and confer-ences.Obviously,the home discipline ofan application domain should always getinvolved.In this chapter,we introduce some ofthe most representative empirical studiesofinformation visualization,not only to outline the proven knowledge ofthedomain,but also to stimulate more empirical evaluations ofinformation visualiza-tions.Every empirical study addresses issues concerning users,tasks,visualizationdesigns,and the visualized information,although individual studies may differ interms oftheir focus and approaches.The organization ofthis chapter essentially follows this sequence:1Users,including cognitive factors such as spatial ability,spatial memory,andassociative memory,as well as the effects ofgender and prior knowledge.2Tasks,including visual search tasks,visual scanning,detecting shortest paths, judging visualized quantitative information.3Visual Features,including focus



context views,2D versus 3D.4Visualized Information,including hyperlinks ofa website and networks of documents.Figure 6.1 is a co-citation network derived from empirical studies ofinformationvisualization between 1993 and 2003.The visualization was generated by CiteSpace(see Chapter 8).A few pivot points in the network are worth noting.The Cone Treespaper by Robertson et al.(1991) is located in the center ofthe map,joining net-works from four ofthe six time slices.Shneiderman’s 1992 tree maps paper is alsoa pivot point,connecting three slices since 1990.Chernoff’s (1973) paper connectstwo slices.So do Inselberg and Dimsdale’s 1990 paper on parallel coordinates andLeBlanc,Ward,and Wittels 1990 paper.Tufte’s 1990 book connects three time slicesbetween 1997 and 2002.Such images ﬁltered by citation and co-citation strengthsdirectly reﬂect where the focus ofempirical studies has been.We begin with a meta-analysis ofempirical studies ofmainly visual informationretrieval tasks,followed by more generic perceptual tasks.We attempt to maintainfocus on such tasks as the starting point ofeach empirical study.Non-experimentalstudies are also introduced in this chapter.The intended message is that empiricalstudies should take a broader view.

174Information Visualization

6.2Meta-analysis

A number offundamental issues must be addressed for the further development of the information visualization field.What is the central research question that moststudies aim to address? What is the optimal task–feature taxonomy for informationvisualization design? What is the most commonly used experimental design? Isthere any consensus that one can draw from the existing empirical findings in theliterature? What is the most powerful visualization feature for a given task? To whatextent are the current empirical findings consistent across different studies?The subsequent meta-analysis used the same methodology as the meta-analysisofhypertext systems (Chen and Rada,1996).This meta-analysis focuses on thefield ofinformation visualization.A comprehensive report ofthe meta-analysiscan be found in (Chen and Yu,2000).As we will see,this is only the first step tobuilding a task–feature taxonomy that can accommodate empirical evidence con-cerning various information visualization technologies.

6.2.1Sampling Empirical Studies

The meta-analysis focuses on experimental studies that include at least one ofthethree types ofindependent variables,users,tasks,and tools.User-related variablesare mainly represented by individual differences in terms ofcognitive factors;tool-related variables include a variety ofinformation visualization features.Measuresto do with users include several cognitive factors such as associative memory (MA),spatial ability (VZ) and visual memory (MV).However,because ofthe smallnumber ofpapers that directly address these cognitive factors,they are excludedfrom the meta-analysis.Instead,we will discuss some ofthese studies in the secondhalfofthis chapter.