Figure 3. An example analysis microtask shows a single chart (a)along with chart-reading subtasks (b) an annotation subtask (c) anda feature-oriented explanation prompt designed to encourage workersto focus on the chart (d). A request for outside URLs (e), encouragesworkers to check their facts and consider outside information.
Hypothesis(orexplanation)generationisakeystepofPirolliand Card’s sensemaking model and it requires human judgment. Developing good hypotheses often involves gen-erating a diverse set of candidate explanations based on un-derstanding many different views of the data. Our tech-niques allow an analyst to parallelize the sensemaking loopby dividing the work of generating and assessing hypothe-ses into smaller
and efﬁciently distributing thesemicrotasks across a large pool of workers.We propose a four-stage workﬂow (Figure1)for crowd-sourcing data analysis. An analyst ﬁrst
rele-vant to a speciﬁc question they have about the data. Crowdworkers then examine and explain these charts in
. Optionally, an analyst can ask other workers toreview these explanations in
. Finally, theanalyst can
view the results
of the process, sorting and ﬁlter-ing the explanations based on workers’ ratings. The analystmay also choose to iterate the process and add additionalrounds of analysis and rating to improve the quality and di-versity of explanations.
Given a dataset, an analyst must initially select a set of chartsfor analysis. The analyst may interactively peruse the datausing a visual tool like Tableau to ﬁnd charts that raisequestions or warrant further explanation. Alternatively, theanalyst may apply data mining techniques (e.g., [10, 16, 33])to automatically identify subsets of the data that require fur-ther explanation. In general, our workﬂow can work withany set of charts and is agnostic to their source.In our experience, analysts often know
that they areinterested in understanding speciﬁc features of the data suchas outliers, strong peaks and valleys, or steep slopes. There-fore, our implementation includes R scripts that apply basic
Figure 4. An example rating microtask showing a single chart (a) alongwith explanations (b) from several workers. The task contains a chart-reading subtask (c) to help focus workers’ attention on the charts anddeter scammers, along with controls for rating individual responses (d),indicating redundant responses (e), and summarizing responses (f).
data mining techniques to a set of time-series charts in orderto identify and rank the series containing the largest outliers,the strongest peaks and valleys and the steepest slopes. Theanalyst can then review these charts or post them directly tocrowd workers to begin eliciting explanations. We leave itto future work to build more sophisticated data mining algo-rithms for chart selection.
For each selected chart, our system creates an analysis mi-crotask asking for a paid crowd worker to explain the visualfeatures within it. Each microtask contains a single chartand a series of prompts asking the worker to explain and/orannotate aspects of the chart (Figure3). The analyst canpresent each microtask to more than one worker to increasethe diversity of responses.
If a large number of workers contribute explanations, the an-alyst may not have the time to read all of them and mayinstead wish to focus on just the clearest, most plausible ormost unique explanations. In the rating stage the analystenlists crowd workers to aid in this sorting and ﬁltering pro-cess. Each
(Figure4)includesasinglechartalong with a set of explanations authored by other workers.Workers rate explanations by assigning each a binary (0-1)
score based on whether it explains the desired fea-ture of the chart. Workers also rate the clarity (how easy itis to interpret) and plausibility (how likely it is to be true)of each response on a numerical (1-5) scale. We combinethese ratings into a numerical
score (0-5) that mea-