, from the years 1991 to 2008. This consistsof 33K articles of varying lengths including newsbulletins, full-length magazine features, and opin-ion columns. We obtained the articles via an institu-tional subscription, and reformatted them in XML
.Certain IT concepts are particularly signiﬁcant inthecontextof thesocial scienceapplication. Our tar-get list consists of 59 IT innovations and concepts.The list includes plurals, common variations, andabbreviations. Examplesof IT conceptsinclude“en-terprise resource planning” and “customer relation-ship management”. To avoid introducing confound-ing factors into our results, we only include explicitmentions and omit pronominal coreference.
3 User interface
Our user interface (ﬁgure 1) uses a drag-and-dropprocess through which workers make decisionsabout whether particular highlighted words withina given sentence reﬂect an opinion about a particu-lar mentioned IT concept or innovation. The useris presented with a sentence from the corpus sur-rounded by some before and after context. Under-neath the text are four boxes: “No effect on opin-ion” (
), “Affects opinion positively” (
),“Affects opinion negatively” (
), and “Can’ttell” (
).The worker must drag each highlighted word inthe sentence into one of the boxes, as appropriate. If the worker cannot determine the appropriate box fora particular word, she is expected to drag this to the
box. The worker is presented with de-tailed instructions which also remind her that mostof words in the sentence are not actually likely to beinvolved in the expression of an opinion about therelevant IT concept
. The worker is not permittedto submit the task without dragging all of the high-lighted words to one of the boxes. When a wordis dragged to a box, the word in context changescolour; the worker can change her mind by clickingan X next to the word in the box.
We will likely be able to provide a sample of sentence dataannotated by our process as a resource once we work out docu-mentation and distribution issues.
We discovered when testing the interface that workers canfeel obliged to ﬁnd a opinion about the selected IT concept. Wereduced it by explicitly reminding them that most words do notexpress a relevant opinion and by placing the
We used CrowdFlower to manage the task withAmazon Mechanical Turk as its distribution chan-nel. We set CrowdFlower to present three sentencesat a time to users. Only users with USA-based IPaddresses were permitted to perform the ﬁnal task.
In this section, we discuss the data processingpipeline (ﬁgure 3) through which we select candi-dates for annotations and the crowdsourcing inter-face we present to the end user for classifying indi-vidual words into categories that reﬂect the effect of the word on the worker.
4.1 Data preparation4.1.1 Initial annotation
Two social science undergraduate students werehired to do annotations on
withthe original intention of doing all the annotationsthis way. There was a training period where they an-notated about 60 documents in sets of 20 in iterativeconsultation with one of the authors. Then they weregiven 142 documents to annotate simultaneously inorder to assess their agreement after training.Annotation was performed in Atlas.ti, an anno-tation tool popular with social science researchers.It was chosen for its familiarity to the social sci-entists involved in our project and because of theirstated preference for using tools that would allowthem to share annotations with colleagues. Atlas.tihas limitations, including the inability to create hier-archical annotations. We overcame these limitationsusing a special notation to connect related annota-tions. An annotator highlights a sentence that shebelieves contains an opinion about a mentioned tar-get on one of the lists. She then highlights the men-tion of the target and, furthermore, highlights the in-dividualwordsthatexpresstheopinionaboutthetar-get, using the notation to connect related highlights.
4.1.2 Candidate selection
While the use of trained annotators did not pro-duce reliable results (section 6.2) in acceptable timeframes, we decided to use the annotations in a pro-cess for selecting candidate sentences for crowd-sourcing. All 219 sentences that the annotators se-lected as having opinions about within-sentence IT