Welcome to Scribd, the world's digital library. Read, publish, and share books and documents. See more
Download
Standard view
Full view
of .
Look up keyword
Like this
1Activity
0 of .
Results for:
No results containing your search query
P. 1
5044

5044

Ratings: (0)|Views: 7|Likes:
Published by Neri David Martinez

More info:

Published by: Neri David Martinez on Feb 09, 2013
Copyright:Attribution Non-commercial

Availability:

Read on Scribd mobile: iPhone, iPad and Android.
download as PDF, TXT or read online from Scribd
See more
See less

11/05/2013

pdf

text

original

 
Opinion Retrieval in Twitter
Zhunchen Luo
College of ComputerNational University of Defense Technology410073 Changsha, Hunan, CHINA
 zhunchenluo@nudt.edu.cn
Miles Osborne
School of InformaticsThe University of EdinburghEH8 9AB, Edinburgh, UK
miles@inf.ed.ac.uk 
Ting Wang
College of ComputerNational University of Defense Technology410073 Changsha, Hunan, CHINA
tingwang@nudt.edu.cn
Abstract
We consider the problem of finding opinionated tweets abouta given topic. We automatically construct opinionated lexicafrom sets of tweets matching specific patterns indicative of opinionated messages. When incorporated into a learning-to-rank approach, results show that this automatically opin-ionated information yields retrieval performance comparablewith a manual method. Finally, topic-related specific struc-tured tweet sets can help improve query-dependent opinionretrieval.
Introduction
Twitter is a popular online social networking service wherepeople often share information or opinions about personal-ities, politicians or products. Most existing work on opin-ion in Twitter concentrates on analysing opinions expressedin tweets for a given topic. To the best of our knowledgethere is no work on actually finding opinionated tweets. Inthis paper, we present the first study of opinion retrieval inTwitter. Relevant opinionated tweets should satisfy two cri-teria: (1) be relevant to the query and (2) contain opinions orcomments about the query, irrespective of being positive ornegative.Search in Twitter can be harder than traditional search,largely due to tweets being often very short, and/or lackingin reliable grammatical style and quality. These factors re-duce the effectiveness of opinion mining based upon tradi-tional NLP techniques. Twitter also presents interesting op-portunities for retrieval. The rich environment presents uswith a myriad of social information over-and-above just us-ing terms in a post (for example author information such asthe number of posts) all of which potentially can improve on(opinion) retrieval performance. Additionally conventionshave emerged in Twitter which structure tweets and thisstructuring can be a valuable hint when retrieving opinion-ated tweets. As an example, people usually add a commentbefore the convention “RT @usename” and many of thesetweets are likely to be subjective. Importantly this structuralinformation is topic independent.In this paper, we use a standard machine learning ap-proach to learn a ranking function for tweets that uses the
Copyrightc
2012, Association for the Advancement of ArtificialIntelligence (www.aaai.org). All rights reserved.
available social features and opinionated feature in addi-tion to traditional topic-relevant features such as the BM25score. The experimental result shows that our ranking func-tion is significantly better than a BM25 baseline for opin-ion retrieval (improving MAP by 60.22%). Additionally wepropose a novel approach, using the social information andstructural information of the tweets, to automatically gener-ate a large number of accurate “pseudo” subjective tweets(PSTs) and “pseudo” objective tweets (POTs). These twotweet sets can be used as a corpus to derive lexicons for es-timating the opinionatedness of a new tweet. We show thatour approach can achieve comparable performance with amethod which using manual tagged tweets corpus.
Related Work
Opinion retrieval in blogs and web documents has beenstudied in depth. Eguchi and Lavrenko (2006) firstly intro-duced opinion ranking formula which combine sentimentrelevance models and topic relevance models into a gen-eration model. This formula was shown to be effective onthe MPQA corpus. Zhang and Ye (2008) and Huang andCroft (2009) also put forward their own way to unify senti-ment relevance models and topic relevance models for rank-ing. Gerani, Carman and Crestani (2009) firstly investigatedlearning-to-rankforblogposts.Allofthisworkisinthecon-text of blogs or web documents, Twitter, however, is a noveldomain and its rich social environment should be consideredwhen modeling relevance.In opinion retrieval estimating the opinionatedness of adocument is essential. He et al. (2008) proposed an approachto calculate opinionatedness of a document based on sub- jective terms. These terms are automatically derived frommanual tagged data. Jijkoun, de Rijke and Weerkamp (2010)present a method for automatically generating topic-specificsubjective lexicons based on extract syntactic clues of man-ual tagged data. Unlike the work introduced above, Zhang,Yu and Meng (2007) used the reviews of some websites as asource of “pseudo” subjective sentences, and the Wikipediadocuments as an external source of “pseudo” objective sen-tences. They assume that the subjective portion should bedominant in the reviews so that the effect of the objectiveportion can be neglected. The situation is the opposite inWikipediadocuments. They then used these datasets to builda SVM sentence classifier to estimate the opinionatedness
507
Proceedings of the Sixth International AAAI Conference on Weblogs and Social Media
 
of a document. An approach based on similar idea we alsoproposed in the context of Twitter, which using social infor-mation and structural information to automatically generate“pseudo” subjective tweets and “pseudo” subjective tweets,for opinion retrieval in Twitter.
Approach
Learning to Rank Framework
Learning to rank is a data driven approach which effectivelyincorporates a bag of features into the retrieval process. Abag of features, related to the relevance of a tweet, are ex-tracted from tweets that have been labelled as being relevant.RankSVM (Joachims Thorsten, 1999) isused to train a rank-ing model.
Social Features
The following features capture useful aspects of Twitter andauthors for opinion retrieval:
Twitter Specific Features
. We use the following featuresrelated to the tweet itself:
Mention
,
URL
,
Hashtag
. In atweet, people usually use “@” preceding a user name toreply other user (Mention). The text of this tweet is morelikely to be “personal content”. Previous work shows that“personal content” is on the whole more likely to containopinions than “official content” (Gerani et al. 2011). There-fore, we use a binary feature indicating whether the tweetcontains “@username” for tweet opinion retrieval. Sharinglinks in tweets is very popular in Twitter. Most tweets con-taining a link usually give an objective introduction to thelinks (e.g., tweets posted by the BBC News). Additionally,spam in Twitter often contains links. Hence, we use a fea-ture indicating whether a tweet contains a link in our rank-ingmodel.Ahashtagreferstoawordinthetweetthatbeginswith the “#” character. It is used to indicate the topic of thetweet. We use a binary feature whether the tweet contains ahashtag in our system.
Author Features
. We use the following features relatedto the author of the tweet:
Statuses
,
Followers
,
Friends
,
Listed
. The number of tweets (statuses) the author has everwritten is related to the activeness of the author. Intuitively,the most active authors are likely to be spammers who postvery large number of tweets. Therefore, we use the numberof statuses as a feature for tweets ranking. The number of followers indicates the popularity of the user. For example,the news media users usually have more followers than nor-mal users. The number of friends also reflects the type of the user. For example, spammers often have large number of ‘friends’. We develop these two features for tweets retrieval.A user can group their friends into different lists accordingto some criteria (e.g., the topic and social relationship). If a user is listed many times, it means that his tweets are in-teresting to a larger user population. We use a feature thatmeasures how many times the author of a tweet has beenlisted for tweet ranking.
Opinionatedness Feature
Obviously estimating the opinionatedness score of a tweet isessential for opinion retrieval task. We adopt a lexicon-based
t
¬
t
Row totalSub. set
O
11
O
12
O
1
Obj. set
O
21
O
22
O
2
Col. total
O
1
O
2
O
Table 1: Table for pearson’s chi-square.
O
1
=
O
11
+
O
12
;
O
2
=
O
21
+
O
22
;
O
1
=
O
11
+
O
21
;
O
2
=
O
12
+
O
22
;
O
=
O
11
+
O
12
+
O
21
+
O
22
.approach, since it is simple and not dependent on machinelearning techniques. However, a lexicon such as the MPQASubjectivity Lexicon
1
which is widely used might not be ef-fective in Twitter, since the textual content of tweet is oftenvery short, and lacks reliable grammatical style and quality.Therefore, we use a corpus-derived lexicon to construct anopinion score for each tweet. We estimate the opinionated-ness score of each tweet by calculating the average opinionscore over certain terms. We use the chi-square value, basedon manual tagged subjective tweets set and objective tweetsset, to estimate the opinion score of a term. The score mea-sures how dependent a term is with respect to the subjectivetweets set or objective tweets set. For all terms in a tweet, weonly keep the terms with a chi-square value no less than
m
when computing the opinion score. The estimated formulaas follows:
Opinion
avg
(
d
) =
t
d,χ
2
(
t
)
m
 p
(
t
|
d
)
·
Opinion
(
t
)
where
p
(
t
|
d
) =
c
(
t,d
)
/
|
d
|
is the relative frequency of a termt in tweet d.
c
(
t,d
)
is the frequency of term t in tweet d.
|
d
|
is the number of terms in tweet d.
Opinion
(
t
) =
sgn
(
O
11
O
1
O
21
O
2
)
·
χ
2
(
t
)
where
sgn
(
)
is sign function.
χ
2
(
t
)
calculates chi-squarevalue of a term.
χ
2
(
t
) =(
O
11
O
22
O
12
O
21
)
2
·
OO
1
·
O
2
·
O
1
·
O
2
O
ij
in Table 1 is counted as the number of tweets havingterm t in the subjective/objective tweets set respectively. Forexample
O
12
is the number of tweets not having term t inthe subjective tweets set.Manually labelling the tweets necessary for construct-ing opinionated scoring is time-consuming and also topic-dependent. For example, tweets about “android” mightcontain opioninated terms “open”, “fast” and “excellent”,but these terms are unlikely to be the subjective clues of tweets related to some news event (e.g., “UK strike”). It isclearly impossible to tag a large number of tweets for everygiven topic. Therefore, we develop an approach to collect“pseudo” subjective tweets (PSTs) and “pseudo” objectivetweets (POTs) automatically.In Twitter, some simple structural information of tweetsand users’ information can be used to generate PSTs and
1
http://www.cs.pitt.edu/mpqa/
508
 
POTs. For example people usually retweet another user’stweet and give a comment before this tweet. Tweets withthis structure are more likely to be subjective. Many tweetsposted by news agencies are likely to be objective tweets andthese tweets usually contain links. We define these two typesof tweets as follows:1)
“Pseudo” Subjective Tweet (PST):
a tweet of the form“RT @username” with text before the retweet. For exam-ple, a tweet “
 I thought we were isolated and no one would want to invest here! RT @BBCNews: Honda announces500 new jobs in Swindon
bbc.in/vT12YY 
” is a Pseudosubjective tweet.2)
“Pseudo” Objective Tweet (POT):
If a tweet satisfiestwo criteria: (1) it contains links and (2) the user of thistweet posted many tweets before and has many follow-ers. This tweet is likely to be an objective tweet. E.g.,
#NorthKorea:#KimJongil died after suffering massiveheart attack on train on Saturday, official news agencyreports
bbc.in/vzPGY5
”.Using the definition introduced above, it is easy for us todesignpatternsandcollectalargenumberofPSTsandPOTsfrom Twitter. We assume that the tweets in the PST set areall subjective tweets and the tweets in the POT set are all ob- jective tweets. Although this is not 100% true, the subjectivetweets portion should be dominant in the PST set so that theeffect of the objective tweets portion can be neglected. It isopposite in the POT set. Since the structural information andauthors’ information are independent of the topic of a tweet,if there are a lot of tweets related to a given topic, it is easilyto collect topic-dependent PSTs and POTs.
Experiments
Dataset and Experimental Settings
To the best of our knowledge, there is no annotated datasetfor opinion retrieval in Twitter. Therefore, we created a newdataset for this task 
2
. We crawled and indexed about 30 mil-lion tweets using the Twitter API in November 2011. Alltweets are English. Using these tweets we implemented asearch engine. Seven people (a woman and six men) wereasked to use our search engine. They were allowed to postany query. Given a query the search engine would present alist of 100 tweets ranked based on the BM25 score. Basedon the principle about the tweet whether expresses opinionabout a given query, people assigned a binary label to everytweet. Finally we totally collected 50 queries and all judgedtweets. The average query length was 1.94 words and theaverage number of relevant tweets per query was 16.62.For learning to rank, SVM light
3
which implements theranking algorithm is used. We use a linear kernel for train-ing and report results for the best setting of parameters. Inorder to avoid overfitting the data we perform 10 fold cross-validation in our dataset. And we use
Mean Average Preci-sion
(MAP) as the evaluation metric.
2
This dataset is available at
https://sourceforge.net/projects/ortwitter/
3
http://svmlight.joachims.org/
Results
We first investigate whether social features can improveopinionretrievalinTwitter.Asabaseline,weusetherankingapproach which uses the Okapi BM25 score of each tweetas a features for modeling. We combine each social featurewith the
BM25
feature within our tweet ranking system. Ta-ble 2 shows the performance of each ranking model. Wecan see that using
Mention
,
URL
,
Statuses
and
Followers
features significant improves the results when used with thebaseline (BM25) in isolation. It suggests some social infor-mation can indeed help opinion retrieval in Twitter. We seethat the
URL
feature is the most effective feature, perhapsbecause most textual content in these tweets are objectiveintroductions. Also, spammers usually post tweets includ-ing links and features dealing with links might help reducespam. The effect of 
URL
,
Statuses
and
Followers
featuresfor tweets ranking also supports our approach of using socialinformation and structural information to generate “pseudo”objective tweets. The improvement of ranking result using
Mention
feature supports the idea that “personal content” ison the whole more likely to contain opinions than “officialcontent” (Gerani et al. 2011).Next we investigate the opinionatedness feature for tweetsranking. To automatically generate PSTs and POTs, we de-sign some simple patterns: For PSTs generation, we choosethe tweets uses the convention “RT @username”, with textbefore the first occurrence of this convention. Additionallywe find that the length of the preceding text should be noless than 10 character. For POTs generation, we choose thetweets which contain a link, the author for each tweet hasno less than 1,000 followers and has posted at least 10,000tweets. In our one month tweets dataset, 4.64% tweets arehigh quality PSTs and 1.35% tweets are POTs.We spot-checked the quality of our automatically har-vested tweets by randomly selected 100 PSTs and 100 POTsand manually inspecting them, judging the extent to whichthere were subjective or objective. In these tweets, 95%PSTs were subjective tweets and 85% POTs were objectivetweets. This supports the idea that our approach can gener-ate a large number of accurate PSTs and POTs. Hence, werandomly choose 4500 English PSTs and POTs to form atopic-independent dataset.Another advantage of our approach is that it is easy togather topic-dependent PSTs and POTs. We use all PSTsand POTs introduced above to implement a search engine.Given a query, the search engine can give any number of query-dependent PSTs and POTs ranked by BM25 score.We generate 4500 query-dependent PSTs and POTs for eachquery. In our corpus-derived approach, we use the PorterEnglish stemmer and stop words to preprocess the text of tweets. Using these tweet datasets we can calculate the valueof opinionatedness score for a new tweets. To achieve thebest performance of tweets ranking, we set the threshold of 
m
is5.02 correspondingtothesignificancelevelof 0.025foreach term in dataset. This setting is the same as Zhang, Yuand Meng (2007)’s work. We call the feature using topic-independent dataset to estimate the opinionatedness scoreas
Q I
feature and using topic-dependent datasets as
Q D
feature. Previous work uses manual tagged tweets to esti-
509

You're Reading a Free Preview

Download
scribd
/*********** DO NOT ALTER ANYTHING BELOW THIS LINE ! ************/ var s_code=s.t();if(s_code)document.write(s_code)//-->