2015 International Conference on Intelligent Computing and Internet of Things (IC1T)
One Methodology for Spam Review Detection Based
on Review Coherence Metrics
Xinkai Yang
College of Information, Mechanical and Electronical Engineering, Shanghai Normal University
Shanghai, China, 200234
E-mail: xkyang@shnu.edu.cn
Abstract-In this paper, we propose an iterative computation
framework to detect spam reviews based on coherent
examination. We first define some reviews' coherent metrics to
analyze review coherence in the granularity of sentence. Then
the framework and its evaluation process are discussed in details.
Keywords- spam review detection; coherence metric; word
transition probability; word concurrence probability
I. INTRODUCTION
Nowadays product (including service, e.g. hotel, restaurant.)
reviews play an important role for consumers' online shopping
activities. Many people read these reviews before making
purchase decision. Normally, the reviews for a product may be
positive or negative. Customers will pay more attention to the
products with positive reviews and avoid the negative ones.
This effect will bring larger amount of business or lead to
potential financial losses [9]. Although lots of product reviews
are posted by real consumers to express their views and share
their shopping experience with other people, more untruthful
reviews appear in the e-business web sites because of financial
reasons. For example, supposing a lodger writes a very
negative review for a certain hotel on a review website due to
its bad service. This review will present an unfavorable
impression of this hotel to its potential customers and damage
its business. In order to avoid business losses caused by this
kind of truthful reviews, the owner of this hotel could employ
or entice some people to write untruthful reviews to promote its
reputation. Because of these pervasive untruthful reviews,
customers are easy to be misled to buy low-quality
services/products, while decent services/products could be
defamed by malicious reviews. Usually, the writers who post
these deceptive reviews to intentionally mislead consumers or
opinion analysis systems are called spammers and the
untruthful reviews are called spam reviews [9]. In order to
show how difficult spam detection is, we first depict one
example reviews in the following part, which was posted to
ctrip.com (English version). This review is posted by one
person who tries to promote that hotel and it is hard to identify
it as one deceptive review:
"If you can put up with the small size of the room (which
is normal in Japan) then you will love this place. Five minute
walk from the main station, modem clean room, and great
978-1-4799-7534-1114/$31.00 ©2015 1EEE
breakfast with espresso machine, outstanding service, and
great price. Can't fmd a thing to complain about it. "
The rest of this paper is organized as follows. In Section II,
some related research works on spam reviews detection are
discussed. Based on coherence examination, we propose a
general framework in Section Ill, followed by the coherent
metrics defined in Section IV. We discuss our evaluation
process in Section V and conclude this paper in section VI.
IT. RELATED WORKS
Mostly, the features of review contents and reviewers'
behaviors are used in early research works to detect spam
reviews [9]. By representing a review using a set of review­
level, reviewer-level and product-level features, a supervised
classification method is applied to detect three types of fake
reviews in ref. [4]. Duplicated reviews are first identified using
the Jaccard ratio, and these reviews are then used as the
training examples for the logistic regression classifier [4, 7]. In
particular, untruthful review detection is performed by using
duplicate reviews as labeled data.
Opinion and sentiment analysis of reviews extract and
aggregate positive and negative opinions from product reviews
[5]. Researchers study the problem of generating feature-based
summ aries of customer reviews of products sold online. Given
a data set of customer reviews of any particular product, the
task involves three subtasks: (1) identifying features of product
that customers have discussed or expressed their opinion on. (2)
For each feature, identifYing review sentences which gives an
opinion (positive or negative) and (3) producing a summary
using the discovered information [5].
The other works study spam reviews by detecting
fraudulent or unfair ratings. The review ratings are clustered
into unfairly high ratings and unfairly low ratings by using
third party ratings products [4]. Those reviews with unfair
ratings can be removed to restore a fair item evaluation system.
A new classification approach is developed to solve helpful
prediction using review content and rating features. The
reviews' rating features include the difference between the
review rating and the average rating of all reviews of the
product. These features are then used to train a helpful
classification method [4]. These works do not address review
99 Harbin, China
spammer detection directly. Some of them do not conduct
evaluation of their techniques on real data.
Xie and others tries to identify singleton reviews which are
the only one review written by each user. A correlated temporal
pattern is proposed to detect such attacks via multidimensional
time series based on aggregate statistics, in order to depict and
mine such correlations [9]. In this way, they map the problem
of singleton spam detection to an abnormally correlated pattern
detection problem. Then a hierarchical algorithm is proposed to
detect the time interval when such singleton spam attacks are
likely to happen.
Recently, a more sophisticated graph-based method is
proposed in ref. [6] to detect review spammers by considering
the relationships among reviews, reviewers and reviewed stores.
This kind of review spammer detection is preferred because
gathering behavioral evidence of spammers is much easier and
scalable than that of spam reviews. A heterogeneous graph to
capture such relationships is constructed based on thorough
analysis [6].
Ill. OUR MODEL
Here we propose a general framework to detect spam
reviews based on coherence examination. The assumptions we
used in the proposed model is discussed firstly. Then several
reviews' coherent metrics are defmed. The iterative
computation framework based on these defmitions is provided
fmally.
Usually spammers will benefit from their activity by
promoting some specific stores' prominence or defaming other
stores. The spam reviews have inner connections with certain
stores, which are either positive or negative depending on the
spammer's intention. Untruthful spam reviews always deviate
from the truth since that lousy stores suffer if customers know
the truth about them [2]. So the spam reviews will bear subtle
semantic incoherence among sentences in the reviews of the
same store. The main opinions shared by most surrounding
reviews, rather than outliers' opinion are helpful for customers
when they read reviews online.
In order to detect such spam reviews which are inconsistent
with the surrounding reviews, we propose a model to analyze
review coherence in the granularity of sentence. Intuitively,
spam reviews always use some untruthful sentiment words to
express their deceptive feelings, to promote or defame a store
through such positive or negative feedback. These sentiment
words are linked with the targeted stores in the spam reviews.
So it is crucial to find the coherent relationships between the
specific store and the sentiment words among them.
Based on the above assumptions, we consider each review
as a bag of sentences, and define a sentence set as S {Sj. S2.
=
S3, .... sn } . For each sentence, we capture the intra-sentence
information through the store-sentiment word pair and the
coherent features which are pair-wise sentence coherence. In
addition, we construct a sentiment word lexicon Vo. We also
100
assume that the review set D = {dj, d2, d3 ... dn} needs to be
detected.
Defmition 1. Store-sentiment word pair Pi! consists of two
elements, one is from the online store set Sf, and the other one
is from the sentiment word set WOo
Pi! {<tj, OJ> I <tjESt, OJ E �J
=
(1)
This word pair is used to maintain the associative
information between a specific store and the sentiment word.
We can infer that spammers will use positive sentiment words
to praise less reliable stores, or use negative sentiment words to
unreasonably complain about more reliable stores. This kind of
store-sentiment word pair may be helpful to find the clues of
such suspicious phenomena.
We adopt a model containing hubs and authorities, which
have some similarity with the popular graph ranking algorithm
[12]. There are two layers in this proposed model. The word
pair layer is considered as hubs and the review layer as
authorities. Fig.2 gives the bipartite graph representation of this
model. If a word pair occurs in one sentence of a review, there
will be an edge between them. In Fig.l, the word pair that has
links to many reviews can be assigned a high weight to denote
a strong associative degree between the store and a sentiment
word.
. . . . . . . (0
Fig. 1. Link graph
We denote the bipartite graph as G <Hp, Rd. Edp>, where
=
Hp {Pij}
= is the set of all store-sentiment word pairs, Rr{dk}
is the set of reviews, Edp {eij I pi!EHp, dkERd} corresponds to
=
the connection between reviews and store-sentiment word
pairs. Each edge ei! is associated with a weight w/E [0, 1]
denoting the contribution of Pi! to the review dk • The weight
w / is computed by the contribution of word pair Pi! in all
sentences of dk• Further metrics will be defined to investigate
the coherence between these sentiment words and other related
words to find more information.
IV. COHERENCE METRICS AND ALGORITHM
In this part we provide several metrics to measure the
coherence of a review based on the flow smoothness
information between sentences: 1) Word transition probability
- conditional probability; 2) Word concurrence probability -
joint probability. Then the framework is provided.
A. Word transition probability
Usually, human writings will demonstrate certain word
transition patterns naturally between two consecutive
sentences. When a word is given in one sentence, certain
words could be observed in its following sentence with some
probability [3]. However, such transition patterns in spam
reviews can be impaired due to their deceptive nature. The
consecutive sentences sand s' are denoted as a pair (s, s'). The
symbol S, represents the sentence set formed by s'. Let f(w, SJ
denote the frequency of W in Sj. Given a set of words W, we
can define one step transition probability p(Wj I wJ as the
probability of observing Wj in one sentence when the word Wj
appears in the previous sentence.
f(wj,SJ
p (Wj I wJ =
Lw W f(w ,SJ
k
kE
For any two sentences s I and s2, the transition probability
to observe S2 after s], i.e. s] S2, can be estimated by
--
choosing the edge with the highest one step transition
probability since s] and S2 have multiple words.
P(SI � S2 ) = max p (Wj I wj)· f(Wj,S2) (3)
If;E5,'l,lf E5,'z
j compute Con(r) using (7);
Based on the above defmition, we provide a coherent
metric of a review r in the following. This metric can be used
to detect spam reviews. We assume the review r includes
sentences {sJ, S2, S3, "',Sn }. So its coherent metric H(r) can be
defined as:
f loge pes1 --+ s 2 --+ ... --+ S n )) } (4)
H (r -)
-
exp, -
n-I
I i�1 1 S'+1 I
In the spam reviews, the word transition patterns can be
impaired; their coherent values will become larger than those
on truthful reviews. In order to make the calculation process
simple, we have one assumption that the transition probability
between two consecutive sentences is independent of each
other, so we can get the following formula.
L :'�-lllog( P(Si ---+ Si+I))
H(r)= exp{-
",,-I }
L... i�11 Si+1 I
B. Word concurrence probability
We first define three probabilities between two
consecutive sentences which have some similarity with ref. [3]:
the probability of observing word w, (denoted as 0, ), the
probability of observing word Wj (denoted as OJ ), and the
probability of observing word Wj in one sentence and word Wj
in the other (denoted as Ojj ) [3]. We defme the word
concurrence metric for word Wj and Wj as log(Ojj / OJ OJ). This
metric does not only measure the concurrence probability in
one consecutive sentence pair, but also include the situation
that the same word might naturally con-occur with many other
words. The concurrence possibility of the w, and wi in the
same sentence is also considered.
Then the sentence concurrence measure for two
consecutive sentences can be defined by using the highest
word concurrence measure:
OJ J
Con(spsj+l) max log(--'- ) (6)
=
lV;E,Sl,lV E,S2
j GiD;
The abnormal word concurrence phenomena in spam
reviews could be observed normally. For example, the fixed
pattern of two associative words in a spam review can hardly
be found in truthful reviews. Therefore, the value of coherent
metric for spam reviews is often lower than that of truthful
reviews. So this coherent metric is also very helpful to detect
spam reviews and it can be jointly used with the former one.
1 "n-l
Can ( r) = -- L- 1-1 Can (S"S'+I)
-
(7)
n -I
(2)
ALGORITHM: Coherence Metrics Computation Process
Input: The reviews set V, store S, and the times to of
iterations
Output: reviews' coherence metric
counter 0 repeat
=
for r E V do
compute H(r) using (5);
end
update r's coherence scores using H(r) and Con(r);
end
counter++;
until counter < to.
Fig. 2 The proposed algorithm based on iterative computation
The above algorithm provides a framework to calculate
reviews' coherent score iteratively, which is self-explanatory.
In practice, this algorithm can converge fast. In every round, it
is only needed to access each review once and calculate its
coherent value based on word transition probability and word
concurrence probability. So the time complexity of this
(5) algorithm is O(Nvlo), where to is the number of iterations. It is
linear to the product of reviews' number N" and iteration times
to which is usually small.
V. OUR EVALUATION
To evaluate our method, we built a labeled dataset using
expert human judges. In our evaluation process, human
experts help to label dataset with reviews from ctrip.com
(website of English version) . This website is famous for its
large amount of traveling, hotel services. It is a typical
ecommerce site. Our review data set was crawled from
ctrip.com in June 2009 only for hotel services. We invite 3
students who major in computer science to take part in our
evaluation. The above algorithm is implemented to identifY
100 most suspicious reviews. Then the students begin to judge
these suspicious reviews. They work independently on spam
reviews identification. They are asked to give their
independent judgments based on several rules: (i) reviews are
full of empty adjectives, (ii) reviews with merely praises
without faults, or purely negative comments (iii) reviews are
left within a short period of time of each other, etc [II]. These
101
infonnation is used to identifY spam revIews from truthful
reviews by human evaluators.
Tn our evaluation, we label a review as a suspicious spam
review if more than one student has an agreement. Since the
students can identifY 46 spam reviews from the 100 suspicious
reviews which are identified by our algorithm, the accuracy of
this our methodology is 46%. Although this accuracy is not
very high compared to other similar works, it is still
meaningful to us. Because we are trying to detect spam
reviews on the semantic level which needs to handle more
complex situations. Few existing research is found in this
direction. In addition, the students evaluators agreed on each
other's judgment.
TABLE I. RESULT OF OUR EvALUAnON
Student 1 Student 2 Student 3
Student t 46 29 36
Student 2 38 30
Student 3 40
The evaluatIOn result IS hsted m table I WhiCh shows the
agreement level among evaluators. For example, student 1
identified 46 suspicious reviews; out of which 29 were
recognized by student 2 and 36 were caught by student 3. The
different results also shows that it is hard to get an agreement
on some revIews among all the elevators without enough
strong evidences.
VI. CONCLUSION
Until today, there are a few attempts have been made on
review spam detection which is a challenging and under­
exploration area [2]. In this paper, we propose a general
framework to detect spam reviews based on coherent
examination. We first discuss some assumptions of spam
reviews, and then we define some review coherent metrics.
The iterative computation framework is also provided. Our
proposed model tries to analyze review coherence in the
granularity of sentence. We define some metrics to investigate
the coherence between sentiment words and other related
words based on the flow smoothness between sentences: word
transition probability and word concurrence probability.
Because we try to identifY spam reviews on the semantic level,
our proposed model can reveal more important clues of spam
reviews based on their word pattern. This work provides a
novel viewpoint for spam review detection and more potential
approaches could be explored in the future.
102
REFERENCES
[I] Castillo c., Donato D., "A reference collection for web spam", SIGIR
Forum 2006,40(2),ppll-24.
[2] Wang, Guan, Sihong Xie, Bing Liu, and Philip S. Yu., "Identify Online
Store Review Spammers via Social Review Graph", ACM Transactions
on Intelligent Systems and Technology,2012.
[3] Sun, Huan, Alex Morales, and Xifeng Yan., "Synthetic review
spamming and defense", Proceedings of the 19th ACM SIGKDD
international conference on Knowledge discovery and data mining -
KDD 13,2013.
[4] http://www.cs.uic.edu.
[5] Sharma, Kuldeep, and King-Ip Lin., "Review spam detector with rating
consistency check", Proceedings of the 51st ACM Southeast Conference
on - ACMSE 13,2013.
[6] Xu, Chang, "Detecting collusive spammers in online review
communities", Proceedings of the sixth workshop on Ph D students in
information and knowledge management - PIKM 13,2013.
[7] Lau, Raymond Y. K., S. Y. Liao, Ron Chi-Wai Kwok,Kaiquan Xu,
Yunqing Xia, and Yuefeng Li. "Text mining and probabilistic language
modeling for online review spam detection", ACM Transactions on
Management Information Systems,2011.
[8] 1. Liu, Y. Cao, C.-Y. Lin, Y. Huang, and M. Zhou,"Low-quality product
review detection in opinion summarization",In EMNLP-CoNLL,2007.
[9] Xie, Sihong,Guan Wang,Shuyang Lin, and Philip S. Yu. "Review spam
detection via temporal pattern discovery",Proceedings of the 18th ACM
SIGKDD international conference on Knowledge discovery and data
mining - KDD 12,2012.
[10] G. Wang,S. Xie, B. Liu, and P. S. Yu,"Review graph based online store
review spammer detection",IEEE 11th International Conference on Data
Mining,pp 1242-1247,2011.
[11] A. Mukhe�jee, B. Liu, and N. Glance, "Spotting fake reviewer groups in
consumer reviews", In Proceedings of the 21st International Conference
on World Wide Web (WWW),2012.
[12] N. Jindal and B. Liu., "Opinion spam and analysis", In Proceedings of
the International Conference on Web Search and Web Data Mining
(WSDM),pp2I9-230,2008.
[13] N. Jindal and B. Liu, "Analyzing and detecting review spam", In
Proceedings of the 7th IEEE International Conference on Data Mining,
pp547-552,2007.
[14] NJindal and B.Liu, "Review spam detection", In Proceedings of the
16th International Conference on World Wide Web, pp1l89-1190, 2007.
[15] E. Lim, V. Nguyen, N. Jindal, B. Liu, and H. Lauw, "Detecting product
review spammers using rating behaviors", In Proceedings of the 19th
ACM International Conference on Information and Knowledge
Management (CIKM), pp 939-948,2010.
[16] c. Dellarocas, "Immunizing online reputation reporting systems against
unfair ratings and discriminatory behavior",In ACM EC,2000.
[17] S.-M. Kim, P. Pantel, T. Chklovski, and M. Pennacchiotti,
"Automatically assessing review helpfulness",In EMNLP,2006.
[18] Kleinberg, J. ,"Authoritative sources in a hyperlinked environent". In J.
ACM 46,5,pp604-632,1999.