Proceedings of the 2013 International Conference on

Pattern Recognition, Informatics and Mobile Engineering (PRIME) February 21-22

A novel Method to measure the Reliability of the

Bollywood Movie Rating System

Rahul Gupta Nidhi Garg Arpan Das

Department of Information &
Communication Technology
Manipal Institute of Technology
Manipal University
Karnataka, India- 576104
rahul.gupta.engg@gmail.com
Department of Computer Science,
SOET
Jaipur National University
Jaipur, Rajasthan, India-302017
nidhigargjaipur@gmail.com
Department of Information &
Communication Technology
Manipal Institute of Technology
Manipal University
Karnataka, India- 576104
pseudo.biological@gmail.com

Abstract—The success of a movie is stochastic but it is no secret
that it is dependent to a large extent upon the level of
advertisement and also upon the ratings received by the major
movie critics. The general audience values their time and money
and hence, refers to the leading critics when making a decision
about whether to watch a particular movie or not. Due to this,
several production houses tends to influence the critics to provide
fraudulent ratings in order to increase one’s business or decrease
other movie’s business. In our paper, we have used a
methodology called Kappa Measure to analyse the concordance
of the Bollywood and Hollywood movie ratings among
themselves. Our study proves that there is a statistically
significant disagreement between Indian critics, implying that the
ratings are biased. The Hollywood ratings showed good
agreement and thus, are more reliable. This peculiarity had gone
unnoticed so far and no previous studies exist regarding such
mismatching patterns in the ratings. Such a result implies that
there is a considerable bias among Indian critics and thus, the
Indian audiences are not getting the benefit of an impartial critic
to guide their judgement. The same methodology was used for
Tamil movies (Kollywood) to further investigate the agreement
among critics with respect to a regional movie industry. The state
of affairs is such that even if a viewer relies on a number of
independent critics to form a judgement about a movie’s worth,
she/he is unlikely to form a clear picture of the movie’s actual
worth. Our paper shows that the Indian viewers should not rely
heavily on movie critics and also that the Bollywood movie rating
system is in serious need of an overhaul.
Keywords-Movie Reviews; Kappa Analysis; Bollywood Movies;
Hollywood Movies; Kollywood Movies; Ratings; Viewers; Critics
I. INTRODUCTION
Ever since the advent of motion picture cameras, movies
have captured the public’s imagination like no other medium.
Today, movies remain one of the prime sources of
entertainment with the movie industry providing employment
to millions of people all around the globe. The two leading
producers of movies in the world are Bollywood and
Hollywood, together constituting 70% of the world’s movies
per year [1]. Out of the two, Bollywood leads with more than
1000 movies per year with Hollywood coming a distant second
with 650 movies per year [2]. The Bollywood industry had
revenues of US $3 billion in 2011, and has been growing at
approx. 10.1% a year [3]. The revenue is expected to reach US
$4.5 billion by 2016 [4]. Hence this shows that the growth in
this industry is phenomenal and will not slow down in the near
future.
Revenue generation is achieved through movie ticket sales
as well as through celebrity endorsement and integrated
product placement within the movies [5]. However, the
common factor in all of these is the number of viewers.
Capturing the audiences’ interest is so critical, that there have
been times when the budget for advertisement has been much
higher than the actual expenditure for producing the entire
movie. In today’s Information Age, the power of
advertisements is high, but so is the ability of viewers to block
and ignore advertisements [6]. Advertisements tend to be
unsolicited and self-laudatory and hence, most viewers rarely
get influenced by them. However, a critic is someone who is
widely regarded as being a voice of authority in matters
regarding movie quality. In such a market, professional movie
critics command a large amount of power and their reviews can
either make or break a movie. The audience too, having a large
amount of choices, tends to rely heavily upon movie critics.
Ideally, the ratings should be based only on pertinent
parameters like script, acting, genre, cinematography,
choreography, creativity, videography, screenplay, production,
location, music, etc. So that it avoids personal biases and
accurately summarizes the movie to the viewers. The reviews
and ratings are published just after the movie is released and
viewers often wait for the critic’s opinion before deciding on
which movie to watch [7]. So once the movie is released, these
critics are the sole source of information regarding its aptness
and popularity. The reliability of these critics’ ratings is being
measured in this paper.
The rest of the paper is organized as follows. Section II
gives brief description about the need for movie ratings.
Section III describes the use of kappa measure for quantifying
the agreement between raters. Section IV exemplifies the
methodology used. In section V, the result is explained and

978-1-4673-5845-3/13/$31.00©2013 IEEE
 2013 International Conference on Pattern Recognition, Informatics and Mobile Engineering (PRIME) 341

various observations are made. Section VI concludes this ratings should be entirely based upon these parameters rather
paper. than on personal preferences.
II. NEED FOR RATINGS For the evaluation of the rating system, Kappa measure has
been used to find the agreement in the ratings given by the
Professional movie critics are supposed to provide an different critics in Bollywood and Hollywood. The results
unbiased overview of a movie for the benefit of the audience, obtained were further compared to a regional Movie Industry
so that they may make a decision about watching that movie. Kollywood. Moreover, this evaluation further suggests the
However, even a casual survey will reveal that the same movie need for improvement required by the Bollywood critics and
often receives widely varying reviews (ranging from highly their evaluation system.
laudatory to sharply condemning) from different critics. The
practical significance of such diverse ratings is that viewers are IV. MECHANISM INVOLVED
unable to get a complete picture and viewership is often
determined by which critic the viewer happens to subscribe to. We have analysed the top 10 movies of 2012 ranked on the
Thus, there is a high likelihood that a movie performs better or basis of net gross earned by each movie [11],[12],[13]. Such a
worse than its actual merit. There are instances where movies choice also had the added benefit of including a diverse
which receive poor ratings perform exceedingly well at film collection of movies. These top movies were produced by
different directors and producers with different actors and crew
festivals and win numerous awards.
members and were shot at different locations.
The contrary case also exists, of movies which receive rave
reviews, but which hardly win any awards [8]. To provide A. Equations for calculation of Kappa Measure
some examples, from Hollywood, “Hollow Man” received 2 If N is the number of subjects that has to be rated, n is the
star ratings from major Hollywood critics, but went on to win 7 number of raters, rating in the number of categories k. The
awards and had 10 nominations including an Academy Awards number of subjects is indexed as i=1...N and number of
Nomination. ”New Rose Hotel” received average ratings of 2 categories as j=1...k
stars and a 1 star rating from The New York Times, but went pj and Pi are calculated which are then used in calculating P
on to receive 2 nominations and also won 2 awards at the and Pe
Venice Film Festival. Among Bollywood movies,”Guzaarish”
received negative reviews from the Hindu and the Rediff critic Calculating pj which is the proportion of all assignments
gave it a 2 star rating. The movie went on to receive which were of jth category:
nominations and awards in almost all award ceremonies of
India. It received a total of 39 nominations and 17 awards [9]. 1 n
pj = ¦ n ij (1)
This paper is based on the concept of applying statistical Nn i = 1
methods towards determining the validity of the current rating Calculating Pi which is the extent to which raters agree for
system. The Fleiss Kappa was chosen as the metric because the the ith subject:
numbers of raters for each movie are many [10]. The aim of the k
1
analysis was to determine the measure of concordance among Pi = [( ¦ n 2 )  ( n )] (2)
the major Bollywood film critics and among the major n( n  1) j= 1 ij
Hollywood film critics. The guiding principle is that the
professional critics should have a high degree of agreement 1 N
among themselves as they are expected to have properly
P= ¦ Pi (3)
N i =1
developed tastes and the ability to judge a movie on relevant
criteria. By introducing a quantitative method to a field which 1 N k
is usually subjective in nature, we intend to introduce a P= ( ¦ ¦ n2  Nn ) (4)
N( n )( n  1) i = 1i = j ij
measure of objectivity.
k
III. ROLE OF KAPPA MEASURE Pe = ¦ P 2 (5)
j
Kappa measure is a statistical tool that measures the j= 1
reliability and agreement between different reviewers. In any P  Pe
part of the world, when the movie is released, it is evaluated by N= (6)
a number of raters. As these ratings holds a prime importance 1  Pe
to the potential viewers, the whole system of rating fails if
there is no agreement between different raters/critics because in Table I helps in interpreting the Kappa value that is finally
that case it becomes very difficult for a viewer to decide whom computed. The range mentioned in the table gives a guideline
to follow. So for an industry to grow and functional in a proper under which the level of agreement belongs.
competitive spirit, the agreement between different critics
should be high and unbiased towards any particular region,
production houses or celebrities. There might be a chance that a
particular movie can affect the critics differently based on their
personal interest but this should never happen as the parameters
mentioned in section I are common for all movies and the
 2013 International Conference on Pattern Recognition, Informatics and Mobile Engineering (PRIME) 342

Table II is used to calculate the N measure described as

TABLE I. INTERPRETATION OF KAPPA VALUES follows:
N Interpretation Using Equation 1:
d0 Poor agreement/No agreement 10 movies and 8 reviewers: N*n =80
0.01-0.20 Slight agreement
Taking first column:
0.21-0.40 Fair agreement 1 0  2  0  2  0  0  2  0  1
p1 = [ ] = 0.100
0.41-0.60 Moderate agreement 80
0.61-0.80 Substantial agreement Taking second column:
0.81-1.00 Almost perfect agreement
1 4  2  0  2  0  1 2  2  3
p2 = [ ] = 0.212
80
B. Calculation of Kappa Measure for Bollywood Taking third column:
In our first investigation, Bollywood movies are compared 1 2  1 5  2  3  4  2  2  3
against Hollywood ones. Separate critics were considered for p3 = [ ] = 0.312
Bollywood and Hollywood. Each critic is associated with a 80
different media house and officially declares the ratings on the Taking fourth column:
day the movie is released. For Bollywood, some of the major 5222 24 3231
p4 = [ ] = 0.325
media houses which publish ratings are Filmfare, Times of 80
India, The Indian Express, Rediff, NDTV, Reuters, Bollywood Taking fifth column:
Hangama and DNA India given by some of the popular critics 0  0  1 1 0  1 0  0  1 0
of India like Taran Adarsh, Rajeev Masand, Anupama Chopra p5 = [ ] = 0.050
and more. They rate each movies on the day of its release on a 80
scale of 5 stars [11]. The more the number of stars, more will Using Equation 2:
be its value and better is the opinion formed about the movie Taking first row:
amongst the general public. But it is often found that a movie 1
P1 = (1 2  12  12  5 2  0 2  8) = 0.357
performs quite differently than predicted by the reviewers , 8(8  1)
thus raising questions on the reliability of these ratings.
Taking second row:
For our set of 10 movies and 8 reviewers, each measuring 1
at a scale of 5, the Kappa measure is calculated. For a better P2 = (0 2  4 2  2 2  2 2  0 2  8) = 0.285
8(8  1)
result any decimal values are converted to the nearest integer,
by using the ceiling function, to have a generalized scale. Table Calculating the sum of Pi for P :
II holds the tabulated information about Bollywood movies, the
N
ratings received and shows the corresponding Pi & p j ¦Pi = 0.357  0.285  ....  0.214 = 2.459 (7)
values.In our methodology of kappa measure, rows denotes i
movie names, columns contains ratings on the scale of 5 given Calculating P using equation 3 and 7:
in terms of stars, published by 8 different critics.
1
N= No. of movies=10 , n= No. of critics=8 P= (2.459) = 0.2459 (8)
(10)
K= Range of rating points =5 Calculating Pe using equation 5:

TABLE II. KAPPA MEASURE FOR BOLLYWOOD MOVIES Pe = 0.1002  .... 0.3252  0.0502 = 0.260 (9)
Movie Name * ** *** **** ***** Pi Finally calculating N using equation 6, 8 and 9:
Ek tha tiger 1 1 1 5 0 0.357 0.246  0.260
Dabangg 2 0 4 2 2 0 0.285 N= = 0.02 (10)
1  0.260
Rowdy Rathore 22 22 11 22 11 00.107
Agneepath 0 0 5 2 1 0.392
Housefull 2 2 2 2 0 0.142 The value of Kappa lies below 0 which means there is very
Barfi 0 0 3 4 1 0.321 poor or no agreement between the ratings given by the different
movie critics of Bollywood.
Jab Tak Hai Jaan 0 1 4 3 0 0.321
Bol Bachchan 2 2 2 2 0 0.142 C. Calculation of Kappa Measure for Hollywood
Now, the above result is compared with Hollywood to
Talaash 0 2 2 3 1 0.178
evaluate the Hollywood critics. In this the same process is
Son of Sardaar 1 3 3 1 0 0.214 adopted: The ratings are given by media houses like Time
Total 8 17 25 26 4 2.459 Magazine, Saturday Review, The Guardian, The New York
Times, The Wall Street Journal, BBC, Video Watchdog, NBC's
0.100 0.212 0.312 0.325 0.050 - Today Show etc written by some popular critics like Peter
p j Bradshaw, Tom Brook, Anthony Lane etc [12]. In table III, the
top 10 movies of Hollywood in 2012 based on the net gross
 2013 International Conference on Pattern Recognition, Informatics and Mobile Engineering (PRIME) 343

earned is chosen similar to that of Bollywood. The ratings of 1

Hollywood movies are available in a much more consolidated P2 = (0 2  0 2  5 2  3 2  8) = 0.464
8(8  1)
form at numerous web portals. we acquired our data on
Hollywood movie ratings from a web portal called Rotten Calculating the sum of Pi for P :
Tomatoes. In Hollywood, there are various styles for giving
ratings like using an alphabet rating system in that instead of N
giving a rating of 5, an A+ is awarded and the same holds good ¦Pi = 0.428  0.464  ...  0.464 = 4.821 (11)
for the other ratings. But the most common form of rating a i
movie involves giving it stars out of 4. So for our general form, Calculating P using equation 3 and 11:
we have converted the numerous rating systems into the scale 1
of 4 stars, so that a common platform can be adopted. The P= (4.821) = 0.482 (12)
(10)
decimal values are rounded off to the nearest integer by using
the ceiling function as was done for the Bollywood movies. Calculating Pe using equation 5:

Pe = 0.0122  0.4122  0.4502  0.1252 = 0.388 (13)

TABLE III. KAPPA MEASURE FOR HOLLYWOOD MOVIES
Finally calculating N using equation 6, 12 and 13:
Movie Name * ** *** **** Pi
0.482  0.388
Avenger 0 0 4 4 0.428 N= = 0.153 (14)
Dark Knight Rises 0 0 5 3 0.464 1  0.388
Skyfall 0 0 5 3 0.464 The value of Kappa lies close to 0.2 which means there is a
The Hobbit 0 4 4 0 0.428 fair agreement between the ratings given by the different movie
Ice Age: Continental Drift 0 4 4 0 0.428 critics of Hollywood which is much better than the Bollywood
Twilight: Breaking Dawn 2 1 7 0 0 0.750
raters.
The Amazing Spiderman 0 5 3 0 0.464
D. Calculation of Kappa Measure for Kollywood
Madagascar 3 0 5 3 0 0.464
After comparing the two popular movie industries i.e
The Hunger Games 0 3 5 0 0.464 Bollywood & Hollywood, a regional movie industry was
Men in Black 0 5 3 0 0.464 chosen. So Tamil Movies(Kollywood) were chosen because of
its general popularity, good fan following and easily available
Total 1 33 36 10 4.820 data for investigation. Table IV, has a collection of top 10
0.012 0.412 0.450 0.125 - Tamil movies of 2012 based on the net gross earned. The
p j ratings are given by media houses like Behindwoods,
IndiaGlitz, Sify, Rediff, The Times of India etc[13].
Table III is used to calculate the N measure described as
follows: TABLE IV. KAPPA MEASURE FOR TAMIL MOVIES

Using Equation 1: Movie Name * ** *** **** ***** Pi

Billa 2 0 2 3 0 0 0.4
10 movies and 8 reviewers: N*n =80 Thuppakki 0 0 2 3 0 0.4
Saguni 0 2 3 0 0 0.4
Taking first column: Maattrraan 0 0 4 1 0 0.6
Thaandavam 0 1 4 0 0 0.6
0  0  0  0  0  1 0  0  0  0 Oru Kal Oru Kannadi 0 1 4 0 0 0.6
p1 = [ ] = 0.012
80 Neethaane En 0 0 4 1 0 0.6
Taking second column: Ponvasantham
Mugamoodi 0 2 0 3 0 0.4
000 4 47 5 5 3 5 Podaa Podi 3 0 2 0 0 0.4
p2 = [ ] = 0.412
80 Attakathi 0 0 4 1 0 0.6
Taking third column: Total 3 8 30 9 0 5.0
0.06 0.16 0.6 0.18 0.0 -
45544033 53
p3 = [ ] = 0.450 p j
80
Taking fourth column: Table IV is used to calculate the N measure described as
4 3 30000000 follows:
p4 = [ ] = 0.125
80 Using Equation 1:
Using Equation 2:
Taking first row: 10 movies and 5 reviewers: N*n =50
1 Taking first column:
P1 = (0 2  0 2  4 2  4 2  8) = 0.428
8(8  1)
00000000 30
Taking second row: p1 = [ ] = 0.06
50
 2013 International Conference on Pattern Recognition, Informatics and Mobile Engineering (PRIME) 344

Taking second column: between independent raters. Thus, when it comes to Hollywood
2  0  2  0  1 1 0  2  0  0 and Kollywood movies, the viewers can have a much higher
p2 = [ ] = 0.16 degree of confidence on movie ratings and also, the economic
50
success of movies are not strongly affected by any particular
Taking third column: critic.
3 2 3 4 4 4 40 2 4
p3 = [ ] = 0.6 On an absolute scale however, it must still be noted that
50
even Hollywood and Kollywood critics only achieve scores
Taking fourth column:
indicating average agreement among themselves. Thus, this
0  3  0  1 0  0  1 3  0  1 implies that the review system, although healthy, is still quite
p4 = [ ] = 0.18
50 far from being ideal. Again, arguments might be raised about
Taking fifth column: the diversity factors that prevent all critics from having a
0000000000 common viewpoint, but as already pointed out, the role of
p4 = [ ]=0 professional raters and judges is to analyze and judge the
50 subject of their study with respect to well-established criteria.
Using Equation 2: It's true that for the layman, personal opinion and preferences
Taking first row: will always exist, but such biases would be ethically improper
1 for professional reviewers.
P1 = (0 2  2 2  3 2  0 2  0 2  5) = 0.4
5(5  1) As per the Kappa Measure Methodology, a low agreement
Taking second row: can be caused due to a faulty rating system or due to improper
1 training of the raters. Thus, the results of this paper point out
P2 = (0 2  0 2  2 2  3 2  0 2  5) = 0.4 shortcomings in either one or both of these aspects. Since the
5(5  1)
star rating system is followed by all 3 movie industries
Calculating the sum of Pi for P : surveyed, and only Bollywood critics showed a negative Kappa
score, it is unlikely that the rating system itself is to be blamed.
N It appears more probable that there is a need for retraining of
¦Pi = 0.4  0.4  ...  0.4  0.4  0.6 = 5.00 (15)
the reviewers. An important consideration is that Bollywood
i
provides a platform for a wide variety of genres based in
Calculating P using equation 3 and 15: different regional settings. However, as already mentioned, it is
1 the job of professional reviewers to establish proper guidelines
P= (5.00) = 0.50 (16) for the uniform evaluation of all movies, irrespective of any
(10)
changes in style. This leads us back to the point that retraining
Calculating Pe using equation 5: of reviewers is essential in case of Bollywood.
Pe = 0.062  0.162  0.62  0.182  0.02 = 0.42 (17) VI. CONCLUSION
Finally calculating N using equation 6, 16 and 17: The use of Kappa measure for establishing the validity of
0.50  0.42 movie ratings is novel and as per our research, unprecedented.
N= = 0.14 (18) In this paper, we have applied this concept to three major film
1  0.42
industries and have come up with a quantitative insight into the
health of the respective movie industry's rating system. Focus
The value of Kappa is similar to that obtained by the
was restricted to the top ten movies of each film industry to
analysis of Hollywood critics. This means that the performance
ensure that the movies surveyed were universal in terms of
of Hollywood and Kollywood movie critics are quite close to
reach with-respect-to the audience. The practical ramifications
each other and there is a fair agreement between the ratings
of our work is that there can now be an increased awareness
given by the different movie critics of Hollywood and
about the need for more rigorous standards with respect to the
Kollywood. Both Hollywood and Kollywood movie critics
critics themselves. Since society relies upon critics to provide
perform much better than the Bollywood raters with respect to
proper feedback, the ultimate effect will be a more healthy
the Fleiss' Kappa methodology.
system of review and rating, which will help raise standards, as
V. RESULTS well as ensure that viewers have a much more rewarding box
office experience.
The use of the Kappa Measure statistical tool with respect
to the Indian cinema throws up a number of interesting REFERENCES
observations. It is seen that the Bollywood critics compare
exceedingly poorly when compared to their Hollywood and [1] Marc Fetscherin, “The main determinants of bollywood movie box
Kollywood counterparts. The Kappa score of -0.02 indicates a office sales,” Journal of Global Marketing, 23:461–476, November
complete lack of agreement among raters, which has the 2010.
practical significance of confusing viewers as well as wrongly [2] Shruti Vinayak Gokhale, “Comparative study of the practice of product
influencing the economic success of movies. Compared to placement in bollywood and hollywood movies,” [Available Online]
Bollywood, Hollywood and Kollywood fare much better and http://scholarworks.sjsu.edu/etd_theses/3860//, 2010.
their critics attain scores that signify a slight agreement
 2013 International Conference on Pattern Recognition, Informatics and Mobile Engineering (PRIME) 345
[3] IMDB, “Highest award winning movies,” [Available Online]
http://www.imdb.com/title/tt0164052/awards, 2000.
[4] Movie Insider, “Movie releases 2012,” [Available Online]
http://www.movieinsider.com/movies/-/2012/, 2012.
[5] J. Richard Landis and Gary G. Koch, “The measurement of observer
agreement for categorical data,” International Biometric Society,
33:159–174, March 1997.
[6] Narayan Devanathan Michelle R Nelson, “Brand placements bollywood
style. Journal of Consumer Behaviour,” 5:211–221, May 2006.
[7] Available Online. Isbnlive movie ratings.
http://ibnlive.in.com/movies/movieratings/, 2012.
[8] Shakuntala Rao, “The globalization of bollywood: An ethnography of
non-elite audiences in india,” The Communication Review, 10:57–76,
February 2007.
[9] J. P Singh and Kate House, “Bollywood in hollywood: Value chains,
cultural voices, and the capacity to aspire,” July 2010.
[10] Andrew B, “Whinston Wenjing Duan, Bin Gu. The dynamics of online
word-of-mouth and product sales - an empirical investigation of the
movie industry,” Journal of Retailing, 84:233–242, June 2008.
[11] Wikipedia. List of bollywood films of 2012. [Available Online]
http://en.wikipedia.org/wiki/List_of_Bollywood_films_of_2012/, 2012.
[12] Wikipedia. List of highest-grossing films. [Available
Online]http://en.wikipedia.org/wiki/List_of_highestgrossing_filmsHighe
stgrossing_films, 2012.
[13] Wikipedia. List of tamil films of 2012s. [Available
Online]http://en.wikipedia.org/wiki/List_of_Tami_films_of_2012, 2012.