Journal of Computational Science 28 (2018) 168–179

Contents lists available at ScienceDirect

Journal of Computational Science

journal homepage: www.elsevier.com/locate/jocs

Travelers decision making using online review in social network sites:

A case on TripAdvisor
Mehrbakhsh Nilashi a,∗ , Othman Ibrahim a , Elaheh Yadegaridehkordi b , Sarminah Samad c ,
Elnaz Akbari d,e,∗ , Azar Alizadeh f
a
School of Computing, Faculty of Engineering, Universiti Teknologi Malaysia, 81310 UTM Johor Bahru, Johor, Malaysia
b
Department of Information Systems, Faculty of Computer Science & Information Technology, University of Malaya, 50603 Kuala Lumpur, Malaysia
c
CBA Research Centre, Department of Business Administration, Collage of Business and Administration, Princess Nourah Bint Abdulrahman University,
Riyadh, Saudi Arabia
d
Department for Management of Science and Technology Development, Ton Duc Thang University, Ho Chi Minh City, Viet Nam
e
Faculty of Electrical & Electronics Engineering, Ton Duc Thang University, Ho Chi Minh City, Viet Nam
f
School of Engineering, University of California, Merced, USA

a r t i c l e i n f o a b s t r a c t

Article history: Digital technology and social media have brought numerous benefits to human society. TripAdvisor,
Received 3 April 2018 which runs on user-generated content, provides a platform for travelers to socialize their opinions on
Received in revised form 22 July 2018 several aspects of hotels. Recommendation agents have played an important role for hotel recommenda-
Accepted 15 September 2018
tions in the tourism domain. They are valuable tools in e-tourism platforms of travel agencies to help the
Available online 16 September 2018
users in their decision-making process. The recommendation of hotels by multi-criteria Collaborative
Filtering (CF) recommender systems is mainly based on their past reviews on several aspects. Hence,
Keywords:
recommending the most appropriate hotel to the user is one of the important tasks that a multi-criteria
Recommender systems
Digital technology and social media
CF needs to do in the e-tourism platform. The aim of this research is to use the multi-criteria ratings in
Online review developing a new recommendation method for hotel recommendations in e-tourism platforms. We use
Collaborative filtering supervised and unsupervised machine learning techniques to analysis the customers’ online reviews. The
TripAdvisor method is evaluated on the data provided by the travelers via TripAdvisor mobile application. The results
of our analysis on the dataset confirm that the use of online reviews in the proposed recommendation
agent leads to precise recommendations in TripAdvisor.
© 2018 Elsevier B.V. All rights reserved.

1. Introduction of such websites that provide a platform for travelers to socialize

Digital technology and social media have played an important
role in developing travel industry businesses [1]. In fact, they have
provided a new distribution channel for businesses to communi-
cate with their customers. The hotels managers and marketers can
effectively interact with their customers through the social net-
working websites [2] such as Facebook, Twitter, and TripAdvisor,
by sharing the information of products and brands. According to
Travel Industry Association of America, the evidence shows that
64% of travelers use search engines for their travel planning [3].
Nowadays, many travelers often use the hotels recommended
by the social media in their trip. Social media can help the travel-
ers in all stages of their decision-making process [4]. TripAdvisor,
as the world’s largest travel site and public social media, is one
∗ Corresponding authors.
E-mail addresses: nilashidotnet@hotmail.com (M. Nilashi),
elnaz.akbari@tdtu.edu.vn (E. Akbari).
their opinions on several aspects of hotels. By the use of this plat-
form, the travelers can share their travel experiences and provide
the feedback on the service quality of hotels [5,6]. The evidence
shows that 60% of the users in US used TripAdvisor to select the
accommodations for their trip plan [2]. As of April 2014, the Tri-
pAdvisor mobile apps have been downloaded 100 million times [2]
which shows that its usefulness in travel industry [2]. By the use
of these apps, the users can share their reviews, photos, and maps
from TripAdvisor.
In online environments, recommender systems are used as
decision support systems to overcome information overload [7,8].
These systems have been used in online platforms to actively rec-
ommend the relevant information to the customers and help them
in the decision making process. Typically these systems do not
need an explicit search query for recommending the items to the
users. They are applications that help users finding useful resources
by automatically scanning the space of choices and selecting the
most interesting ones [7,9,10]. One of the most useful types of rec-

https://doi.org/10.1016/j.jocs.2018.09.006
1877-7503/© 2018 Elsevier B.V. All rights reserved.
 M. Nilashi et al. / Journal of Computational Science 28 (2018) 168–179
ommender systems are Collaborative Filtering (CF) recommender
systems [11–13]. They are mainly used as information retrieval
systems based on past users activities and ratings on the items.
The assumption is that they operate and recommend the items
to an active/target user based on the other users’ preferences on
the items. These systems are widely used in e-commerce websites
([14,15]). As an example, Netflix uses CF as part of its personal-
isation approach, and although details are not open to use, it is
widely believed that Amazon.com also uses CF for its personalised
recommendations. CF algorithms mainly aggregate feedback for
items from different users and use the similarities between items
and items (item-based) or between users and users (user-based) to
provide recommendations to a target user.
Recommender systems with their ability to suggest a person-
alised set of items to the users can enhance performances of online
travel agencies, with increasing results in overall satisfaction of
the whole ecommerce process and, as a consequence, in the eco-
nomic performance [16]. As an example, TripAdvisor which has
been one of the important elements in travel marketing is used by
travel agencies to grow and market the accommodation products
and businesses of hotels. This platform uses recommender systems
technologies to support the user in his/her trip destination tasks.
The users’ ratings are based on the several aspects, such as Service,
Value and Cleanliness, on hotels. Also, users in TripAdvisor can pro-
vide their reviews on hotels. It has a complete rating system for the
users to provide their opinions on the hotels.
Previous research shows that multi-criteria ratings can sig-
nificantly enhance the accuracy of CF-based recommendation
algorithms in tourism domain [17]. Multi-criteria CF recommender
systems, can provide a better platform for the users to provide their
opinion about the items on several aspects. The advantage of these
systems is that the users do not need to rely solely on one aspect of
an item which is provided by traditional CF recommender systems.
Taking the advantages of multi-criteria ratings will accordingly lead
to accuracy improvements of recommenders systems. These types
of recommender systems have been more successful in [18]. The
aim of this paper is to propose a new method based on multi-
criteria CF for hotel recommendation in TripAdvisor. We rely on
the ensemble learning of supervised (fuzzy-rule based) and unsu-
pervised (clustering) machine learning techniques for developing
the method. It is believed that these techniques can improve the
prediction accuracy and scalability of traditional recommendation
systems based on CF.
This paper is structured as follows. We present the related works
in Section 2. In Section 3, the methodology is presented. In Section
4, the results of method evaluation are provided. Finally, in Sections
5 and 6 we respectively present the discussion and conclusion of
our work.
2. Related works
Several studies have been conducted for the development of
recommender systems in the tourism context. Loh et al. [19]
proposed a recommender system for travel agents using text min-
ing techniques. Bigdeli and Bahmani [20] conducted a study for
tourism recommender systems and used the similarity approaches
of cosine-based and correlation-based for items prediction task.
They evaluated the methods using the data collected from a big
travel agency in Tehran. Esparcia et al. [21] presented an add-on
for social-net tourism recommender. They developed the recom-
mender system based on multi-agent system technology. Kenteris
et al. [22] developed a mobile tourism recommender system using
CF techniques. The system was able to captures context-aware
user evaluations and ratings for recommendations task. Blanco-
Fernandez et al. [23] proposed an adaptive recommender system,
169
TripFromTV, which was able to access through domestic and mobile
consumer devices. They evaluated the system experimentally with
a set of 117 users. Al-hassan et al. [24] developed an ontology-
based recommender system for tourism services recommendation.
They developed framework for the recommender system was based
on User Data Collector (UDC), Knowledge Base Repository (KBR),
Intelligent Recommendation Engine (IRE) and Customized User
Interface (CUI). Chu et al. [25] developed a tourism recommender
system, TRSO, based on a hybrid approach using association rules
and ontology. Namahoot et al. [26] developed a tourism recom-
mender system using KNN and semantic web for Thailand. They
implemented the system using Java Server Page (JSP). Zheng et al.
[27] developed a tourism destination recommender system to solve
cold-start problem of user- and item-based CF. They evaluated
the method on Tongcheng dataset with different metrics. Zhang
et al. [28] a temporal recommender system for online tourism
websites based on the temporal browsing historie. Their method
outperformed the CF, popularity-based and content-based meth-
ods. Kashevnik et al. [29] developed a multi-model approach for
context-aware recommender systems in the tourism context.
Several studies have also been conducted in case of multi-
criteria recommender systems in tourism domain. Li et al. [30]
applied a tensor decompostion technique in developing a rec-
ommendation system system based on multi-criteria CF. Three
multi-criteria recommendation methods were developed by Liu
et al. [31]. They used the clustering techniques to cluster the data
befor prediction and recommendation tasks. Jannach et al. [17]
developed a recommender system based on item- and user-based
CF by the use of a regression technique. They evaluated the method
on the datasets in movie and tourism domains. Nilashi et al. [13]
used clustering and support vector regression (SVR) techniques in
developing a recommender system. They used YahooMovie data
for the method evaluation. They found that the use of multi-criteria
ratings can significantly improve the CF based techniques for movie
recommendations.
3. The problem statement and the research contributions
The use of machine learning techniques is crucial in developing
the recommender systems. In case of multi-criteria collaborative
filtering recommender systems, the use of supervised machine
learning techniques has been prominent and their contribution has
been promising. As in multi-criteria recommender systems the pre-
diction of overall ratings provided by users on items is important,
selecting the appropriate and robust supervised machine learn-
ing technique can play important role in accuracy improvement
of these systems. In previous studies, the methods developed for
multi-criteria recommender systems for tourism applications have
relied on single inductive learning methods.
From the literature review on multi-criteria CF recommender
systems for tourism applications, we found that the most methods
in developing recommender systems use single inductive learning
methods and there is no implementation of ensembles machine
learning techniques in this context. In addition, there is no study
by clustering ensemble techniques and prediction ensemble learn-
ing for multi-criteria CF recommendation. The previous research
only apply the cluster ensemble techniques in CF context [32],
however, a few studies take the advantages of cluster ensembles
techniques for recommendation systems based on multi-criteria
CF. Furthermore, there is no study to investigate the advantages of
fuzzy rule-based technique and its ensemble learning for multi-
criteria CF recommender systems. Accordingly, in this work we
try to develop a recommender system for hotel recommendation
based on multi-criteria CF using ensembles of machine learning
techniques. We use fuzzy rule-based for prediction task and SOM
170 M. Nilashi et al. / Journal of Computational Science 28 (2018) 168–179

(Self-Organizing Map) and EM (Expectation Maximization) for data niques. The online reviews on hotels are clustered and then used
clustering. We also use CART (Classification and Regression Trees) for prediction of the customers’ choices for hotels.
for decision rule discovery from the users’ ratings. Overall, our con-
tributions in this work in developing the recommender system are
4. Methodology of research
as follows:
The aim of this work is to propose a new recommendation
method using machine learning techniques. Our method includes
• We use multi-criteria ratings in developing the recommender
clustering (SOM and EM), CART, fuzzy rule-based machine learning
system. The ratings are obtained from the TripAdvisor website
techniques. These techniques are respectively used for dimension-
which have been provided by the TripAdvisor’ mobile apps.
ality reduction, decision rules generation and prediction tasks. The
• We use clustering (SOM and EM), CART and fuzzy rule-based
main objective of using clustering technique is to consider the most
machine learning techniques for hotel recommendations in the
similar observations/items for recommendation task. We believe
e-tourism platforms.
that the use of clustering will make the recommendation system
• We use the fuzzy rule-based technique to predict the customers’
more scalable. Using such a clustering approach also helps us to
choices for hotels. Fuzzy rule-based systems are based on fuzzy
improve the accuracy of recommendations which depends on sev-
logic approach. In contrast with the crisp machine learning tech-
eral factors such as level of data sparsity and noise of the data. We
niques, fuzzy logic can handle problems with imprecise and
select two robust clustering techniques, SOM and EM, for this aim.
incomplete data. The models based on fuzzy rule-based tech-
These techniques will generate clusters of users’ ratings data in sev-
nique are more intuitive and easier to understand than other
eral distinct groups. We use CART for discovering the rules from the
crisp-based prediction machine learning techniques (e.g., neural
data in TripAdvisor which are in a tensor with three dimensions,
network). Fuzzy rule-based technique introduces the concept of
Users, Hotels and Criteria. The users provide the ratings on hotels
degree in the verification of a condition which makes it a flexible
based on several criteria and, therefore, the main objective is to pre-
technique in handling the uncertainties. In addition, one of the
dict the overall ratings from the ratings in the criteria dimension. In
advantages of fuzzy rule-based technique is to formalize human
multi-criteria CF, the data are kept in a 3 dimensional tensor which
reasoning which the reasoning rules are set in natural language. In
includes Users (U1 -Um ), Hotels (H1 -Hn ) and Criteria (C1 -Ck ) dimen-
this research, we develop the fuzzy-rule based models for ensem-
sions. Two main tasks of a multi-criteria CF algorithm are prediction
ble learning. It has been shown that solely using a single inductive
and recommendation. The algorithm will predict an active item and
learning method cannot achieve a generalization performance in
recommend it to an active user. In this process, the recommender
all possible prediction tasks [47]. In additions, the accuracy of
system will produce a top-N list of items to be recommended to an
using supervised learning methods more depends on the nature
active user.
of the problem investigated and observed data. Despite this con-
Fig. 1 shows the proposed research model based on multi-
sideration, ensemble learning has proved to have an excellent
criteria CF by the use of clustering (EM and SOM) and prediction
performance in numerous learning tasks of practical applications
machine learning techniques (CART and Fuzzy Rule-Based). The
(Hansen and Salamon, 1990). In ensemble learning paradigm, a
key aspect of our method is the use of ensemble learning for pre-
collection of different predictors is constructed whose individ-
dicting the overall ratings. We propose to rely on fuzzy rule-based
ual responses are then combined to label test instances [42].
method to learn the prediction models. In the first step, the data
Accordingly, combining the outputs of several predictors results
is pre-processed (1). In the second step, SOM and EM clustering
in improved accuracy in many prediction problems. Specifically,
with an ensemble learning using HGPA is performed. Next, CART is
the generalization performance of the ensemble learning has
applied to automatically generate the fuzzy rules (3). Finally, pre-
been proved to be much better than a single individual ensem-
diction models are constructed by fuzzy rule-based method with
ble member. In fact, these improvements in performance and
an ensemble learning in each cluster (4).
accuracy are because of the combination of accurate predictors
The prediction models are separately learnt for items and users.
which their errors are complementary. In this research, multi-
Then we combined them by a weighted approach. To do so, we fol-
ple components of Mamdani fuzzy-rule based learner are trained
low the previous studies [17,33,34], to estimate the parameters for
for overall ratings prediction task. Accordingly, integration by
each item and user in each cluster. In addition, a heuristic gradi-
average approach was applied for final ensemble decision [43].
ent descent procedure is applied for this task as shown in Eq. (1).
As a straightforward approach, the final decision by integration
Here, wu and wi are the weights, r̂ user and r̂ item are respectively
by average approach is obtained from the sum of results gener-
the user-based and item-based regression functions, and r̂ is the
ated by each fuzzy-rule based model divided by the number of
output.
fuzzy-rule based models used for prediction task.
• We propose to rely on CART for decision rule discovery from the user
r̂u,i = wu ∗ r̂u,i item
+ wi ∗ r̂u,i (1)
ratings on hotels. The rules generated by this technique can be
used in the users’ decision process. CART is a supervised machine Sparsity is one of the main drawbacks of CF recommender sys-
learning technique which the rules generated by this techniques tems, which is also known as the new community problem. If the
are simple and understandable in linguistic forms. The reason for number of ratings is low, then ineffective recommendations will be
using this technique is that one of the important parts of fuzzy generated by a CF algorithm. In order to provide effective and rele-
rule-based technique is database of rules. The fuzzy rule-based vant recommendations, collaborative recommendation algorithms
technique needs the decision rules of this part for prediction task. need a sufficient amount of ratings data, dependent on the specific
However, this technique cannot generate the rules in an auto- algorithm implementation. In this study, we take the advantages
matic manner and needs human intervention. Accordingly, we of clustering and fuzzy rule-based techniques to solve this issue.
have applied CART to automatically generate decision rules to be Further, to solve the sparsity issue in the dimensions, we use Pear-
used in fuzzy rule-based technique. son Correlation Coefficient (PCC) as a nearest neighbor approach
• SOM and EM with ensemble learning are used for data clustering. as presented in Eq. (2). We also present the online phase in Fig. 1.
Clustering has been effective for precise recommendation in the In this phase, a target user is selected and accordingly the target
previous studies [31]. To further improve the accuracy of single items are predicted in a list as recommendations. In this step, the
clustering, in this research we used clustering ensemble tech- rating prediction and recommendation tasks are performed. The
 M. Nilashi et al. / Journal of Computational Science 28 (2018) 168–179 171

Fig. 1. Proposed recommendation method.

 
rating prediction task involves using each algorithm to predict the neighbourhood radius and the winner neuron and rn∗
 − rj  indi-
rating of a list of unrated items based on the known preferences cates the distance between the jth neuron and winner neuron in the
of a given active user. In the recommendation task, the ranking a map space.
list of unrated items for the active user is done and then the top-K
recommendations (first K items of the ranked list) are presented.
⎧  2  2
⎪ 0 if ru,i − r̄u = 0 or if rv,i − r̄v = 0 or if I = 0
⎪
⎪
⎨

i ∈ Iu v
   i ∈ Iuv
simu,v ru,i − r̄u rv,i − r̄v (2)
⎪  i ∈ Iu v
⎪
⎪
⎩
 2 
 2 otherwise
i ∈ Iuv
ru,i − r̄u i ∈ Iu v
rv,i − r̄v

In Eq. (2), Iuv indicates a set of items which has been rated by both
user u and v rate, Iu and Iv indicate a set of items that has been
respectively rated by users u and v.

4.1. Clustering 4.1.2. EM clusteing

4.1.1. SOM clustering
Clustering, one of the popular data mining techniques, tries to
discover patterns of a dataset by finding fixed set of clusters or cat-
egories in order to uncover meaningful groups [44]. According to
Jain, 2010 [45] and Ghazanfar and Prügel-Bennett, 2014 [46] “Hier-
archical clustering” and “Partitional clustering” are two basic types
of clustering algorithm. Hierarchical clustering produces a nested
set of clusters that are ordered as a tree, while, partitional cluster-
ing generates all clusters once as a partition. Among the clustering
methods, Self-Organizing Map (SOM) and k-means are the widely
used in real business applications. SOM is a method based on an
artificial neurons clustering technique. In this study, we used SOM
for clustering task. We used Matlab software and developed SOMs
of SOM PAK toolbox for clustering task [35]. The models of predic-
tion were trained and accordingly clustering of the data has been
performed. In addition, we used the following equation to update
all neurons j for each epoch e and training example p.
EM algorithm, proposed by Dempster et al. [36], has been widely
used in the prior studies as a unsupervised learning method. In
Algorithm 1, the EM algorithm is presented. From the EM algo-
rithm, it can be seen that EM is mainly divided into two main steps
which are E-step and M-step. In the E-step of EM algorithm, the
belongness of each data point to a component is estimated. In M-
step, on the basis of the estimation in E-step of EM, re-estimating of
the parameters is performed. In the last step of EM, we should check
for convergence. In addition, E-step and M-step need to be repeated
until convergence is met which in EM this can be performed based
on convergence of the parameters or on the log likelihood function.
It should be noted that in our study the convergence was reached
after 10 iterations.
4.2. CART
  Breiman et al. [37] introduced Classification and Regression
 ←− TR
TR j
 +g
j neigh (,
rn∗
 − rj )[pi − TR
 ]
j (3) Trees (CART) as a data analysis technique. This technique is mainly
used to model a response variable (y) based on some input vari-
In the above equation, pi is the ith training example, TR  and
  j ables (x). The hypothesis space of decision/regression trees is the
gneigh (, rn∗
 − rj ) are respectively indicate the map type for the set of piecewise constant functions h (.) defined on the input space
jth neuron and neighbourhood function, s and n* are respectively by:
172 M. Nilashi et al. / Journal of Computational Science 28 (2018) 168–179

⎧
⎪ h1 if a ∈ R1 Co-association based functions, hypergraph partitioning and
⎪
⎪
⎨ h2 if a ∈ R2
majority voting are the most used cluster ensemble approaches
h(a) , (4) (consensus functions). In this study, we use hypergraph partition-
⎪
⎪ ··· ing. In hypergraph partitioning, hyper-edges on a graph are used
⎪
⎩
hL if a ∈ RL to represent the clusters [38]. Cluster-based Similarity Partitioning
Algorithm (CSPA), HyperGraph Partitioning Algorithm (HGPA) and
Meta-Clustering Algorithm (MCLA) are the most used functions for
transforming a set of clusterings into the hypergraph representa-
where Ri , . . ., RL is a recursive partition of the input space into T
tion. In this study, we use HGPA [38] which use the minimal cut
regions and h is the constant prediction associated to each region
algorithms in finding good hypergraph partitions.
Ri (a class or a number).
This technique is an appropriate choice for finding the relation-
4.3. Ensemble of fuzzy rule-based method
ships between the factors of a model by generating decision rules
which can be easily understood by the human. This technique sim-
Fuzzy set theory is introduced for the firt time by [39]. Fuzzy
ply translates data into a set of ÏF . . . THEN . . .r̈ules in order to
logic method is based on fuzzy set thories which has been widely
effectively predict the output of a model. In CART, recursive parti-
used in decision support systems. Fuzzy logic method mainly uses
tion creates the decision trees. In order to represent the structure
linguistic variables to approximate human reasoning. A Member-
of the recursive partition and how it is defined in terms of the input
ship Function (MF) of a fuzzy set is a continuous curve which is
attributes, a decision tree structure is used. Each interior node of
used in the fuzzification and defuzzification steps of a fuzzy logic
such a tree is labeled with a test defined as a function of the attribute
system (see Fig. 2). They are simple curves but accurate in mod-
values and each terminal node defines a region R. To find the region
elling the decision systems. They accept ranges between 0 and 1. In
associated to a point a, it is sufficient to propagate it into the tree
this research, two types of membership functions which are Trian-
according to test issues until it reaches a terminal node. Each region
gular and Gaussian are used for developing the prediction models.
Ri is thus described by a sequence of tests and the whole tree simply
They are shown in Eqs. (5) and (6), respectively.
translates into a set of ÏF . . . THEN . . .r̈ules. In this research, inducing ⎧
regression trees is performed on all clusters of datasets. The con- ⎪ 0 x≤a
⎪
⎪
struction is applied in the two distinct steps. In the first step, while ⎪
⎪ x−a
in the growing phase, when establishing top down method, the tree ⎨ b−a a≤ x ≤b
is constructed from splitting the nodes in a recursive manner. In the f (x, a, b, c) = c−x (5)
⎪
⎪ b ≤ x≤c
second step, while in the pruning phase, when establishing the bot- ⎪
⎪ c −b
⎪
⎩
tom up method, the tree by eliminating the irrelevant branches is
0 c≤x
pruned i.e. by removing the subsequent nodes, a node to a leaf is
(x−c)
transformed. This is done during of the second step where it tried −
f (x, c, ) = e 2 2 (6)
to choose the best performing tree. It should be noted that, with the
growing set and pruning set, the aim of regression tree is to search
and find the optimal tree on the pruning set.
 M. Nilashi et al. / Journal of Computational Science 28 (2018) 168–179 173

Fig. 2. Membership functions for the s = design of recommender system.

In Eq. (5), the x coordinates of the three corners of the underlying
triangular MF are determined by the parameters {a, b, c} (with a < b
< c). In Eq. (6), a Gaussian MF is determined complete by c and ; c
represents the MFs centre and  determines the MFs width. These
MFs have been used in designing many decision support systems
as they are can be simply implemented and are more accurate. In
Fig. 2, some of the MFs are visualized for designing the proposed
knowledge-based recommender system.
In defuzzification steps several techniques can be used Largest
of Maximum (LOM), Mean of Maximum (MOM), Bisector of Area
(BOA), Centroid of Area (COA) and Smallest of Maximum (SOM)).
They provide a crisp value form membership of output linguistic
variables.
In ensemble learning paradigm, a collection of different predic-
tors is constructed whose individual responses are then combined
to label test instances [42]. Accordingly, combining the outputs of
several predictors results in improved accuracy in many prediction
and classification problems. Specifically, the generalization perfor-
mance of the ensemble learning has been proved to be much better
than a single individual ensemble member. In fact, the combina-
tion of accurate predictors leads to improvements in performance
and accuracy because the process tends to cancel their individual
Fig. 3. Ensembles of fuzzy rule-based technique combined with CART.
errors. The final ensemble decision can be reached by combin-
ing the individual predictions of the ensemble members. To do
so, different approaches, for example, average prediction, major- 5. Results and discussion
ity voting and non-linear decision functions can be used. It should
be noted that in general there is no strong evidence supporting 5.1. Dataset
that more complex combination approaches can perform better
performance. This study uses TripAdvisor data [40] for the evaluation of
In this research, multiple components of fuzzy rule-based the prediction ensemble models constructed by fuzzy rule-based
method are used for predicting users’ overall assessment value of technique and for the recommendation techniques validation. In
hotels. Accordingly, integration by average approach was applied TripAdvisor platform, the users can rate the hotels on seven dif-
for final ensemble decision [43]. As a straightforward approach, ferent aspects which are: {Cleanliness}, {Check in/front desk},
the final decision by integration by average approach is obtained {Business Service}, {Location}, {Rooms}, {Value}, and {Service}.
from the sum of results generated by each fuzzy rule-based model The ratings range is from 0 to 5 stars. The users also can give
divided by the number of fuzzy rule-based models used for predic- their overall rating based on the seven criteria rating on the hotels.
tion task. The proposed ensemble structure is depicted in Fig. 3. As Accordingly, we have a tensor (A ) of data with the ratings
m×n×k
can be seen from this figure, we use CART for fuzzy rule discovery (k) (Overall Ratings and Criteria Ratings), users (m) and items (n).
to be used in the fuzzy rule-based technique. In Tables 1 and 2, we present the information of TripAdvisor data
used for the method evaluation.
174 M. Nilashi et al. / Journal of Computational Science 28 (2018) 168–179

Table 1
The users’ ratings on hotels in TripAdvisor dataset.

User ID Overall ratings Value Criteria Rooms Criteria Location Cleanliness Check in/front Service Criteria Business Hotel ID
Criteria Criteria desk Criteria Service Criteria

11 5 −1 5 5 5 5 5 5 hotel 572859
14 5 5 5 5 5 5 5 5 hotel 566077
18 5 5 4 −1 5 −1 5 −1 hotel 565550
20 3 3 4 2 5 1 1 2 hotel 566077
22 4 3 3 3 4 3 4 3 hotel 570888
45 5 5 5 5 5 3 3 5 hotel 570888

Table 2
Number of ratings in TripAdvisor data for the methods evaluation.

Overall ratings Value Criteria Rooms Criteria Location Criteria Cleanliness Criteria Check in/front desk Criteria Service Criteria Business service Criteria

N 15 254 15 254 15 254 15 254 15 254 15254 15 254 15 254

Mean 2.456 2.675 2.498 1.127 2.786 1.5486 3.643 1.632

The dataset includes 15 254 records of users’ ratings on seven Table 3

MSE and R2 for SOM and EM clusters.
criteria of hotels and an overall rating, 5 indicating the best pref-
erence and -1 indicating null value. This dataset is very sparse as Size R2 MSE
many items are not rated by the users. As few rating are available SOM
for the items, accordingly the algorithms must be robust in predict- 4 0.7228 0.0978
ing the criteria and overall ratings (solving the sparsity issue) for 6 0.9224 0.0432
precise recommendations. 9 0.8423 0.0799
12 0.8574 0.0749
16 0.8842 0.0675
5.2. Method evaluation
EM
5 0.9445 0.0389
In this research, EM and SOM were used to cluster the TripAd- 6 0.9558 0.0334
visor data. For SOM, SOM 2 × 2 (4 clusters), SOM 2 × 3 (6 clusters), 8 0.9141 0.0507
SOM 3 × 3 (9 clusters), SOM 3 × 4 (12 clusters), SOM 4 × 4 (16 clus- 10 0.9031 0.0585
12 0.8138 0.0838
ters) were considered. In addition, EM was applied on the data
to generate five clustering sizes which were 5, 6, 8, 10 and 12.
After clustering the data using EM and SOM, we used CART to find Table 4
the relationships of input variables (Value aspect, Rooms aspect, R2 for the methods.
Location aspect, Cleanliness aspect, Check in/front desk aspect, Method Ensemble Size R2
Service aspect, Business service aspect) and output variable (Over-
Fuzzy rule-based 5(2 × 2 + 2 × 3 +3 × 3+ 3 × 4+ 4 × 4) 0.932
all Rating). The relationships are defined by decision rules. For Ensemble + SOM Ensemble
example, the first decision rule is presented as follows: Rule 1: IF 4(2 × 2 + 3 × 3+ 3 × 4+ 4 × 4) 0.915
“Rooms aspect” in [Moderate] AND “Check in/front desk aspect” 3(2 × 3 + 3 × 3+ 3 × 4) 0.902
in [Moderate] AND “Service aspect” in [Low] AND “Value aspect” 2(2 × 2 + 3 × 3) 0.867

in [Moderate] AND “Location aspect” in [Moderate] THEN “Overall 5(5,6,8,10,12) 0.961

Rating” = Moderate. ‘In the second rule, however, we have: Rule Fuzzy rule-based Ensemble + EM 4(5,6,8,10) 0.957
Ensemble 3(5,6,8) 0.937
2: Rule 1: IF “Rooms aspect” in [Moderate] AND “Check in/front
2(5,6) 0.917
desk aspect” in [Moderate] AND “Service aspect” in [Low] AND
“Value aspect” in [Moderate] AND “Location aspect” in [Low] THEN
“Overall Rating” = Low. A part of decision rules (decision trees) where actual (O), predicted (O) and n repectively indicate the rate
are visualized in Fig. 4. These decision rules were used in fuzzy provided for overall rating provided by user, predicted overall rat-
rule-based system for overall ratings prediction (see Fig. 5). ing by the recommender system and number of used user ratings.
To evaluete the fuzzy-rule based method, we used R2 (coefficient In Table 3, we presents the results of prediction evaluation by
of determination) and MSE (Mean Squared Error) for the perfor- these two metrics. The results are presented for SOM clusterings,
mace prediction of overall ratings. For coefficient of determination, SOM 2 × 2 (4 clusters), SOM 2 × 3 (6 clusters), SOM 3 × 3 (9 clusters),
this matric consider a renge of [0, 1] which closer to one indicates SOM 3 × 4 (12 clusters), and SOM 4 × 4 (16 clusters), and 5, 6, 8, 10
that the method has good performance in predicting overal ratings. and 12 clusters of EM.
The formulas of these metrics are presented as follows: The results of R2 for two methods (Fuzzy rule-based Ensem-
ble + SOM Ensemble and Fuzzy rule-based Ensemble + EM Ensemble)

n

(actual (O) − predicted (O))

2 are also presented in Table 4. All methods used Triangular and Gaus-
sian membership fumnctions. From Table 4, it can be found that
o=1
MSE = (7) good R2 values (R2 = 0.961 and R2 = 0.932) are respectively obtained
n
by Fuzzy rule-based Ensemble + EM Ensemble and Fuzzy rule-based

n
2
Ensemble + SOM Ensemble methods. In addition, the best accuracy
(actual (O) − predicted (O)) of prediction is obtained for cluster ensemble size 5 in SOM and
o=1 EM clustering. Overall, from the results of this experiment it can
R2 = 1 − (8)
n be concluded that fuzzy rule-based with clustering ensemble size 5
¯ (O))2
(actual (O) − actual outperforms fuzzy rule-based with other clustering ensemble sizes
o=1 as presented in Table 4.
 M. Nilashi et al. / Journal of Computational Science 28 (2018) 168–179 175

Fig. 4. A part of decision trees discovered from TripAdvisor data.

Fig. 5. Overall rating prediction based on features of hotels by fuzzy inference system.
176 M. Nilashi et al. / Journal of Computational Science 28 (2018) 168–179

Fig. 6. MAE and sparsity levels.

5.3. Recommendation method evaluation

We used statistical and decision support metrics to evaluate

the prediction accuracy and recommendation quality of the multi-
criteria CF recommendation agent. We split the data into two sets,
training and test sets, trained the methods on training set, and used
test set to calculate the MAE (Mean Absolute Error), Precision and
F1 metrics. The split was 60% for training and 40% for testing the
algorithms.
Our analysis results are presented in Table 5. From these anal-
yses, we can see that Fuzzy Rule-Based Ensemble + EM Ensemble
method has achieved the highest F1 value (0.934) and lowest MAE
value (0.753) compared to the previous methods in the litera-
ture. Also, we reported Precision@7 (precision at Top-7) for the
methods. The highest value for Precision@7 (0.948) is achieved by
Fuzzy Rule-Based Ensemble + EM Ensemble. From the results, it can be
found that for Fuzzy Rule-Based Ensemble + EM Ensemble the best F1
value is obtained for cluster ensemble size 5. For Fuzzy Rule-Based Fig. 7. Throughput of the methods.
Ensemble + SOM Ensemble method, the best F1 is also achieved
for the clustering with ensemble size 5. In order to compare the
In this study, we also investigate the effect of different levels
proposed method with the total-reg (total aggregation function
of sparsity on the predition accuracy of recommendation meth-
based on linear regression), standard CF (a traditional single-rating
ods. To do so, six sparsity levels are considered for the TripAdvisor
user-based CF approach) presented in Adomavicius and Kwon [41]
dataset in two groups, low level (90.5%, 91%, 91.5%, 92%, 92.5% and
and the method which use Support Vector Regression (SVR), we
93%) and high level (80.5%, 81%, 81.5%, 82%, 82.5% and 83%). Then,
also conduct a several experiments on TripAdvisor dataset and
we apply the method on the dataset with these sparsity levels and
present the results in Table 5. To provide good prediction models
accordingly compare the results with the other recommendation
by SVR, the Radial Bias Function (RBF) kernel is selected and for two
algorithms using MAE metric. As can be seen from Fig. 6a and b,
unknown parameters, C and , the best values are set using k-fold
the MAE values of two methods Fuzzy rule-based Ensemble + SOM
cross-validation (k = 10) and exhaustive search for determining SVR
Ensemble and Fuzzy rule-based Ensemble + EM Ensemble for all spar-
parameters. Specifically, we tried exponentially growing sequences
sity levels of dataset are lower than SVR, Total-Reg and Standard CF.
of C form 2−10 to 210 and  from 2−9 to 28 to find the optimal values.
In addition, from this figure it can be seen that the increasing ratio of
The prediction models (regression functions regression functions)
the MAE values for SVR, Total-Reg and Standard CF are significantly
for SVR are developed for the user- and item-based CF approaches.
higher than other methods. The results also reveal that the meth-
Then, the overall prediction was based on both regression func-
ods which use ensemble learning techniques have provided the
tions. The results of comparisons are presented in Table 5. From
best prediction accuracies compared to the other methods for the
the results presented in Table 5, it can be found that our method
dataset that is sparser. This is because of the fact that the methods
is better than the methods developed based on SVR, single-rating
which used fuzzy-rule based with clustering ensembles are more
user-based CF and total aggregation function using linear regres-
effective in solving the sparsity issue and accordingly are more
sion. Compared to the Total-Reg and Standard CF, our method which
accurate.
used multi-criteria ratings is more accurate and helps to effec-
The proposed method is also evaluated for throughput. We
tively improve the recommendation accuracy in tourism domain.
define the throughput as the number of recommendation per sec-
In addition, it can be seen that the method which used SOM with
ond. We try to consider different sizes of cluster ensembles of EM
clustering ensemble size 5(2 × 2+2 × 3+3 × 3+3 × 4+4 × 4) and EM
and SOM for this evaluation. To show the effectiveness of through-
with cluster ensemble size 5(5,681,012) are relatively provide bet-
put of our method, we compare the method with the SVR, Total-Reg
ter Precison@7, F1 and MAE values. Overall, it can be found that
and Standard CF methods. The results are presented in Fig. 7. In
our proposed method based on multi-criteria rating which relies on
fact, the plots in these figures show the throughput of the methods
ensemble learning of fuzzy-rule based and clustering techniques is
as a function of the cluster size. The results clearly show that the
accurate in providing the hotels recommendations to the users.
throughput of the proposed method which uses clustering ensem-
 M. Nilashi et al. / Journal of Computational Science 28 (2018) 168–179
Table 5
The results for F1, MAE and Precison@7.
Method Ensemble Size
Standard CF [41] – 0.643
Total-Reg [41] – 0.655
SVR – 0.813
Fuzzy rule-based Ensemble + SOM Ensemble 5(2 × 2 + 2 × 3 +3 × 3+ 3 × 4+ 4 × 4)
4(2 × 2 + 3 × 3+ 3 × 4+ 4 × 4)
3(2 × 3 + 3 × 3+ 3 × 4)
2(2 × 2 + 3 × 3)
Fuzzy rule-based Ensemble + EM Ensemble 5(5,6,8,10,12)
4(5,6,8,10)
3(5,6,8)
2(5,6)
ble techniques and ensembles of fuzzy rule-based technique is
higher than Total-Reg and Standard CF methods. In addition, the
throughput of the proposed method is much better than other
methods by incorporating the EM clustering technique for data
clustering. Specifically, the method provides the best throughput
when the clustering ensemble size is 5. Overall, we can conclude
that the clustering techniques can significantly improve the scala-
bility of the recommendation systems in tourism domain.
6. Discussion
The online reviews in public social network websites can be
helpful in travelers’ decision-making process. They can be effec-
tively used in machine leaning techniques to make hotel-choice
decisions for the travelers. Improving the accuracy of collabora-
tive filtering systems is an important task that is highly considered
in the recent research in the context of recommender systems.
Multi-criteria CF recommender systems have proven to be effec-
tive in recommending the products to the user with high accuracy.
Accordingly, some limited studies have focused to incorporate the
multi-criteria rating into the CF systems to take its advantages in
improving the performance of CF-based recommender systems.
This research also aimed at developing a tourism recommendation
agent based on multi-criteria CF by the use of supervised and unsu-
pervised machine learning techniques. Our aim was to accurately
predict the overall rating of users on hotels based on multi-criteria
ratings with a new approach. A main contribution of our work was
generating the decision rules from the data using decision trees
technique. From our analysis on the TripAdvisor data, we found
that this technique is effective to perform this task and discover
the appropriate decision rules. Most importantly, we could use
these decision rules in fuzzy rule-based system of proposed rec-
ommender system to be used for the prediction of overall ratings.
To show the effectiveness of using the incorporated techniques
in the proposed recommender system, several experiments on a
real-world dataset, TripAdvisor, showed that accurate prediction
of overall ratings can significantly improve the recommendation
accuracy in tourism domain. In addition, the results of our experi-
ments on the TripAdvisor data showed that our method can better
work for the dataset which are sparser. Furthermore, the cluster-
ing techniques provided scalable recommendations of hotels. These
can benefit the tourism recommender systems to overcome the
sparsity issues and help them in predicting the hotels which have
been rated by small number of users. Overall, as the use of multi-
criteria rating based on our analysis demonstrated to be effective in
improving the performance of tourism recommender systems, our
proposed method can be potentially implemented in tourism web-
sites for accurate recommendation of hotels to the users in their
decision making process.
177
Precison@7 F1 MAE
0.638 1.276
0.647 1.189
0.793 0.893
0.908 0.897 0.788
0.901 0.891 0.795
0.889 0.885 0.802
0.886 0.879 0.806
0.948 0.934 0.753
0.927 0.918 0.772
0.919 0.911 0.776
0.909 0.901 0.781
7. Conclusion and future work
Social media tools have played an important role in the travel
planning process and customers’ travel choices. They have pro-
vided useful distribution channels for tourism marketers to better
promote their products to online travelers. They suggest differ-
ent channels for customers to share their feedback and reviews
on tourism products. These reviews are important for the travelers
who want to book the hotels before a trip. The aim of this work is to
develop a new method to use the travelers’ reviews on the hotels
recommend them appropriate hotels tailored to their preference.
Our method is based on multi-criteria ratings. The ratings are col-
lected from TripAdvisor. We used EM and SOM for clustering the
users’ ratings on hotels and fuzzy rule-based technique for the pre-
diction to predict the customers’ choices for hotels in TripAdvisor.
We developed the clustering and prediction techiques incorporated
to the recommendation method for ensemble learnig of the data.
The proposed recommendation method was evaluated on TripAd-
visor data using MAE, Precision and F1 metrics. The results of the
method evaluation confirmed that the proposed recommendation
agent by the use of EM and SOM ensemble with ensemble of fuzzy
rule-based technique achieve the best prediction accuracy com-
pared to the other methods. The results also demonstrated that the
proposed user- and item based recommendation method is effec-
tive in solving sparsity problem of multi-criteria CF. Our approach
could also further reduce the sparsity by the use of clustering and
fuzzy rule-based techniques. Our empirical analysis was on TripAd-
visor data, however future work could be developed to adapt the
proposed approach for the other contexts such as movie and music
recommendations. In addition, in this study we did not investi-
gate the scalability issue of multi-criteria CF, hence future work
can be conducted for incremental learning of prediction models in
fuzzy rule-based system to overcome the scalability drawback of
recommender systems.
References
[1] N. Chung, C. Koo, The use of social media in travel information search,
Telemat. Inform. 32 (2) (2015) 215–229.
[2] S. Okazaki, L. Andreu, S. Campo, Knowledge sharing among tourists via social
media: a comparison between facebook and TripAdvisor, Int. J. Tour. Res. 19
(1) (2017) 107–119.
[3] TIA, Travelers’ Use of the Internet, Travel Industry Association of America,
Washington, DC, 2005.
[4] M. Sigala, E. Christou, U. Gretzel, Social Media in Travel, Tourism and
Hospitality: Theory, Practice, and Cases, Printed and Bound in Great Britain by
the MPG Books Group, UK, 2012.
[5] M. Nilashi, M. Salahshour, O. Ibrahim, A. Mardani, M.D. Esfahani, N. Zakuan, A
new method for collaborative filtering recommender systems: the case of
yahoo! Movies and tripadvisor datasets, J. Soft Comput. Decis. Support Syst. 3
(5) (2016) 44–46.
[6] M. Nilashi, K. Bagherifard, M. Rahmani, V. Rafe, A recommender system for
tourism industry using cluster ensemble and prediction machine learning
techniques, Comput. Ind. Eng. 109 (2017) 357–368.
178 M. Nilashi et al. / Journal of Computational Science 28 (2018) 168–179
[7] G. Bordogna, G. Pasi, A flexible multi criteria information filtering model, Soft
comput. 14 (8) (2010) 799–809.
[8] M. Rashidi, A.R.C. Hussin, M. Nilashi, Entropy-based ranking approach for
enhancing diversity in tag-based community recommendation, J. Soft
Comput. Decis. Support Syst. 3 (1) (2015) 1–7.
[9] C. Mettouris, G.A. Papadopoulos, Ubiquitous recommender systems,
Computing 96 (3) (2014) 223–257.
[10] Q. Shambour, M.A. Hourani, S. Fraihat, An item-based multi-criteria
collaborative filtering algorithm for personalized recommender systems, Int.
J. Adv. Comput. Sci. Appl. 7 (8) (2016) 274–279.
[11] M. Nilashi, O. bin Ibrahim, N. Ithnin, Hybrid recommendation approaches for
multi-criteria collaborative filtering, Expert Syst. Appl. 41 (8) (2014)
3879–3900.
[12] M. Nilashi, O. bin Ibrahim, N. Ithnin, Multi-criteria collaborative filtering with
high accuracy using higher order singular value decomposition and
neuro-fuzzy system, Knowl. Based Syst. 60 (2014) 82–101.
[13] M. Nilashi, D. Jannach, O. bin Ibrahim, N. Ithnin, Clustering-and
regression-based multi-criteria collaborative filtering with incremental
updates, Inform. Sci. 293 (2015) 235–250.
[14] A.B. Barragáns-Martínez, E. Costa-Montenegro, J.C. Burguillo, M. Rey-López,
F.A. Mikic-Fonte, A. Peleteiro, A hybrid content-based and item-based
collaborative filtering approach to recommend TV programs enhanced with
singular value decomposition, Inform. Sci. 180 (2010) 4290–4311.
[15] E. Davoodi, K. Kianmehr, M. Afsharchi, A semantic social network-based
expert recommender system, Appl. Intell. 39 (1) (2013) 1–13.
[16] U. Rabanser, F. Ricci, Recommender Systems: Do They Have a Viable Business
Model in e-tourism? Information and Communication Technologies in
Tourism 2005, 2005, pp. 160–171.
[17] D. Jannach, Z. Karakaya, F. Gedikli, Accuracy improvements for multi-criteria
recommender systems, Proceedings of the 13th ACM Conference on
Electronic Commerce (2012) 674–689.
[18] M. Fuchs, M. Zanker, Multi-criteria ratings for recommender systems: an
empirical analysis in the tourism domain, September, International
Conference on Electronic Commerce and Web Technologies (2012) 100–111.
[19] S. Loh, F. Lorenzi, R. Saldaña, D. Licthnow, A tourism recommender system
based on collaboration and text analysis, Inf. Technol. Tour. 6 (3) (2003)
157–165.
[20] E. Bigdeli, Z. Bahmani, Comparing accuracy of cosine-based similarity and
correlation-based similarity algorithms in tourism recommender systems,
September, Management of Innovation and Technology, 2008. ICMIT 2008.
4th IEEE International Conference on (2008) 469–474.
[21] S. Esparcia, V. Sánchez-Anguix, E. Argente, A. García-Fornes, V. Julián,
Integrating information extraction agents into a tourism recommender
system, Hybrid Artificial Intelligence Systems (2010) 193–200.
[22] M. Kenteris, D. Gavalas, A. Mpitziopoulos, A mobile tourism recommender
system, June, Computers and Communications (ISCC), 2010 IEEE Symposium
on (2010) 840–845.
[23] Y. Blanco-Fernandez, M. Lopez-Nores, J.J. Pazos-Arias, A. Gil-Solla, M.
Ramos-Cabrer, Exploiting digital TV users’ preferences in a tourism
recommender system based on semantic reasoning, IEEE Trans. Consum.
Electron. 56 (2) (2010).
[24] M. Al-hassan, H. Lu, J. Lu, Personalized e-government services: tourism
recommender system framework, April, International Conference on Web
Information Systems and Technologies (2010) 173–187.
[25] Y. Chu, H. Wang, L. Zheng, Z. Wang, K.L. Tan, TRSO: a tourism recommender
system based on ontology, October, International Conference on Knowledge
Science, Engineering and Management (2016) 567–579.
[26] C.S. Namahoot, N. Panawong, M. Brückner, A tourism recommendation
system for Thailand using semantic web rule language and K-NN algorithm,
Int. Inform. Inst. (Tokyo) Inform. 19 (7B) (2016) 3017.
[27] X. Zheng, Y. Luo, Z. Xu, Q. Yu, L. Lu, Tourism destination recommender system
for the cold start problem, KSII Trans. Internet Inform. Syst. 10 (7) (2016).
[28] Q.M. Zhang, X. Yu, J. Wang, Temporal recommender system in online tourism
website, Proceedings of the International MultiConference of Engineers and
Computer Scientists vol. 1 (2016).
[29] A.M. Kashevnik, A.V. Ponomarev, A.V. Smirnov, A multimodel context-aware
tourism recommendation service: approach and architecture, J. Comput. Syst.
Sci. Int. 56 (2) (2017) 245–258.
[30] Q. Li, C. Wang, G. Geng, Improving personalized services in mobile commerce
by a novel multicriteria rating approach, April, Proceedings of the 17th
International Conference on World Wide Web (2008) 1235–1236.
[31] L. Liu, N. Mehandjiev, D.L. Xu, Multi-criteria service recommendation based
on user criteria preferences, October, Proceedings of the Fifth ACM
Conference on Recommender Systems (2011) 77–84.
[32] C.F. Tsai, C. Hung, Cluster ensembles in collaborative filtering
recommendation, Appl. Soft Comput. 12 (4) (2012) 1417–1425.
[33] Y. Koren, Factor in the neighbors: scalable and accurate collaborative filtering,
ACM Trans. Knowl. Disc. Data (TKDD) 4 (1) (2010) 1.
[34] F. Gedikli, F. Bagdat, M. Ge, D. Jannach, RF-REC: fast and accurate computation
of recommendations based on rating frequencies, Seventh IEEE International
Conference on E-Commerce Technology (CEC) (2011) 50–57.
[35] T. Kohonen, E. Oja, O. Simula, A. Visa, J. Kangas, Engineering applications of
the self-organizing map, Proceedings of the IEEE 84 (10) (1996) 1358–1384.
[36] A.P. Dempster, N.M. Laird, D.B. Rubin, Maximum likelihood for incomplete
data via the EM algorithm, J. R. Stat. Soc. Ser. B 39 (1977) 1–38.
[37] L. Breiman, J.H. Friedman, R.A. Olshen, C.J. Stone, Classification and Regression
Trees, Wadsworth Belmont, CA, 1984.
[38] A. Strehl, J. Ghosh, Cluster ensembles-a knowledge reuse framework for
combining partitioning, J. Mach. Learn. Res. 3 (December) (2002) 583–617.
[39] L.A. Zadeh, Fuzzy sets, Inf. Control. 8 (3) (1965) 338–353.
[40] P. O’Connor, User-generated Content and Travel: a Case Study on
Tripadvisor.cOm. Information and Communication Technologies in Tourism
2008, 2008, pp. 47–58.
[41] G. Adomavicius, Y. Kwon, New recommendation techniques for multicriteria
rating systems, IEEE Intell. Syst. 22 (3) (2007).
[42] L.I. Kuncheva, Classifier ensembles for changing environments. In
International Workshop on Multiple Classifier Systems (pp. 1-15), Springer,
Berlin, Heidelberg, 2004.
[43] J. Soto, P. Melin, O. Castillo, A new approach for time series prediction using
ensembles of ANFIS models with interval type-2 and type-1 fuzzy integrators,
in: Computational Intelligence for Financial Engineering & Economics (CIFEr)
IEEE Conference on (pp. 68-73), IEEE, 2013.
[44] D.H. Park, H.K. Kim, I.Y. Choi, J.K. Kim, A literature review and classification of
recommender systems research, Expert Systems with Applications 39 (11)
(2012) 10059–10072.
[45] A.K. Jain, Data clustering: 50 years beyond K-means, Pattern recognition
letters 31 (8) (2010) 651–666.
[46] M.A. Ghazanfar, A. Prügel-Bennett, Leveraging clustering approaches to solve
the gray-sheep users problem in recommender systems, Expert Systems with
Applications 41 (7) (2014) 3261–3275.
[47] K. Woods, W.P. Kegelmeyer, K. Bowyer, Combination of multiple classifiers
using local accuracy estimates, IEEE transactions on pattern analysis and
machine intelligence 19 (4) (1997) 405–410.
Mehrbakhsh Nilashi received his Ph.D. degree in Com-
puter Science in the faculty of Computing, Universiti
Teknologi Malaysia in 2014. His researches are mainly in
the fields of Soft Computing, Machine Learning, Multi-
criteria Decision Making, Information Retrieval, Recom-
mender Systems with a special focus on Multi-Criteria
Recommender Systems, Health Informatics, tourism man-
agement and Decision Support Systems. His contributions
have been published in prestigious peer-reviewed jour-
nals and international conferences.
Othman Ibrahim is an Associate Professor in the faculty
of computing, Universiti Teknologi Malaysia (UTM). His
research interests are in Information Systems, Information
Systems Theory, Knowledge Management, E-Government
and E-Business. He received his Ph.D. degree from UMIST
(University of Manchester Institute of Science and Tech-
nology.
Elaheh Yadegaridehkordi received her PhD degree in
Information Systems in the faculty of Computing, Uni-
versiti Teknologi Malaysia in 2015. Her research interests
include tourism management, cloud computing, mobile
learning, and recommendation systems. Her contribu-
tions have been published in prestigious peer-reviewed
journals and international conferences.
Sarminah Samad is an Associate Professor in CBA
Research Centre, Department of Business Administration,
Collage of Business and Administration, Princess Nourah
Bint Abdulrahman University, Riyadh, Saudi Arabia. Her
research expertise is focused on Technology Adoption,
Social Informatics, and Business Research. Her contribu-
tions have been published in prestigious peer-reviewed
journals and international conferences.
 M. Nilashi et al. / Journal of Computational Science 28 (2018) 168–179 179

Elnaz Akbari is a postdoctoral researcher in the field
of electrical engineering . Her major fields of research
activity include nanomaterials based sensors and machine
learning in nanomaterials. Her contributions have been
published in prestigious peer-reviewed journals and
international conferences.
Azar Alizadeh is a PhD student in the School of Engineer-
ing, University of California, Merced, USA. Her research
interests are in the Information Systems and Human Com-
puter Interaction. She published several papers in the
prestigious journals.