Evaluating Recommendations

When a recommendation algorithm predicts ratings for items, one must evaluate
how accurate its recommendations are.
(1) accuracy of predictions,
(2) relevancy of recommendations, or
(3) rankings of recommendations
 1. Evaluating Accuracy of Predictions
• In this, we can measure how close predicted ratings are to the true
ratings.
• It is similar to supervised learning – in which true ratings are given.
• Based on predictions ratings, compare with true ratings.
• For comparing ,the simplest method used.- mean absolute error (MAE).
• It computes the average absolute difference between the predicted
ratings and true ratings.
• where n is the number of predicted ratings
• ˆ r’ij is the predicted rating.
• rij is the true rating
 1. Evaluating Accuracy of Predictions
• Normalized mean absolute error (NMAE) normalizes MAE by dividing it by
the range ratings can take,
• where rmax is the maximum rating items can take
• and rmin is the minimum.
• the summation of squared errors in the root mean squared error (RMSE):
1. Evaluating Accuracy of Predictions
 2.Evaluating Relevancy of
Recommendations
• In this, we ask users if they find the recommended items relevant to
their interests.
• Given a set of recommendations to a user, the user describes each
recommendation as relevant or irrelevant.
• Based on the selection of items, we can have four types of items as
mentioned in the table below.
 2.Evaluating Relevancy of
Recommendations
• Given this table, we can define measures that use relevancy information
provided by users. Precision is one such measure.
• It defines the fraction of relevant items among recommended items:
•

• Similarly, we can use recall to evaluate a recommender algorithm, which

provides the probability of selecting a relevant item for recommendation:

• We can also combine both precision and recall by taking their harmonic
mean in the F-measure:
2.Evaluating Relevancy of
Recommendations
 3.Evaluating Ranking of
Recommendations
• Often, we predict ratings for multiple products for a user.
• Based on the predicted ratings, we can rank products based on their
levels of interest of the user and then evaluate this ranking.
• Comparing the ranking of products using rank correlation.
• Rank correlation measures the correlation between the predicted
ranking and the true ranking.
• One such technique is the Spearman’s rank correlation.
• Let xi, 1<= xi<= n, denote the rank predicted for item i, 1<= i<= n.
• Similarly, let yi, 1<= yi<= n, denote the true rank of item i from the
user’s perspective. then Spearman’s rank correlation is defined as
3.Evaluating Ranking of
Recommendations
• where n is the total no.of.items.
• Another rank correlation measure, is Kendall’s tau.
• We say that the pair of items (i ,j) are concordant if their ranks (xi, yi),
and (xj ,yj) are in order ,

• A pair of items is discordant if their corresponding ranks are not in

order:
 3.Evaluating Ranking of Recommendations
• When xi = xj or yi = yj, the pair is neither concordant nor discordant.
• Let ‘c’ denote the total number of concordant item pairs and
• ‘d’ the total number of discordant item pairs.
• Kendall’s tau computes the difference between the two, normalized by the total
number of item pairs

• Kendall’s tau takes value in range [ -1, 1].

• When the ranks completely agree, all pairs are concordant and Kendall’s tau takes
value 1, and
• when the ranks completely disagree, all pairs are discordant and Kendall’s tau takes
value 1.
3.Evaluating Ranking of
Recommendations
3.Evaluating Ranking of
Recommendations
• We get the number of concordant pairs by 1 3 counting all the "+".
•21
• 34++
•42+
• 5 10 6 + + + +
• 6 5 LO + + + +