STUDY OF ACCURACY METRICS

ACCURACY METRICS
Accuracy metrics are essential for evaluating the performance of recommender systems,
whether it's assessing the accuracy of predicted ratings for user-item combinations or the quality
of top-k recommendations. These metrics help gauge how well the system is at providing
relevant and accurate suggestions.

ACCURACY OF ESTIMATED RATINGS

In recommender systems, accuracy metrics are crucial for assessing the quality of
recommendations, and they can be categorized based on whether they evaluate rating predictions
or top-k rankings. When evaluating rating predictions, the ratings of a set E of entries in the
ratings matrix are typically hidden, and the goal is to measure how well the system's predictions
match the actual ratings.

● Mean Squared Error (MSE) : MSE is a common metric used to evaluate the accuracy
of estimating ratings. For each user-item combination (u, j) in the set E, the error euj is
computed as the difference between the predicted rating (ˆruj) and the actual rating (ruj).
MSE is defined as the average squared error over all entries in set E, divided by the size
of the set.

● Root Mean Squared Error (RMSE) : RMSE is the square root of the MSE, providing a
more interpretable measure of the prediction error.
 ● Mean Absolute Error (MAE) : MAE is an alternative to MSE that measures the
absolute difference between predicted and actual ratings, making it less sensitive to
outliers.

ACCURACY OF ESTIMATED RANKINGS

In some cases, recommender systems focus on estimating rankings rather than explicit ratings. In
these scenarios, different accuracy metrics are used to assess the quality of the predicted
rankings.

● Rank-Correlation Measures : Rank-correlation measures, such as Spearman's rank

correlation coefficient, evaluate the similarity between the predicted rankings and the
ground-truth rankings. These metrics assess how well the recommender system orders
items compared to the actual preferences.

● Utility-Based Measures : Utility-based measures take into account the utility (e.g.,
relevance, satisfaction) of the items in the predicted rankings. They aim to capture the
overall quality of recommendations beyond just ranking.

● Receiver Operating Characteristic (ROC) : The ROC curve and related measures are
used when dealing with unary (implicit feedback) data sets, where the focus is on the
ability to discriminate between relevant and irrelevant items. The ROC curve plots true
positive rate against false positive rate.