This article has been accepted for inclusion in a future issue of this journal.
Content is final as presented, with the exception of pagination.
IEEE Transactions on Broadcasting ( Volume: 64, Issue: 2, June 2018 )
IEEE TRANSACTIONS ON BROADCASTING
Toward Better Statistical Validation of Machine
Learning-Based Multimedia Quality Estimators
Manish Narwaria
Abstract—Objective assessment of multimedia quality using
machine learning (ML) has been gaining popularity especially in
the context of both traditional (e.g., terrestrial and satellite broad-
cast) and advance (such as over-the-top media services, IPTV)
broadcast services. Being data-driven, these methods obviously
rely on training to find the optimal model parameters. Therefore,
to statistically compare and validate such ML-based quality pre-
dictors, the current approach randomly splits the given data into
training and test sets and obtains a performance measure (for
instance mean squared error, correlation coefficient etc.). The
process is repeated a large number of times and parametric tests
(e.g., t test) are then employed to statistically compare mean (or
median) prediction accuracies. However, the current approach
suffers from a few limitations (related to the qualitative aspects
of training and testing data, the use of improper sample size for
statistical testing, possibly dependent sample observations, and a
lack of focus on quantifying the learning ability of the ML-based
objective quality predictor) which have not been addressed in lit-
erature. Therefore, the main goal of this paper is to shed light on
the said limitations both from practical and theoretical perspec-
tives wherever applicable, and in the process propose an alternate
approach to overcome some of them. As a major advantage,
the proposed guidelines not only help in a theoretically more
grounded statistical comparison but also provide useful insights
into how well the ML-based objective quality predictors exploit
data structure for learning. We demonstrate the added value of
the proposed set of guidelines on standard datasets by comparing
the performance of few existing ML-based quality estimators. A
software implementation of the presented guidelines is also made
publicly available to enable researchers and developers to test and
compare different models in a repeatable manner.
Index Terms—Multimedia quality, machine learning, statistical
analysis.
I. I NTRODUCTION
ULTIMEDIA signals have become a part of our daily
M lives, thanks to the availability of low cost devices cou-
pled with the rapid growth of traditional and advanced multi-
media broadcast services. In particular, fixed/mobile advanced
media delivery fueled by the emergence of IPTV, cloud ser-
vices and over-the-top (OTT) media services has enabled the
consumers to enjoy more immersive viewing experience of
3DTV, HDR, 4K etc., from the comfort of their premises.
As a result, our interaction with multimedia has not only
Manuscript received December 30, 2017; revised April 12, 2018; accepted
April 19, 2018.
The author is with the Department of ICT, Dhirubhai Ambani Institute
of Information and Communication Technology, Gandhinagar 382007, India
(e-mail: manish_narwaria@daiict.ac.in).
Color versions of one or more of the figures in this paper are available
online at http://ieeexplore.ieee.org.
Digital Object Identifier 10.1109/TBC.2018.2832441
0018-9316 c 2018 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
1
increased quantitatively but the quality of such interaction
has also evolved. In particular, todays end users are more
demanding in terms of their multimedia experience, and per-
ceptual quality is one of the intrinsic factors that affects such
interaction. Thus, assessment of perceptual quality is an impor-
tant aspect in todays multimedia communication systems [1].
To that end, subjective assessment performed by human sub-
jects is still considered the most accurate methodology and
remains the most reliable and accurate method, given appro-
priate laboratory conditions and a sufficiently large subject
panel. However, subjective assessment may not be feasible
in certain situations (e.g., real-time multimedia transmission),
and an objective approach is more suitable in such scenarios.
Objective assessment of multimedia quality involves the use
of computational models which are expected to predict quality
scores in a repeatable fashion and such that the objective pre-
dictions align well with the subjective opinion of perceptual
signal quality. It is however important to stress that objec-
tive approaches may not exactly mimic the subjective opinion
in all situations, and are not meant to entirely replace sub-
jective assessment. Instead they can provide approximate and
relative estimates of perceptual quality, within the context of
the applications such as DTT broadcast, IPTV, multimedia
compression etc.
While there has been substantial research effort towards
developing objective quality estimators for multimedia sig-
nals (including single or multi modal signals such as image,
video, speech, audiovisual, graphics etc.), issues related to how
closely the human opinion can be mimicked and those related
to computational efficiency (these have obvious consequences
on practical deployment) exist. In that context, a data driven
approach has also been viewed as a plausible solution. Even
though interest in such methods has existed for several years,
there have been renewed and concerted efforts to exploit such
data driven methods for the said purpose [2]–[12].
The use of ML for objective quality estimation is partic-
ularly suitable for broadcast applications where the quality
of the received or transmitted content needs to be assessed
objectively based on limited signal information. Not surpris-
ingly, ML has been exploited in the past for the said purpose.
Gastaldo et al. [2] presented one of the first comprehensive
methods for estimating quality of MPEG video streams, and
is based on circular back propagation neural networks. A no
reference method was presented in [3], which is based on
mapping frame-level features into a spatial quality score fol-
lowed by temporal pooling. The method developed in [4] is
based on features extracted from the analysis of discrete cosine
 This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
IEEE Transactions on Broadcasting ( Volume: 64, Issue: 2, June 2018 )
2 IEEE TRANSACTIONS ON BROADCASTING
transform (DCT) coefficients of each decoded frame in a video
sequence, and subsequent quality prediction using a neural
network. Another ML based video quality estimator was pre-
sented in [5] where symbolic regression-based frame work
was trained on a set of features extracted from the received
video bit stream. The ML based quality estimator proposed
in [6] works on the similar principle of analyzing several fea-
tures such as distinguishing the type of codec used (MPEG
or H.264/AVC), DCT coefficients, estimation of the level of
quantization used in the I-frames, etc. The next step is to apply
support vector regression to predict video quality. The objec-
tive quality estimator proposed in [7] was based on polynomial
regression model, where the independent variables (or fea-
tures) were based on spatial and temporal quantities derived
from video spatiotemporal complexity, bitrate, and packet loss
measurements. Mocanu et al. [11] employed deep learning
(deep belief networks) and bit stream specific features to pre-
dict quality objectively in a video transmission network. Deep
learning has also been employed for quality measurement in
live video streaming [12]. Moreover, promising results from
related disciplines such as computer vision and the availabil-
ity of required hardware (e.g., GPU-accelerated computing)
have opened up possibilities of developing efficient ML based
implementations of quality predictors.
For the case of objective multimedia quality assessment, the
use of ML methods is a two-stage process: feature extraction
(representing the given multimedia data via a set of per-
ceptually meaningful and possibly lower dimensional feature
values) and feature pooling (combining or fusing of features
into a quality score). The second stage typically uses regres-
sors, and hence the objective quality predictions (scores) are
continuous (such scores can of course be further binarized via
thresholding or can be used for pairwise stimuli comparisons).
More recently, deep networks (such as the convolutional neural
networks, deep belief networks etc.) have also gained popu-
larity [11], [12] where feature extraction process is implicitly
handled by the ML method (instead of using hand-crafted
features).
Irrespective of whether objective quality estimators use ML
or not, statistical testing plays an important role in their valida-
tion and benchmarking. Such validation studies are obviously
crucial before objective predictors can be deployed in practice.
Note that statistical tests (both parametric and nonparametric)
are extensively used not only to validate objective methods
against subjective data but also to statistically compare two or
more objective quality predictors, in order to find the better
metric for the given application or to rank them.
Regarding the statistical comparison and validation of non-
ML based estimators (i.e., which do not require any training)
the procedure has been by and large standardized (e.g., the
ITU recommendations P.1401 [13], J.149 [14], [15] or VQEG
recommendations [16]) and uses a performance measure such
as such correlation coefficient (Pearson, Spearman etc.), the
root mean squared error (RMSE), outlier ratio etc., to quan-
tify the agreement between subjective opinion and objective
predictions (or comparing those metrics from several objective
methods). Then, statistical inferences are drawn (or differ-
ent methods compared) by using confidence intervals (CIs)
for correlation coefficients, using F-test on the residuals,
the use of ANOVA and t test [15], [17], determining the
significance of the difference between the outlier ratios or
RMSEs [13]. Likewise, recommendation ITU-T J.149 [14]
uses classification errors of quality metric.
As far as the statistical comparison and validation of ML
based quality predictors is concerned, the current approach is
based on repeated and random splits of data (i.e., predictions
from ML based methods and the corresponding subjective
scores for the given multimedia content) into training and test
sets [2]–[12]. In each iteration, a performance measure (like
mean squared error, correlation coefficient etc.) is obtained.
Then, the means (or in some cases median) of such repeated
performance measure for each ML based estimator are sta-
tistically compared via pairwise t. However, because of the
requirement of training the current approach needs to be exam-
ined more closely in terms of the factors that can affect the
validation process. These include qualitative aspects of train-
ing and testing data, determining the appropriate sample size
when splitting the given data into training and test sets, the
issue of possibly dependent sample observations and the anal-
ysis pertinent to the learning ability of the method (note that
these issues are not relevant in case of statistical comparison of
non-ML based predictors because there is no training involved
and hence a question of train-test split typically does not arise).
A survey of literature (e.g., refer to [2]–[12] for some existing
efforts in ML based quality estimation for video or [13]–[16]
for standardized recommendations) reveals that these impor-
tant issues have not been thoroughly examined (either from
theoretical or practical view points) although few works such
as [4], [9], and [11] have considered the practical implications
of the first issue regarding the qualitative aspects of training
and testing data (also refer to some related works on statisti-
cal comparison of classifiers [18] or analysis of their learning
ability [19]).
Therefore, the main aim of the paper is to shed light on
these factors, and in the process present a set of new guide-
lines to overcome the drawbacks of the current approach.
The proposed guidelines offer the advantage of focusing on
practical use-case scenario and quantifying the learning abil-
ity of the ML based quality estimator. Therefore, the use of
these guidelines helps to make more informed conclusions and
recommendations about metric performance. In contrast, the
existing approach tends to treat ML based methods as black
boxes and focuses primarily on global, binary decisions about
metric performance. A software implementing the presented
guidelines is also made publicly available,1 in order to achieve
the goal of reproducible research.
The remainder of the paper is organized as follows.
Section II discusses the limitations and additional consider-
ations in statistical validation of ML based quality predictors.
Following this, we present in Section III a theoretical anal-
ysis concerning dependent (correlated) sample observations
and how that affects the sampling distribution of the t test
statistic. Next, Section IV proposes a framework (set of guide-
lines) for more accurate statistical comparison and validation
1 https://sites.google.com/site/narwariam/home/research
 This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
IEEE Transactions on Broadcasting ( Volume: 64, Issue: 2, June 2018 )
NARWARIA: TOWARD BETTER STATISTICAL VALIDATION OF ML-BASED MULTIMEDIA QUALITY ESTIMATORS
Fig. 1. The current approach for statistical comparison and validation of M ML based quality estimators. The drawbacks associated with this approach are
discussed in Section II.
of ML based quality predictors to ameliorate some of the
discussed drawbacks. Section V presents a permutation test
based method for systematic analysis of ML based quality
predictors. Experimental results and analysis are reported in
Section VI. We provide concluding thoughts in Section VII.
II. L IMITATIONS OF THE C URRENT A PPROACH FOR
S TATISTICAL VALIDATION OF ML BASED
Q UALITY P REDICTORS
The main steps involved in the current approach for statisti-
cal comparison and validation of ML based quality estimators
are shown in Figure 1. Note that most existing papers [2]–[12]
follow this approach.
First, the given subjective data (comprising of multimedia
content and the corresponding subjective scores) is randomly
split into train and test sets (assume that the number of samples
(stimuli) in train and test sets are respectively Ntrain and Ntest )
Niter times. It is typically ensured that the content in training
and test sets do not overlap. In each iteration, the ML based
methods are trained on the training set and assessed for their
prediction performance on the test set. This results in a set
of Niter performance measures (e.g., correlation coefficient,
mean squared error, percentage of correctly classified content
etc.) for each ML based quality estimator. Then, a descriptive
statistic (mean, median etc.) of the performance measure in
each iteration is taken as the overall performance measure.
Suppose our goal is to compare the performance of M ML
based quality predictors. Denote the performance measure for
(j)
such comparison as ρi where i is the iteration index i =
1, . . . , Niter and j = 1, . . . , M. Then, the matrix to be analyzed
will of the form
⎡ ⎤
(1) (M)
ρ1 · · · ρ1
⎢ . .. .. ⎥
A=⎢ ⎣ .
. . ⎥
. ⎦ (1)
(1) (M)
ρNiter · · · ρNiter
In order to carry out a statistical comparison, t test is
applied to compare the mean values (or the Wilcoxon test for
median) of each column of matrix A in a pairwise manner.
3
This provides a symmetric M × M decision matrix D in which
the entry (i, j) set to 0 if the null hypothesis is accepted (i.e.,
the methods in row i and column j are deemed to be statisti-
cally similar according to the t test) or 1 otherwise. Therefore,
D is simply written as
⎡ ⎤
− ··· 1
⎢ .. ⎥
D = ⎣ ... ..
. .⎦ (2)
1 ··· −
As the same method is not compared to itself via t test,
the diagonal entries in D are not considered. Inferences on
relative method performance are then drawn from the deci-
sion matrix D. However, such an approach of comparing
ML based quality estimators suffers from a few limitations
that possibly render the conclusions and inferences question-
able. Specifically, following drawbacks can be identified in the
current approach.
A. Violation of Assumption of Independent Observations
The application of statistical tests such as the t test, ANOVA
etc. assumes that the observations in each test sample are inde-
pendent and identically distributed random variables (iid) [20].
However, notice that while the values in each column of
matrix A are generated from a random test sample, the pop-
ulation from which the said test sample is drawn is severely
limited both in terms of the variety in content and the size. The
reason is that typical subjective tests include limited source
content for obvious practical reasons. Note that even with
newer paradigms such as crowd sourcing, the population size
may not grow exponentially that will guarantee independence
of the random test samples on which the ML based predictors
are tested. Moreover, crowd sourcing can introduce additional
factors into the subjective rating process (as it offers no con-
trol in terms of display, ambient light, viewing distance etc.).
A limited population size will therefore be almost always a
challenge for statistical testing. Particularly, it can possibly
result in overlapping content, i.e., test sets across iterations
can share one or more test conditions. For instance, images
corresponding to a reference (source) image can appear in the
 This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
IEEE Transactions on Broadcasting ( Volume: 64, Issue: 2, June 2018 )
4 IEEE TRANSACTIONS ON BROADCASTING
(j)
testing set more than once. Consequently, the ρi values in
the j-th column of A are not entirely independent. This can
possibly result in correlated samples observations. That is, the
performance measure in one iteration can be relatively easily
predicted if same/similar images will appear in the test set in
another iteration. This may render the statistical comparison
and subsequent inferences questionable especially if the said
correlation is high.
In general and to our best knowledge, the said issue has
not received sufficient research attention especially from the
view point of objective multimedia signal quality estimation.
We will therefore examine it more closely from a theo-
retical perspective in Section III, and propose a solution
in Section IV.
B. The Issue of Arbitrary Sample Size
The reader will note that the dimensionality of each column
of A is Niter . It implies that mean (or median) is computed
(j)
over Niter observations (i.e., ρi values). Consequently, the
sample size for the t test depends on the number of iterations
Niter . Since sample size is used to compute the test statistic,
it implies that the decision of the statistical test (i.e., accep-
tance or rejection of null hypothesis) can be influenced by
the arbitrary choice of Niter . In particular, as the value of the
test statistic varies inversely with the sample size (this can be
easily seen from the definition of the test statistics in t test,
ANOVA etc. [20]), the chances of rejecting the null hypothesis
can be increased by simply choosing a large Niter . In the lim-
iting case, as Niter → ∞, all the entries of the decision matrix
D will tend to 1. Practically, it implies that an infinitesimal
(j)
increase (decrease) in mean of ρi values will lead to the
conclusion that there is statistically significant performance
difference between the corresponding pair of ML based qual-
ity predictors. However, such conclusion is not reliable as it
may be due to a large Niter , and not necessarily reflect sta-
tistically meaningful differences in prediction performances
(we provide an example later in Section VI to support our
arguments).
C. Lack of Practical Considerations
The third drawback of the current approach concerns lack of
practical considerations. Specifically, two points can be raised
in this regards.
First, the random train-test splitting (used in the current
approach) does not provide adequate control towards incor-
porating practically useful and independent conditions in the
test set. As a result, the comparison of ML based quality pre-
dictors might be based on non-exhaustive set of conditions.
While increasing the number of randomizations by using a
large value Niter may mitigate the said problem to some extent,
it can influence the statistical decisions (as discussed in the
previous drawback).
Second, we note that the current approach simply empha-
sizes theoretical (statistical) differences. That is, it provides
the information if a treatment effect (in our case, the term
treatment effect implies the magnitude of the difference in per-
formance measures of ML based estimators) is present or not.
However, it does not give an explicit insight into how large the
treatment effect is, and whether the observed effect is practi-
cally meaningful or not. Obviously, such information is crucial
to make a more balanced and practically reasonable conclu-
sion about the performance of ML based quality predictors. It
is therefore a recommended practice [21]–[24] that p values
be supplemented by effect size so that the results of statistical
tests are more fairly interpreted. This important point is, unfor-
tunately, either completely ignored or not emphasized enough
in the current approach of comparing ML based methods.
In fact, this is also a limitation even in case of testing of
non-ML based predictors as most existing recommendations
(such as P.1401 [13], J.149 [14] or [15]) do not explicitly
recommend the use of effect size.
D. Ignoring Data Uncertainties in Performance Evaluation
In the current approach, the most widely used perfor-
mance measure (ρ) is the correlation coefficient (e.g., Pearson,
Spearman or Kendall correlation coefficients) between the pre-
dicted and target (subjective) quality scores [2]–[16]. In many
cases, the traditional root mean squared error RMSE is also
employed. These measures, however, are less effective because
they do not explicitly consider the uncertainties in the subjec-
tive rating process. Consequently, they penalize any deviation
from the subjective quality score (MOS). This is not desir-
able in the context of multimedia quality estimation because
each MOS represents the mean of individual opinion scores
from a finite set of observers, i.e., sample of observers, and
is therefore a random variable. Thus, a more accurate perfor-
mance measure ρ should consider the said variability in the
individual subjective scores, and at the same time emphasize
differences in quality levels that are practically perceivable.
We define such a measure in Section IV-D.
E. Lack of Focus on Learning Ability of the Model
A notable limitation of the current approach stems from its
focus on prediction accuracies (e.g., maximizing correlation
coefficient or minimizing mean squared error). Consequently,
the aspect of learning ability (i.e., how well can the model
learn structure from the training data and use it to predict
quality on new test samples) is not explicitly considered in
method comparison. We argue that the latter is also important
because it can provide useful information about robustness of
the model, and hence lead towards a more holistic assess-
ment of ML based quality predictors (instead of treating them
as black boxes). Our assertion is motivated by the idea that
data pertaining to multimedia quality can be assumed to be
structured because there is a relation (which is unknown and
possibly non-linear) between the subjective quality scores and
the multimedia content (possibly represented by a set of fea-
tures). Then goal of ML based quality prediction is to exploit
this structure to find an optimal set of weights. Hence, it is
relevant to ask if a given model (in the context of this paper,
the term model refers to the collective entity consisting of
features and an ML algorithm) is able to learn the underly-
ing patterns of perceptual quality, given a set of labeled data.
In other words, it is important to systematically quantify the
 This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
IEEE Transactions on Broadcasting ( Volume: 64, Issue: 2, June 2018 )
NARWARIA: TOWARD BETTER STATISTICAL VALIDATION OF ML-BASED MULTIMEDIA QUALITY ESTIMATORS
weakness (if it exists) of the model in terms of its learning
ability, and analyze its response to unstructured data.
It is also important to differentiate learning ability from pre-
diction accuracy. As mentioned, the former focuses on the
ability to learn real data patterns (assuming that such pattern
or structure is present in the data). On the other hand, the
latter refers to how close the objective predictions are to the
given test data (subjective scores), and therefore does not pro-
vide explicit information about the ability of the model to
distinguish real patterns of perceptual quality from noisy (or
random) ones.
5
X = [X1 , . . . , Xn ] (where n is the sample size) where the
sample observations are assumed to be iid random variables,
the sample mean X ∼ N(μ, σn ) where μ and σ 2 respectively
2
denote the mean and variance of the corresponding population.
Note that this result holds only when the sample observations
are iid. However, the case of dependent2 sample observations,
one needs take into consideration the covariances between the
random variables. This follows from the standard rule that the
variance of sum of n correlated (dependent) random variables
is the sum of their covariances [20], i.e.,
n 
F. Summary of Limitations of Current Approach
As discussed, the current approach suffers from few theoret-
ical and practical drawbacks in the context of comparing ML
based objective quality predictors. Therefore, the inferences
and conclusions drawn from its use are possibly questionable.
More importantly, it does not provide meaningful information
about learning ability of the method under investigation and
lacks practical considerations which are obviously critical for
the purposes of method deployment in real applications. In
light of these limitations, there is need for a better approach
towards more meaningful comparison and validation of ML
based quality predictors. Such approach will be useful in gen-
eral but even more relevant in scenarios such as broadcast
services where ML based quality estimation is expected to be
a more common and plausible approach to quality monitoring.
Thus, we will present in Sections IV and V an alternate set of
guidelines to ameliorate some of the mentioned issues in the
current approach. Before doing that, we analyze the theoret-
ical implications of using dependent (i.e., correlated) sample
observations for statistical testing.
III. E FFECT OF D EPENDENT S AMPLE O BSERVATIONS : A
T HEORETICAL P ERSPECTIVE
Recall from Section II-A that the sample observations
denoted by the values in each column of matrix A are likely
to be dependent. This is because the random train-test split
procedure used in the current approach can lead to sharing
of test conditions across iterations. Therefore, the subsequent
application of statistical tests such as the t-test using this data
can affect the sampling distribution of the test statistic (note
that most parametric statistical tests assume that the sam-
ple observations are independent and identically distributed,
i.e., iid random variables). This in turn can lead to incorrect
decisions since the test statistic may not exactly follow the
expected theoretical distribution. In the following, we analyze
the implications of dependent sample observations and specif-
ically discuss the case of t test as it is widely used in the
current approach to validating ML based quality predictors.
A. Increased Variance of Sampling Distribution of Mean
The genesis of the problem lies in the fact that the t-test (or
for that matter any other parametric test like ANOVA) used
to compare the mean values of each column of matrix A in
a pairwise manner is based on the fundamental central limit
theorem (CLT) [20], [25]. The CLT states that for a sample
1 1 
n n
 
Var X = Var Xi = 2 Cov Xi , Xj
n n
i=1 i=1 j=1
⎛ ⎞
1 ⎝ 
n

= 2 Var(Xi ) + 2 Cov Xi , Xj ⎠ (3)
n
i=1 i=j
where we have used the fact that Cov(Xi , Xj ) = Var(Xi ) when
i = j. To further simplify (3), we proceed as follows. First,
under exchangeability [26] , we can write Cov(Xi , Xj ) = ri,j σ 2
where ri,j is the pairwise correlation coefficient. Moreover,
assume that the variables are pair-wise equicorrelated, i.e., we
let r = ri,j . The reader will also note that there are n2 = n(n−1)
2
covariances in the second term of eq. (3) and that the variance
of each Xi is σ 2 . Hence, we can write (3) as
 σ2 n−1 2
Var X = + rσ (4)
n n
Thus, we find that variance of sample mean increases with
r and the value corresponding to independent sample observa-
tions can be obtained by simply setting r = 0. The increased
variance will result in a change of shape of the distribution of
t-test statistic and it may no longer follow the theoretical t dis-
tribution. Hence, in practice the aim is to keep the value of r
to a minimum. While typical subjective studies achieve this by
employing independent human subjects (observers) or allow-
ing sufficient breaks between test sessions, the aforementioned
random training-testing split does not allow sufficient control
and can lead to larger values of r. Hence, the issue of inde-
pendent sample observations is more applicable in case of ML
based quality predictors, and should be considered in order to
obtain more accurate results from the subsequent use of sta-
tistical tests. The exact impact on the sampling distribution of
the test statistic in t-test is analyzed next.
B. Effect on Sampling Distribution of t-Statistic
As the current approach uses pairwise t-test to compare the
mean values of each column of matrix A, it will be of interest
to examine the impact of the increased variance of sampling
distribution of mean as quantified by eq. (4).
We begin by considering two populations p1 and p2 with
means μ1 and μ2 and same variance σ 2 (homogeneity of vari-
ance). Let the corresponding samples be denoted by x1 =
[x11 , . . . , x1n1 ] and x2 = [x21 , . . . , x2n2 ] where n1 and n2 are
the sample sizes. In our case, the sample observations within
2 Quantification and analysis of data dependence is non-trivial in general.
In our case, we make the simplification assumption of using correlation
coefficient as a measure of dependence.
 This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
IEEE Transactions on Broadcasting ( Volume: 64, Issue: 2, June 2018 )
6 IEEE TRANSACTIONS ON BROADCASTING

each sample denote the performance measure (e.g., correlation (we present further experimental evidence in Section VI-A).
coefficient) from two distinct ML based quality estimators. In other words, when the sample observations are correlated,
Therefore, it is reasonable to assume that the sample observa- the computed t statistic does not follow the theoretical t dis-
tions across the two samples are independent (uncorrelated). tribution, and hence using it for statistical inference can lead
However, as discussed, the observations within each sample to incorrect decisions. Therefore, using a t test to compare the
are now dependent (i.e., correlated), unlike the usual case mean values of each column of matrix A as done in the current
where they are assumed to be independent. Finally, the sample approach may be less accurate because r1 and r2 are wrongly
observations are assumed to be identically distributed. This is set to zero despite the possible dependencies between sample
not unreasonable since P(x1i < c) = P(x2i < c), ∀i and some observations.
fixed constant c, i.e., the probability that observations in sam- In order to correct this, two solutions are possible: (1) a
ples x1 and x2 take a value less than (or equal) to a fixed theoretical solution wherein one needs to find the modified the-
constant is the same. oretical t distribution which is a function of not just the degrees
Let x1 , x2 and s21 , s22 denote the sample means and vari- of freedom but also depends on r1 and r2 , (2) a more practical
ances, respectively. Further, let r1 and r2 denote the pairwise approach where we attempt to alter the experimental design so
correlation coefficient, i.e., dependency between observations that r1 and r2 are qualitatively minimized. The first solution
within each sample. In addition, in the considered application, is more challenging because it requires rigorous mathemat-
the sample sizes will be equal, i.e., n1 = n2 = n because ML ical treatment and will typically lead to application-specific
based quality estimators are tested on same and equal number rather than general solution to the said problem (for instance
of test conditions. Then the goal of the analysis is to infer if refer to [27] for the case of financial data analysis or [28]
μ1 = μ2 (the null hypothesis) or not. To define the t-statistic, which presented asymptotic solutions in case of spatial data
we use the result from the CLT, i.e., where long range dependence is present). The second solution
⎛  ⎞ is more feasible since it involves a more careful experimental
σ 2 n − 1 design and/or data collection (for e.g., refer to [29] where nec-
x1 ∼ N ⎝μ1 , + r1 σ 2 ⎠ (5)
n n essary adjustments are made for more accurate medical data
analysis) which is typically achievable when comparing ML
Notice that we have used eq. (4) to compute the standard based quality estimators. Therefore, in this paper, we propose
deviation of the sampling distribution. Similarly we can obtain, to eliminate or reduce dependency among sample observations
⎛  ⎞
by proper data partitioning as detailed in the next section.
σ 2 n − 1
x2 ∼ N ⎝μ2 , + r2 σ 2 ⎠ (6)
n n IV. P RACTICAL AND S TATISTICAL S IGNIFICANCE IN
C OMPARING ML BASED Q UALITY E STIMATORS :
Then, the difference between the samples means will also be
T HE P ROPOSED A PPROACH
normally distributed, i.e.,
⎛  ⎞ In the previous section, we highlighted few drawbacks of
σ 2 the current approach. With the goal of remedying them, we
x1 − x2 ∼ N ⎝μ1 − μ2 , (2 + (n − 1)(r1 + r2 ))⎠ (7) present a modified set of guidelines in Figure 2. The major
n
steps are discussed below.
By standardization, we have
x1 − x2 − (μ1 − μ2 ) A. Meaningful Partitioning of Data
 ∼ N(0, 1) (8)
σ2 As discussed, the random train-test split used in current
n (2 + (n − 1)(r1 + r2 ))
approach ensures that training and testing sets do not over-
Further, assume that the null hypothesis H0 is true, i.e., lap (or share content). This, is of course desirable. However,
μ1 = μ2 and use the pooled variance s2p as an unbiased this process does not necessarily constrain the different test
estimator of the population variance σ 2 . Thus, under H0 , the sets (generated in each iteration) to be completely indepen-
denominator in eq. (8) can be modified accordingly and the dent of each other. As a result, the independence of elements
t-statistic defined as (observations) in each column of A is questionable, and vio-
x1 − x2 lates one of the basic requirements for further statistical testing
t=  (9) (j)
of such data. Hence, in the first step, we restrict ρi values
sp 1n (2 + (n − 1)(r1 + r2 )) to be obtained on disjoint test sets (with each set represent-
where ing a particular condition to be examined in the context of
perceptual quality). For instance, in image or video quality
s21 + s22 prediction task, the disjoint test sets can be obtained based
s2p = (10)
2 on the number of reference content used in the dataset. In
We note that the statistic t defined in eq. (9) will follow the this manner, each test will consist of all images from one
theoretical t distribution with n1 + n2 − 2 = 2(n − 1) degrees source/reference content. Likewise, in broadcast applications,
of freedom only when r1 = r2 = 0, i.e., when observations the type of distortions induced by different codecs (at varying
within each sample are independent. Otherwise, it will devi- bit rates and/or packet loss rate etc.) or the type of content
ate from the theoretical t distribution as r1 and r2 increase (e.g., based on color and temporal information) can be used
 This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
IEEE Transactions on Broadcasting ( Volume: 64, Issue: 2, June 2018 )
NARWARIA: TOWARD BETTER STATISTICAL VALIDATION OF ML-BASED MULTIMEDIA QUALITY ESTIMATORS
Fig. 2. Schematic flow diagram of the steps involved in the proposed approach for comparing methods using single dataset (i.e., splitting the same dataset
for generating training and test sets).
to obtain physically meaningful set of conditions for which
quality prediction is desired. In other words, we advocate a k
fold cross validation (CV) wherein each fold comprises of data
that is typically from independent (disjoint) sets that can be
conveniently defined according to the conditions to be tested
and the eventual goal of analysis (e.g., testing a certain com-
bination of conditions in joint audiovisual compression). By
using the k fold CV approach meaningfully, we can modify
matrix A as
⎡ ⎤
(1) (M)
ρ1 ··· ρ1
⎢ .. .. .. ⎥
Acondition = ⎢
⎣ . . .
⎥
⎦ (11)
(1) (M)
ρNcondition · · · ρNcondition
Note that the dimensionality of each column of Acondition
will not be arbitrary but equal to the number of meaningful or
desired independent conditions/categories in the dataset, unlike
that of A. Therefore, unlike the current approach, we seek to
emphasize more the qualitative aspects of data partitioning
(train-test split), and reduce dependency arbitrary number of
randomizations.
B. Practical Considerations in Method Comparison
Practical significance is not always the same as statisti-
cal significance [21]–[24]. As mentioned in Section II-C, this
aspect has been largely ignored in the current approach. Recall
that statistical tests (like the t test in the current approach) only
provide information if a treatment effect is present or not but
do not explicitly answer the question whether the amplitude of
the treatment effect (recall that in our case this can be differ-
ence in mean correlation coefficients or difference in median
7
AUC values etc.) is practically meaningful or not. Thus, in
the proposed set of guidelines, we seek to use effect size to
quantify practical significance. It can be computed by com-
paring each column pair (i, j) of Acondition . A commonly used
measure of effect size is Cohen’s d which is computed as
ρ(i) − ρ(j)
d(i, j) =  (12)
(sρ (i))2 +(sρ (j))2
2
where ρ(i), ρ(j) and sρ (i), sρ (j) denote the means and standard
deviations of i-th and j-th column, respectively.
The effect sizes d(i, j) for each comparison are stored in the
matrix E. Effect size value higher than a threshold th can be
deemed practically significant (typically, th ≥ 0.8 is consid-
ered large effect). It is important to emphasize that we do not
use effect size as an explicit statistical test. Rather it is used
to assess how large the observed effect is, and thus provides
a context about the possible practical differences between the
methods being compared. In order to establish if the treat-
ment effect is actually present or not, we will use either use
ANOVA or nonparametric confidence intervals (as explained
in Section IV-C).
Lastly, since we advocate data partitioning according to
physically meaningful conditions (instead of random splits),
recall that the dimensionality of each column of Acondition will
be Ncondition (the number of meaningfully disjoint conditions in
the dataset, and will depend on the dataset as well as context
(j)
of analysis). Thus, each element ρi represents the perfor-
mance on a practically more meaningful chunk of data which
in turn provides useful initial information about the possible
strength (or weakness) of ML based quality predictors for that
 This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
8 IEEE(TRANSACTIONS
IEEE Transactions on Broadcasting
(.)
particular condition. Therefore, a comparison of ρi values
across rows of matrix Acondition can reveal meaningful local
information about method performance (instead of making
global observations, as is the case with the current approach).
Such information is also obviously useful from view point of
method development and calibration.
C. Testing the Presence of Treatment Effect
Treatment effect, in our context, is the relative overall (for
instance the average change) change in performance measure
(j)
ρi as M different ML based predictors are tested. We say that
the treatment effect is present if there is a systematic change in
(j)
ρi ) over a set of test conditions. Otherwise, we conclude that
the treatment effect is absent, i.e., the observed differences in
(j)
ρi for different predictors are merely attributed to chance.
To that end, a statistical test can be employed and a decision
taken based on the chosen significance level α [20].
Recall that in the current approach, t test is applied to
compare the mean values (or the Wilcoxon test for median)
of each column of matrix A in a pairwise manner [2]–[12].
This means there are multiple statistical comparisons which
may suffer from drawbacks related to inflated type I error
probability. While p-value or significance level adjustments
have been proposed in literature to control the family-wise
error error [18], these adjustment procedures are debatable due
to practical reasons (for instance refer to [30]–[32]). Perhaps
the more important concern due the use multiple t-tests is the
fact that each comparison simply ignores information from
other groups/samples. Hence, the comparison is more local-
ized and this is not in line with main goal of analysis which
is to jointly compare a set of M ML based quality estimators.
Therefore, in the proposed guidelines, we advocate the use of
one-way ANOVA [20] on the data matrix Acondition to test the
null hypothesis that means of populations represented by the
samples (groups) in columns of Acondition are equal or not.
1) ANOVA Based Statistical Significance Testing: If
ANOVA rejects the null hypothesis, then it implies that a
treatment effect is detected. It means that at least two out of
the M ML based predictors exhibit systematically large dif-
(j)
ferences in terms of the mean ρi values. In order to find
such pairs, one can use posthoc tests [20]. However, most
posthoc tests mainly rely on adjusted significance (or p-values)
to compensate for the inflated type I error rate. This, however,
can have undesirable consequences including inflation of type
II error [30]–[32]. Moreover, such posthoc tests (or for that
matter ANOVA) do not reveal information about the magni-
tude of the observed treatment effect. Thus, in the proposed
guidelines we use the information in matrix E to identify such
statistically different pairs (i.e., we conclude that there is sta-
tistical and practical difference between methods i and j if
E(i, j) > th). Such approach allows to meaningfully consider
practical differences and avoids multiple comparison adjusted
post hoc procedures. On the other hand, if the null hypothesis
is not rejected by ANOVA, we conclude that there is no treat-
(j)
ment effect and the observed differences in mean values ρi
are attributed to random sampling errors (i.e., method pairs
are statistically similar). In this case, effect size matrix E can
still be computed and reported especially if the p value for
ON BROADCASTING
Volume: 64, Issue: 2, June 2018 )
ANOVA is close to the chosen significance level α, in order
to have better interpretation of the results [21]–[24].
As a final note on the use of ANOVA, we point out that
the often cited assumption of normality required in ANOVA
(and other parametric tests such as the t-test or confidence
interval for mean) is not on the data but only on the sampling
distribution of mean (which is guaranteed by the central limit
theorem) [25]. Moreover, these tests are accurate even in case
of violation of homogeneity of variance provided that the sam-
ple (group) size is the same [25]. This is true in the considered
application because ML based predictors will be obviously
tested on the same and equal number of tests conditions.
Thus, in the proposed guidelines, we strongly recommend the
use of ANOVA as an omnibus test. However, in occasional
(j)
cases such as extremely skewed ρi values, where mean can-
not be used, we recommend the use of nonparametric CIs,
which are described in the next subsection (Section IV-C2).
We however draw caution over the use of nonparametric tests
(including the ones based on ranks such as Mann-Whitney
U test or the Kruskal-Wallis’ test) as these generally have
lower statistical power [33] and higher sensitivity to homo-
geneity of variance condition [34] as compared to parametric
ones.
2) Construction of Nonparametric Confidence Intervals:
Nonparametric CIs are popular and have been used in many
applications [35]–[38] and are obtained empirically via boot-
srapping [35]–[37]. In our case, we construct them for the
(j)
chosen summary statistic for ρi values. We begin by treating
(j)
ρi values in matrix Acondition as independent sample observa-
tions. In other words, each column of Acondition can be assumed
to be a random sample drawn from the corresponding pop-
ulation. Let the true population parameter of interest be θ .
Then, the goal is to make inferences about θ by using θ̂ (the
estimated parameter value from sample). However, practically
(j)
since ρi is computed on a limited number of stimuli, we can
obtain an interval estimate of θ̂ (instead of a point estimate)
by constructing its CI via resampling [35]–[37]. Such interval
estimate provides more descriptive information about θ . Note
that we will obtain the said CI for each column of Acondition ,
and therefore drop the index j in the sequel for convenience
in notation. The procedure to obtain nonparametric CIs is as
follows:  
• Obtain θ̂ from the given data ρ1 , ρ2 , . . . , ρNcondition .
• Obtain L repeated bootstrapped samples
 ∗ ∗ 
ρ1 , ρ2 , . . . , ρN∗ condition , and the ordered vector
∗  
θ̂ = θ̂1∗ , θ̂2∗ , . . . , θ̂L∗ such that θ̂1∗ < θ̂2∗ · · · < θ̂L∗ .
• Using pivots to construct the desired CI, we com-
pute the lower and upper bound for the CI as
 ∗

∗
2θ̂ − θ̂ (1−α /2) 2θ̂ − θ̂ (α /2) where α is the desired
∗
significance level, and θ̂ (h)
is the hth percentile
value [35]–[37].
As indicated in Figure 2, the methods for which the non-
parametric CIs do not overlap and E(i, j) > th can be deemed
to be meaningfully different (practically and statistically).
However, if the CIs overlap we conclude that the methods
are statistically similar (E(i, j) values can be computed in this
case if the overlap is minimal).
 This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
IEEE Transactions on Broadcasting ( Volume: 64, Issue: 2, June 2018 )
NARWARIA: TOWARD BETTER STATISTICAL VALIDATION OF ML-BASED MULTIMEDIA QUALITY ESTIMATORS
D. Performance Evaluation by Considering Data
Uncertainty and Effect Size
As mentioned in Section II-D, the use of correlation coef-
ficient or RMSE as the performance measure ρ does not take
into account the inherent subjective uncertainty (variabilities).
Therefore, in the proposed guidelines, we recommend the use
of AUC (area under curve) as ρ. The definition of AUC is
motivated by the fact that a classification error is made when
the subjective score and the objective score (in our case, from
an ML based predictor) lead to different conclusions on a pair
of data points [14]. The idea was further extended to include
Receiver Operating Characteristics (ROC) [39] based compar-
ison [40]–[42], and states that a better performance measure
should be able to distinguish (classify) different quality lev-
els by considering the dispersion (uncertainty) in the opinion
scores. This can be treated as a binary classification prob-
lem and analyzed based on ROC analysis [40]–[42]. The Area
Under Curve (AUC) is then to evaluate discrimination abili-
ties [39]. However, in most of the previous works [40]–[42],
statistical significance has been used as the criterion to distin-
guish between quality levels of a pair of stimuli. That is, if a t
test between the individual scores of two stimuli results in the
rejection of null hypothesis, then the corresponding pairs are
said to be different. Otherwise, they are deemed as same. The
AUC values from the ROC analysis performed only on statisti-
cally different pairs are then used to compare objective quality
predictors. While the described AUC based performance mea-
sure takes into account subjective uncertainties, it suffers from
two limitations.
First, the use of statistical significance (as done in [40]–[42])
to classify pairs as same or different is problematic because
it may not always be the same as practical significance. As
noted in Sections II-C and IV-B, practically insignificant
differences may sometimes be statistically significant (and
vice-versa) [21]–[24]). Thus, existing works [40]–[42] tend to
rely exclusively on p values which provides the information
if an effect is present or not. However, it does not answer
the question whether the treatment effect is large enough to
be practically relevant or not. Second, the use of multiple
comparisons (t tests in this case) is in many cases accompa-
nied by adjustments to significance levels (or the p-values) in
order to control the family-wise error error. Such adjustments
are, however, debatable due to practical reasons (for instance
refer to [30]–[32]).
Therefore, we propose to rely less on a purely p-value based
statistical decision, and rather focus on practically meaningful
differences via the use of effect size. It also helps to avoid
multiple comparisons as we do not explicitly make a statistical
decision using the effect size. We proceed as follows.
1) Compute the effect size between stimuli pair (i, j) based
on Cohen’s d,
MOSi − MOSj
d(i, j) = 
s2i (n1 −1)+s2j (n2 −1)
(n1 −1)+(n2 −1)
where MOSi , MOSj and s2i , s2j denote the sample means
(mean opinion score) and variances respectively, and
9
n1 and n2 are the sample sizes (i.e., the number of
human participants who rated the multimedia qual-
ity). As before, if d(i, j) > th, then the corresponding
stimuli pair can be deemed to exhibit practically mean-
ingful differences in terms of perceptual quality. As
mentioned, this is the major difference from previous
works [40]–[42]. As a result, each stimuli pair (i, j) will
be assigned either a label 1 if d(i, j) > th and zero
otherwise. This forms the ground truth label vector.
2) Next, we define the difference in predicted scores from
the ML based quality predictor (MLQP) for stimuli pair
(i, j) as MLQP (i, j) = MLQP(i) − MLQP(j). Then,
for each stimuli pair (i, j) a label of 1 is assigned if
MLQP (i, j) > throc and 0 otherwise. Note that throc is
different from th because the latter is used to fix a thresh-
old based on which we conclude whether the stimuli
pair (i, j) has practically different quality level. Thus,
the value for th is chosen once and fixed for the experi-
ment, and is essentially employed to obtain ground truth
label vector as explained in the previous point. By con-
trast, throc is allowed to vary so that we obtain a 2D plot
of false positive rate against the true positive rate [39].
3) Then, the AUC value is obtained by computing the area
under the curve in the said 2D plot. It will be higher
if the ML based estimator is able to correctly recognize
the stimulus of higher quality in the pair (only for pairs
which have practically different quality levels).
V. A SSESSING L EARNING A BILITY OF ML
BASED Q UALITY P REDICTOR :
A P ERMUTATION T EST
(j)
While performance measures ρi can be used to quan-
tify and compare prediction accuracies, these do not always
provide explicit information about the response of an ML
based quality predictor to unstructured data. As discussed in
Section II-E, this is important because ML based methods
are typically treated as black boxes. Hence, the ability of a
model to distinguish real data patterns from random ones can
potentially provide insights into the learning process [19].
Ojala and Garriga [19] presented two such tests in the con-
text of classification problem. The first test is designed to
assess whether the classifier has found a real class structure in
the data (note that this test has already been used traditionally
in computational biology [43] and attribute selection in deci-
sion trees [44]). The second test studies whether the classifier
is exploiting the dependency between the features in classifica-
tion. In our context, we are more interested in the second test
primarily because the goal is to assess if the user-defined fea-
tures share meaningful dependencies to capture the complex
mapping between content and its quality. Moreover, it is also
of our interest to study if the chosen of ML algorithm (regres-
sor) is able to exploit the said feature dependency (if it exists)
(13)
to learn a more accurate and generalizable mapping function f .
In the following, we describe a permutation test which
is based on the second test in [19] but with two impor-
tant modifications to test ML based quality predictors. First,
unlike [19] which dealt with classification problem, we deal
 This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
IEEE Transactions on Broadcasting ( Volume: 64, Issue: 2, June 2018 )
10 IEEE TRANSACTIONS ON BROADCASTING

with regression since in most cases ML based predictors are In multimedia quality assessment, the elements of X typically
designed and trained on continuous valued opinion scores. represent the value of an attribute (feature) that is related to
This is achieved via data dependent binning process (as perceptual quality. Further, a continuous value yi ∈ R repre-
explained in Section V-B). Second, we define a modified senting the MOS (average of ratings from a some specified
criterion for error metric so that only practically (or statisti- number of human subjects) is associated with each data point
cally) meaningful differences are penalized (and not simply the Xi . Then the labeled data set will be D = {(Xi , yi )}ni=1 .
changes in point-based estimates as is the case with correlation In order to obtain randomized versions of D, feature values
coefficient or RMSE). contained in X need to be permuted. However, such permuta-
tion is restricted to feature values within each class label [19].
A. Motivation Since the target values in the considered case are continu-
ous, we first employ a strategy to discretize them, and in the
As already mentioned, testing the response of an ML based
process bin them into B bins. In order to preserve the distri-
quality predictor to unstructured data has been largely ignored
bution, we obtained the boundary points of the histogram of
in [2]–[12] on ML based multimedia quality assessment. We
the continuous values. Next, thresholding was applied so that
argue that such test is useful as it provides insights into the
the continuous values between the specific boundary points
learning ability of a given model, and helps to assess the com-
are all assigned the same label. For instance, values smaller
petence of the model from the view point of learning (training).
than the first boundary point were all assigned to class label 1,
To develop such a test, we can borrow the idea of a permu-
the values between first and second boundary points assigned
tation test [19], [20], and assess systematically the effect of
to class label 2 and so on. The main advantage of the said
feature changes on the prediction performance. We assume
binning process is that it preserves the original data distri-
that the given ML based quality predictor was trained on
bution by considering data dependent thresholds (instead of
a structured data of perceptual quality, and that it has been
using pre-defined threshold values). It is also worth high-
able to exploit the underlying structure (i.e., inter dependency
lighting that binning of continuous MOS values into discrete
between feature values and their relation to MOS) for learning
classes (bins) is practically meaningful since the class labels
the mapping function f . Hence, when we test this predictor
can assume a similar meaning as those in standard subjec-
on untrained but structured data D, the corresponding error
tive rating methodologies. For instance, the absolute category
e(f , D) is expected to be lower than the error e(f , D
) on
rating ACR which is a popular and standard rating method
unstructured (randomized) data D
. This is because the inter-
employs discrete labels: 1 (worse), 2 (poor), 3 (fair), 4 (good)
dependency between features has been disrupted in D
due to
and 5 (excellent). Thus, the binning of continuous MOS offers
random feature permutation. Accordingly, we can define the
a physically valid interpretation of the resultant class labels.
null hypothesis as H0 : the given ML based quality predictor
As a result of binning, yi → c where c = [c1 , c2 , . . . cB ] rep-
has not learnt the dependency between feature values for pre-
resents the discrete class label vector. Note that such binning
dicting the quality score. The empirical p value in this case
is used only for feature permutations, and subsequent error
can be simply defined as the ratio of number of times the error
metric computations will be based on the continuous yi values.
on random data is less than that on structured data to the total
number of randomizations. C. Error Metric for Permutation Test
A high p value points out to three possibilities [19]: (a)
The goal of ML based quality predictor is to predict the
there are no dependencies between the features that are used
quality score (continuous) of new data points (i.e., new test
in the ML based predictor; (b) there are some dependencies
signal) by training a regressor from D. The learned function
between the features in the data but they do not help to
f maps the feature values into an objective quality score, i.e.,
obtain a more accurate mapping function f ; or (c) there
f : X → R.
are useful dependencies between the features in the data
To compute the final error for the permutation test, we
that help to find a better f but the chosen ML algorithm
begin with the observation that any performance measure
(regressor) is not able to exploit them. In our context, if an
ρ (including correlation coefficient, MSE, area under the
ML based quality predictors obtains a high p value in the
curve AUC [14], etc.) is always computed on a finite sam-
said permutation test, then it implies that either the feature set
ple. Specifically, in each iteration, the test set has Ntest stimuli
is sub-optimal (possibility (a) or (b)) or the chosen regressor
for which quality is predicted and ρ computed against the
is not competent enough, i.e., possibility (c). In either case,
corresponding subjective ground truth. Thus, ρ is a random
a high p value reveals potential deficiency in the given ML
variable and will be associated with a CI. The said CI can
based quality predictor. Remedial action might include using
be computed via analytic methods. However, such methods
additional context specific features or modifying the existing
depend on few assumptions. For instance, some require a fixed
features. In addition, a different regressor that is better at
distribution of positive (i.e., class label 1) and negative (i.e.,
exploiting feature interdependency may be employed.
class label 0) scores (recall that in our context, we assign a
label 1 if d(i, j) > th and 0 otherwise) [45], [46], while oth-
B. Data Dependent Binning of Continuous MOS ers assume a fixed classifier error rate [47]. Such assumptions
In order to tackle the problem of regression, we proceed as may not be satisfied in the considered application. Moreover,
follows. Let X be an n × m data matrix (the i-th row and j-th CLT based approximations are applicable only in case of sum
column vectors can be denoted as Xi and Xj , respectively). or mean iid samples (this is also true for computing CI around
 This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
IEEE Transactions on Broadcasting ( Volume: 64, Issue: 2, June 2018 )
NARWARIA: TOWARD BETTER STATISTICAL VALIDATION OF ML-BASED MULTIMEDIA QUALITY ESTIMATORS
correlation coefficient after Fisher’s transformation [20]), and
not to other performance measures such as the AUC or RMSE∗
values. Therefore, we propose the use of nonparametric CIs
of ρ. The nonparametric CI on data D can be denoted as
Iρ (D) = [Lρ (D) Uρ (D)] where the subscript ρ denotes the
chosen performance measure (or error metric). An error (or
failure) is said to have occurred if Uρ (D
) ≥ Lρ (D), i.e., if
the confidence intervals on structured data D and unstructured
(randomized) data D
overlap or the lower bound of the former
is less than (or equal to) the upper bound of the latter. For
reasons outlined in Sections II-D and IV-D, we recommend
the use of AUC (defined in Section IV-D) as the performance
measure (error metric) for the permutation test.
D. Permutation Test Description
To apply the permutation test, we follow similar notation
as [19], and proceed as follows.
• For the original data D = {(Xi , yi )}ni=1 , bin the continuous
MOS, i.e., yi → c to obtain D = {(Xi , cj )}.
• Let X(c) be the submatrix of X in class label c, i.e.,
X(c) = {Xi |yi = c}, of size lc × m.
• Let π1 , . . . , πk (k = 1, . . . , m) be k independent permu-
tations of lc elements.
• Let X(c)
be a randomized version of X(c) where each
πj is applied independently to the column of X(c)j . Thus,
X(c)
= [π1 ( X(c)1 ), . . . , πk (X(c)k )].
• Let X
= {X(c)
|c = c1 , c2 , . . . cB }. Then, one random-
ized version D
= {(X
i , yi )}ni=1 can be obtained. increases. This follows and supports the theoretical analysis
• The empirical p value for the permutation test (denoted as
p1 to distinguish it from the one obtained from ANOVA)
can be computed as

 
D ∈ D : Uρ D
≥ Lρ (D) + 1 tribution, and hence using it for statistical inference can lead
p1 = (14)
Niter .Ncondition + 1
where D denotes the set of Niter randomized versions D
posed guidelines take a step towards reducing or eliminating
of the original data D.
VI. E XPERIMENTAL R ESULTS AND D ISCUSSION
In this section, we present experimental results to analyze
and compare few existing ML based methods by using both
the current (the steps are shown in Figure 1) and the proposed
set of guidelines (the steps involved are shown in Figure 2 and
the details have been presented in Sections IV and V). Before
that, we present experimental evidence to support the theoret-
ical analysis made in Section III and examine the effect of
dependent (correlated) samples observations on the sampling
distribution of t.
A. Correlated Sample Observations: Effect on Empirical
Sampling Distribution of t-Statistic
To examine the impact of correlated sample observations,
we generated the sampling distribution of the test statistic t
defined in eq. (9). As discussed in Section III-B, t will follow
the theoretical t distribution with n1 + n2 − 2 = 2(n − 1)
degrees of freedom only when r1 = r2 = 0, i.e., when
observations within each sample are independent. To test this,
we first employed a Gaussian AR(1) process in order to gen-
erate the population with correlated elements. The process in
11
its simplest form can be expressed as
xt = μ + r(xt−1 − μ) + t (15)
where t ∼ N(0, σ2 ) is zero mean error term, r is the corre-
lation coefficient and μ is the population mean. In order to
obtain the sampling distribution of t, we generated two popu-
lations (having the same mean and variance) using the above
Gaussian AR(1) process (we set μ1 = μ2 = 0.5, σ 2 = 1).
Then, each population was repeatedly sampled to obtain x1 and
x2 (sample size n was set to 1000 which is equal to the typical
number of iterations Niter used in the current approach) and the
statistic t computed in each iteration. Note that the said pro-
cedure assumes the null hypothesis H0 to be true and ensures
homogeneity of variance. In order to compare and see the
effect of dependent observations, we obtained the said empir-
ical distribution of t in three cases, each time taking different
values of r1 and r2 .
The resulting sampling distributions for the three cases are
shown in Figure 3. The first plot corresponds to r1 = r2 = 0,
i.e., independent sample observations and this is the most typ-
ical use case of the t-test. As can be noted the empirical
sampling distribution of the t statistic follows reasonably well
the theoretical t distribution with the corresponding degree of
freedom. In contrast, when r1 and r2 are not zero, one can
clearly see that the empirical sampling distribution does not
follow the theoretical curve. Particularly, we notice that as r1
and r2 increase, the said deviation from the theoretical curve
made in Section III-B.
In summary, when the sample observations are correlated,
the computed t statistic does not follow the theoretical t dis-
to incorrect decisions. This is one of the major drawbacks
with the current approach. The reader will notice that the pro-
the said dependencies and hence allow more accurate statistical
analysis (using ANOVA or nonparametric CIs).
B. Test Dataset and Methods Compared
For the experiments, we chose the specific domain of
image quality assessment primarily due to easy availabil-
ity of datasets (along with individual subjective scores) and
implementations of few ML based objective quality predictors.
We use the CSIQ [48] database as a testbed for the
experiments. There are no particular reasons for choosing this
dataset apart from the fact that it provides subjective quality
scores (in the form of difference MOS, DMOS) and standard
deviations of raw scores. The images in the dataset were
generated from 30 reference (original) images which were
distorted by 6 different distortion types at 4 to 5 distortion
levels. This resulted in 866 distorted images. Similarly, we
chose 6 existing and representative ML based methods for
comparison keeping in mind the diversity of features used.
These include the methods which use features based on statis-
tics of natural image [49], gradient magnitude and Laplace
transform [8], singular value decomposition [9] (SVD),
gradient-weighted histogram of local binary pattern [50],
natural scene statistics of contrast-distorted images [51] and
 This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
IEEE Transactions on Broadcasting ( Volume: 64, Issue: 2, June 2018 )IEEE
12 Transactions on Broadcasting ( Volume: 64, Issue: 2, ON
IEEE TRANSACTIONS June 2018 )
BROADCASTING

Fig. 3. Sampling distribution of t statistic (computed via eq. (9)) for different r1 and r2 values. In each plot, the continuous curve indicates the theoretical
t-distribution with 2(n − 1) = 1998 degrees of freedom.

(j)
phase of Fourier transform [52]. We respectively denote these the test set in different iterations. In effect, the ρi values
methods as BRISQUE, GMBIQA, SVDIQA, GLBPIQA, on which pairwise t test is applied are not necessarily
CDIQA and FTSIQ. independent. Then, in light of the analysis in Section III-B
and the experimental results in Figure 3, the reliability of
C. Application of the Current Approach and Analysis the t test result is questionable because the computed t
of the Drawbacks statistic may not follow the theoretical t distribution.
• While mean AUC values for each method allows a global
First, we follow the current approach of comparing the 6
ML based methods. To that end, we employ 80-20 train-test comparison (for instance AUC value for FTSIQ is clearly
split, i.e., 80% data for training and remaining 20% for better than that of CDIQA), it does not reveal method
testing. Since there are 866 images in our testbed, this leads deficiencies for specific conditions. For instance, ques-
to Ntrain = 693 and Ntest = 173, and we ensure that the tions such as which ML based predictor performs the
training and testing sets do not share content (i.e., no overlap). best for a particular distortion type or for a class of source
(j) images, remain unanswered.
We choose AUC as the performance measure ρi where i is
• The effect of Niter values is evident by comparing D1000
the iteration index i = 1, . . . , Niter and j = 1, . . . , 6. Hence,
the matrix A will be Niter × 6 (columns 1 to 6 respectively and D30 . In particular, notice that all entires of D1000 are
correspond to GLBPIQA, SVDIQA, GMBIQA, CDIQA, 1 implying that all the methods are statistically differ-
FTSIQ and BRISQUE). The mean AUC values over Niter ent from each other. However, the conclusions from D30
iterations for the 6 methods were: GLBPIQA = 0.8201, obviously do not agree with those from D1000 . Thus, sta-
SVDIQA = 0.9403, GMBIQA = 0.9143, CDIQA = 0.5439, tistical conclusions (inferences) may simply depend on
FTSIQ = 0.9770 and BRISQUE = 0.8308. Next, statistical the arbitrary choice of Niter .
• There is no explicit information about the learning ability
comparison is carried out by applying t test pairwise on
columns of A to obtain the symmetric decision matrix D. We of any of the methods.
obtained the said decision matrix in two cases: Niter = 1000
and Niter = 30, and respectively denoted as D1000 and D30 . D. Test Results Using Proposed Guidelines
⎡ ⎤ As explained in Section IV-A, the first step in the pro-
− 1 1 1 1 1
⎢1 − 1 1 1 1⎥ posed guidelines is meaningful data partitioning. To that end,
⎢ ⎥
⎢1 1 − 1 1 1 ⎥ one reasonable strategy is to partition according to reference
⎢
D1000 = ⎢ ⎥ (16)
1 1 1 − 1 1 ⎥ images. Hence, we divide the data such that each test set con-
⎢ ⎥ sists of distorted images from one reference image while the
⎣1 1 1 1 − 1⎦
1 1 1 1 1 − distorted versions of the remaining reference images formed
⎡ ⎤ that training set. This will lead to a total of 30 test sets (one
− 1 1 1 1 0 for each reference image). The said partitioning helps to assess
⎢1 − 0 1 1 1⎥
⎢ ⎥ the performance according to the type of source (reference)
⎢1 0 − 1 1 1⎥ (j)
⎢
D30 = ⎢ ⎥ (17) content. The AUC is chosen as the performance measure ρi .
⎢1 1 1 − 1 1⎥ ⎥ Therefore, the matrix Acondition in this case will be 30 × 6.
⎣1 1 1 1 − 1⎦ The entries of matrix Acondition are shown in Figure 4, for
0 1 1 1 1 − sake of better visualization. As noted in Section II-C, this
In line with the analysis in Section II, we can summarize provides meaningful initial information about method per-
the drawbacks of the current approach with regards to the formances. We have indicated in Figure 4 the boundary at
experimental results on CSIQ dataset. AUC = 0.7. As can be seen, all the methods do not have
• As random train-test split is used (of course ensuring that consistent AUC values across conditions, and fall below 0.6 in
the training and testing sets do not share content), it does certain cases. Because AUC value equal to 0.5 corresponds to
not guarantee that same content will not appear again in random guess, these highlight the limitations or inability of the
 This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
IEEE Transactions on Broadcasting ( Volume: 64, Issue: 2, June 2018 )
NARWARIA: TOWARD BETTER STATISTICAL VALIDATION OF ML-BASED MULTIMEDIA QUALITY ESTIMATORS 13

(j)
Fig. 5. Histograms of ρi values for the methods GLBPIQA, SVDIQA,
Fig. 4. Initial performance comparison based on AUC values according to GMBIQA (first row) and CDIQA, FTSIQ, BRISQUE (second row). Figure
source images in CSIQ dataset. Figure best viewed in color. best viewed in color.

corresponding ML based method for that condition (content).

Such information is useful not just for method comparison
but also during method development as it points to specific
deficiencies which for instance could be mitigated by use of
additional features and/or ML method.
We can also see from Figure 4 that FTSIQ achives AUC
values ∼ 0.5 for conditions 7 and 20 (i.e., source images 7 and
20 in this case). Hence, it may not be suitable in discriminating
perceptual quality levels in such content. However, the current
approach may not reveal such information. Indeed, as analyzed
in the previous sub-section, the decision matrix D1000 (or D30 )
simply leads to the conclusion that FTSIQ is statistically the
Fig. 6. Nonparametric confidence intervals (significance level α = 0.05)
best performer out of the 6 methods (this has its limitations (j)
for median of ρi values for the methods GLBPIQA, SVDIQA, GMBIQA,
as also highlighted in Section IV-C). CDIQA, FTSIQ and BRISQUE (left to right). Figure best viewed in color.
In order to get further insights into practical significance, the
effect size matrix E (based on Cohen’s d) is given below. We
find that CDIQA exhibits large (negative) values implying that ANOVA is recommended as explained in Section II-C and
its performance is much lower than other methods. This is also highlighted in Figure 2). To that end, we can visualize the
(j)
reflected in Figure 4 where this method has lower AUC val- histograms of ρi (shown in Figure 5) for the different ML
ues (< 0.6) for several conditions. On the other hand, FTSIQ based quality predictors. As can be seen, these are somewhat
exhibits relatively large positive effect size in comparison to skewed to the right (except for the method CDIQA). Thus,
(j)
other methods and this is in line with the observations from to demonstrate we choose median of ρi values for each
Figure 4 where it achieves AUC values > 0.7 on majority of method as a summary statistic and obtain the nonparametric
test conditions.3 It may be emphasized that we are not making CIs. These are shown in Figure 6 (we used significance level
statistical inferences from Figure 4 or from matrix E. Rather of α = 0.05 and L = 1000).
the goal is to gain insights into local performance comparison With regards to the application of the proposed permuta-
of different methods, and assess if one method might be prac- tion test, the p1 values (we used B = 5, α = 0.05 and
tically better than other for a given condition. Additionally, th = 0.8) obtained were as follows: GLBPIQA = 0.43,
such analysis can help during method development. SVDIQA = 0.31, GMBIQA = 0.35, CDIQA = 0.78, FTSIQ
= 0.15 and BRISQUE = 0.51. We note that the resultant p1 are
⎡ ⎤ higher than 0.05 (the chosen significance level) for the 6 ML
− −1.10 −0.84 3.34 −2.75 0.48
⎢ 1.10 − −2.08 1.91 ⎥
based quality predictors. As discussed in Section V-A, the high
⎢ 0.3700 5.65 ⎥
⎢ ⎥ p1 values indicate that either there is no meaningful interde-
⎢ 0.84 −0.37 − 5.46 −2.58 1.62 ⎥
E=⎢ ⎥ pendency between feature values that can improve prediction
⎢−3.34 −5.65 −5.46 − −8.33 −3.32⎥
⎢ ⎥ or the employed regressor (i.e., ML algorithm) is unable to
⎣ 2.75 2.08 2.58 8.33 − 3.99 ⎦
exploit feature interdependency (if it exists) or a combination
−0.48 −1.91 −1.62 3.32 −3.99 −
of these two. In any case, the test reveals deficiencies which
(18) are not revealed with the current approach.
Next, in order to draw statistical comparisons, we can Overall, we find from Figure 6 that the nonparametric CIs
either use ANOVA or nonparametric CIs, depending on overlap for 4 of the 6 methods. Therefore, according to the
whether mean can be used as a descriptive statistic (if yes, proposed guidelines in Figure 2, we can conclude that these
4 methods (GLBPIQA, SVDIQA, GMBIQA and BRISQUE)
3 For the purposes of this paper, we are not overtly interested in analyzing
are statistically at par in terms of quality prediction accord-
or comparing specific methods. Indeed FTSIQ is a reduced-reference method
while others like CDIQA are no-reference. Thus, the performance differences ing to source content (i.e., image type), and the observed
(j)
are not entirely unexpected. differences in their mean ρi are attributed to chance (or
 This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
14
IEEE Transactions on Broadcasting ( Volume: 64, Issue: 2, June 2018 )
sampling error). In this case, though some of the effect sizes
(refer to rows 1, 2,3 and 6 of the matrix E in eq. (18)) which
correspond to these 4 methods) are large (say > 0.9), such
treatment effects can be merely attributed to chance (as the
CIs overlap). On the other hand, for CDIQA we find that
the pairwise effect sizes are quite high in magnitude and its
nonparametric CI does not overlap. This implies that other
methods are statistically and practically better than CDIQA.
Likewise, we can conclude that FTSIQ is statistically and
practically better than the remaining methods.
Thus, in the proposed guidelines, our conclusions about
relative performances are well supplemented by additional
information from Figures 4 and 5, effect size matrix E,
Figure 6 (or ANOVA if mean is used as summary statistic)
and p1 values from the permutation test. This allows to
compare and test methods on different aspects leading to a
more grounded inference making process.
VII. F INAL R EMARKS
With the growing demands for more immersive quality of
experience from consumers, quality monitoring in multimedia
content delivery especially via broadcast services assumes a
significant role in todays scenario. To that end, ML based qual-
ity predictors offer a plausible solution. Moreover, promising
results from related disciplines such as computer vision and the
availability of required hardware (e.g., GPU-accelerated com-
puting) have opened up possibilities of developing efficient
ML based implementations of quality predictors. However,
proper validation and benchmarking of such ML based quality
estimators is important prior to deployment. In that context, the
main goal of the paper was to highlight few drawbacks asso-
ciated with the current approach of statistical comparison and
validation. These stem primarily from lack of considerations
to theoretical and practical aspects of statistical testing.
Therefore, the main goal of the paper was to raise awareness
about some of the identified issues in the current approach.
We also provided theoretical analysis concerning dependent
(correlated) sample observations. Further, we discussed several
other limitations related to sample size, the lack of assessment
of the magnitude of treatment effect and an almost exclusive
reliance on p values to compare ML based quality predic-
tors. We also argued that assessment of learning ability is an
important aspect to validate such learning based predictors,
and discussed the use of a permutation test to that end.
Essentially, the proposed guidelines treat statistical compar-
ison of ML based quality estimators as a multi-dimensional
problem. Accordingly, we seek to assess the predictors more
holistically in terms of their local performance on specific test
conditions, their learning ability and the magnitude of treat-
ment effect (in order to quantify the practical significance of
the observed differences). In contrast, the current approach
tends to reduce this task to binary and global statistical deci-
sion making, and does not reveal systematic weaknesses (or
strengths) of the predictors. In order to provide a tool for
practical use, a software implementing the proposed guidelines
is made publicly available.4
4 https://sites.google.com/site/narwariam/home/research
IEEE TRANSACTIONS ON BROADCASTING
R EFERENCES
[1] P. Coverdale, S. Moller, A. Raake, and A. Takahashi, “Multimedia qual-
ity assessment standards in ITU-T SG12,” IEEE Signal Process. Mag.,
vol. 28, no. 6, pp. 91–97, Nov. 2011.
[2] P. Gastaldo, S. Rovetta, and R. Zunino, “Objective quality assessment
of MPEG-2 video streams by using CBP neural networks,” IEEE Trans.
Neural Netw., vol. 13, no. 4, pp. 939–947, Jul. 2002.
[3] J. Xu, P. Ye, Y. Liu, and D. Doermann, “No-reference video quality
assessment via feature learning,” in Proc. IEEE Int. Conf. Image Process.
(ICIP), Paris, France, Oct. 2014, pp. 491–495.
[4] K. Zhu, C. Li, V. Asari, and D. Saupe, “No-reference video qual-
ity assessment based on artifact measurement and statistical analysis,”
IEEE Trans. Circuits Syst. Video Technol., vol. 25, no. 4, pp. 533–546,
Apr. 2015.
[5] N. Staelens et al., “Constructing a no-reference H.264/AVC bitstream-
based video quality metric using genetic programming-based symbolic
regression,” IEEE Trans. Circuits Syst. Video Technol., vol. 23, no. 8,
pp. 1322–1333, Aug. 2013.
[6] J. Søgaard, S. Forchhammer, and J. Korhonen, “No-reference video qual-
ity assessment using codec analysis,” IEEE Trans. Circuits Syst. Video
Technol., vol. 25, no. 10, pp. 1637–1650, Oct. 2015.
[7] B. Konuk, E. Zerman, G. Nur, and G. B. Akar, “A spatiotemporal no-
reference video quality assessment model,” in Proc. IEEE Int. Conf.
Image Process., Melbourne, VIC, Australia, Sep. 2013, pp. 54–58.
[8] W. Xue, X. Mou, L. Zhang, A. C. Bovik, and X. Feng, “Blind
image quality assessment using joint statistics of gradient magnitude
and Laplacian features,” IEEE Trans. Image Process., vol. 23, no. 11,
pp. 4850–4862, Nov. 2014.
[9] M. Narwaria and W. Lin, “SVD-based quality metric for image and
video using machine learning,” IEEE Trans. Syst., Man, Cybern. B,
Cybern., vol. 42, no. 2, pp. 347–364, Apr. 2012.
[10] A. Mittal, A. K. Moorthy, and A. C. Bovik, “No-reference image quality
assessment in the spatial domain,” IEEE Trans. Image Process., vol. 21,
no. 12, pp. 4695–4708, Dec. 2012.
[11] D. C. Mocanu et al., “No-reference video quality measurement: Added
value of machine learning,” J. Electron. Imag., vol. 24, no. 6, 2015,
Art. no. 061208, doi: 10.1117/1.JEI.24.6.061208.
[12] M. T. Vega, D. C. Mocanu, J. Famaey, S. Stavrou, and A. Liotta, “Deep
learning for quality assessment in live video streaming,” IEEE Signal
Process. Lett., vol. 24, no. 6, pp. 736–740, Jun. 2017.
[13] “Methods, metrics and procedures for statistical evaluation, qual-
ification and comparison of objective quality prediction mod-
els,” Int. Telecommun. Union, Geneva, Switzerland, Rep. ITU-T
Recommendation P.1401, Jul. 2012.
[14] “Method for specifying accuracy and cross-calibration of video qual-
ity metrics (VQM),” Int. Telecommun. Union, Geneva, Switzerland,
Rep. ITU-T Recommendation J.149, Jul. 2004.
[15] ITU-T Tutorial, “Objective perceptual assessment of video quality: Full
reference television,” Int. Telecommun. Union, Geneva, Switzerland,
Rep. JSTP-OAVQ (2004), May 2005.
[16] “Final report from the video quality experts group on the validation
of objective quality metrics for video quality assessment,” Video Qual.
Experts Group, NTIA/ITS, Boulder, CO, USA, Rep. FR-TV Phase II,
Mar. 2003.
[17] “Method for the subjective assessment of intermediate quality levels
of coding systems,” Int. Telecommun. Union, Geneva, Switzerland,
Rep. ITU-R Recommendation BS.1534-3, Oct. 2015.
[18] J. Demšar, “Statistical comparisons of classifiers over multiple data sets,”
J. Mach. Learn. Res., vol. 7, pp. 1–30, Dec. 2006.
[19] M. Ojala and G. C. Garriga, “Permutation tests for studying classifier
performance,” J. Mach. Learn. Res., vol. 11, pp. 1833–1863, Aug. 2010.
[20] G. Roussas, An Introduction to Probability and Statistical Inference.
London, U.K.: Academic Press, 2015.
[21] G. M. Sullivan and R. Feinn, “Using effect size-or why the P value is
not enough,” J. Grad. Med. Educ., vol. 4, no. 3, pp. 279–282, Sep. 2012.
[22] R. Hoekstra, S. Finch, H. A. L. Kiers, and A. Johnson, “Probability
as certainty: Dichotomous thinking and the misuse of P values,”
Psychonomic Bull. Rev., vol. 13, no. 6, pp. 1033–1037, 2006.
[23] M. Baker, “Statisticians issue warning over misuse of P values,” Nature,
vol. 531, no. 7593, p. 151, Mar. 2016.
[24] K. Kelley and K. Preacher, “On effect size,” Psychol. Methods, vol. 17,
no. 2, pp. 137–152, 2012.
[25] M. Narwaria, L. Krasula, and P. L. Callet, “Data analysis in multi-
media quality assessment: Revisiting the statistical tests,” IEEE Trans.
Multimedia, to be published, doi: 10.1109/TMM.2018.2794266.
 This article has been accepted for inclusion in a future issue of this journal. Content is final as presented, with the exception of pagination.
IEEE Transactions on Broadcasting ( Volume: 64, Issue: 2, June 2018 )
NARWARIA: TOWARD BETTER STATISTICAL VALIDATION OF ML-BASED MULTIMEDIA QUALITY ESTIMATORS
[26] B. De Finetti, G. Koch, and F. Spizzichino, “Exchangeability in prob-
ability and statistics,” in Proc. Int. Conf. Exchangeability Probability
Stat., Rome, Italy, Apr. 1982, pp. 97–112.
[27] T. F. Crack and O. Ledoit, “Central limit theorems when data are
dependent: Addressing the pedagogical gaps,” J. Financ. Educ., vol. 36,
nos. 1–2, pp. 38–60, 2010.
[28] S. Lahiri and P. M. Robinson, “Central limit theorems for long
range dependent spatial linear processes,” Bernoulli, vol. 22, no. 1,
pp. 345–375, 2016.
[29] K. Sainani, “Statistically speaking,” PM R, vol. 2, no. 9, pp. 858–861,
2010.
[30] K. J. Rothman, “No adjustments are needed for multiple comparisons,”
Epidemiology, vol. 1, no. 1, pp. 43–46, 1990.
[31] T. V. Perneger, “What’s wrong with bonferroni adjustments,” BMJ,
vol. 316, no. 7139, pp. 1236–1238, 1998.
[32] R. J. Feise, “Do multiple outcome measures require P-value adjust-
ment?” BMC Med. Res. Methodol., vol. 2, no. 1, p. 8, Jun. 2002.
[33] P. J. Mumby, “Statistical power of non-parametric tests: A quick guide
for designing sampling strategies,” Marine Pollution Bull., vol. 44, no. 1,
pp. 85–87, 2002.
[34] E. Kasuya, “Mann–Whitney U test when variances are unequal,” Animal
Behav., vol. 61, no. 6, pp. 1247–1249, Jun. 2001.
[35] B. Efron and T. Hastie, Computer Age Statistical Inference: Algorithms,
Evidence, and Data Science, 1st ed. New York, NY, USA: Cambridge
Univ. Press, 2016.
[36] T. J. DiCiccio and B. Efron, “Bootstrap confidence intervals,” Stat. Sci.,
vol. 11, no. 3, pp. 189–212, 1996.
[37] J. Carpenter and J. Bithell, “Bootstrap confidence intervals: When,
which, what? A practical guide for medical statisticians,” Stat. Med.,
vol. 19, no. 9, pp. 1141–1164, 2000.
[38] A. C. Davison and D. V. Hinkley, Bootstrap Methods and
Their Application (Cambridge Series in Statistical and Probabilistic
Mathematics). Cambridge, U.K.: Cambridge Univ. Press, 1997.
[39] T. Fawcett, “An introduction to ROC analysis,” Pattern Recogn. Lett.,
vol. 27, no. 8, pp. 861–874, Jun. 2006.
[40] P. Hanhart, L. Krasula, P. L. Callet, and T. Ebrahimi, “How to benchmark
objective quality metrics from paired comparison data?” in Proc. 8th Int.
Conf. Qual. Multimedia Exp. (QoMEX), Lisbon, Portugal, Jun. 2016,
pp. 1–6.
[41] L. Krasula, K. Fliegel, P. L. Callet, and M. Klíma, “On the accuracy
of objective image and video quality models: New methodology for
performance evaluation,” in Proc. 8th Int. Conf. Qual. Multimedia Exp.
(QoMEX), Lisbon, Portugal, Jun. 2016, pp. 1–6.
[42] L. Krasula, M. Narwaria, K. Fliegel, and P. L. Callet, “Preference of
experience in image tone-mapping: Dataset and framework for objective
measures comparison,” IEEE J. Sel. Topics Signal Process., vol. 11,
no. 1, pp. 64–74, Feb. 2017.
15
[43] R. Maglietta et al., “Selection of relevant genes in cancer diagnosis
based on their prediction accuracy,” Artif. Intell. Med., vol. 40, no. 1,
pp. 29–44, 2007.
[44] E. Frank and I. H. Witten, “Using a permutation test for attribute
selection in decision trees,” in Proc. 15th Int. Conf. Mach. Learn.,
San Francisco, CA, 1998, pp. 152–160.
[45] D. Bamber, “The area above the ordinal dominance graph and the
area below the receiver operating characteristic graph,” J. Math.
Psychol., vol. 12, no. 4, pp. 387–415, 1975. [Online]. Available:
http://www.sciencedirect.com/science/article/pii/0022249675900012
[46] J. A. Hanley and B. J. McNeil, “The meaning and use of the area under
a receiver operating characteristic (ROC) curve,” Radiology, vol. 143,
no. 1, pp. 29–36, 1982, doi: 10.1148/radiology.143.1.7063747.
[47] C. Cortes and M. Mohri, “Confidence intervals for the area under
the ROC curve,” in Proc. 17th Int. Conf. Neural Inf. Process. Syst.
(NIPS), Vancouver, BC, Canada, 2004, pp. 305–312. [Online]. Available:
http://dl.acm.org/citation.cfm?id=2976040.2976079
[48] E. C. Larson and D. M. Chandler, “Most apparent distortion: Full-
reference image quality assessment and the role of strategy,” J. Electron.
Imag., vol. 19, no. 1, pp. 1–21, 2010.
[49] A. Mittal, A. K. Moorthy, and A. C. Bovik, “No-reference image quality
assessment in the spatial domain,” IEEE Trans. Image Process., vol. 21,
no. 12, pp. 4695–4708, Dec. 2012.
[50] Q. Li, W. Lin, and Y. Fang, “No-reference quality assessment for
multiply-distorted images in gradient domain,” IEEE Signal Process.
Lett., vol. 23, no. 4, pp. 541–545, Apr. 2016.
[51] Y. Fang et al., “No-reference quality assessment of contrast-distorted
images based on natural scene statistics,” IEEE Signal Process. Lett.,
vol. 22, no. 7, pp. 838–842, Jul. 2015.
[52] M. Narwaria, W. Lin, I. V. McLoughlin, S. Emmanuel, and L.-T. Chia,
“Fourier transform-based scalable image quality measure,” IEEE Trans.
Image Process., vol. 21, no. 8, pp. 3364–3377, Aug. 2012.
Manish Narwaria received the Ph.D. degree in
computer engineering from Nanyang Technological
University, Singapore, in 2012. He was a Researcher
with IRCCyN-IVC Lab, France, before joining DA-
IICT, India, as an Assistant Professor in 2015.
His major research interests include the area of
multimedia signal processing with focus on percep-
tual aspects toward content capture, processing, and
transmission.