2014 Sixth International Workshop on Quality of Multimedia Experience (QoMEX)
A QUALITY OF EXPERIENCE MODEL FOR ADAPTIVE MEDIA PLAYOUT
Benjamin Rainer and Christian Timmerer
Multimedia Communication (MMC) Research Group, Institute of Information Technology (ITEC)
Alpen-Adria-Universität Klagenfurt, Klagenfurt, Austria, {firstname.lastname}@itec.aau.at
ABSTRACT
In the past decade Adaptive Media Playout (AMP) has been
intensively studied with respect to the detection of when to
increase or decrease the playback rate in order to maintain a
certain buffer fill state. In this paper we subjectively assess
the QoE of AMP with respect to non-periodically and ran-
domly selected content sections of a video sequence by using
crowdsourcing. Furthermore, we introduce metrics that allow
to quantify the distortion for audio and video that are caused
by increasing or decreasing the playback rate. With these pre-
liminaries we study the correlation between the introduced
metrics and the subjectively assessed QoE. Therefore, we de-
rive a utility model that allows estimating the QoE with the
introduced metrics. We instantiate and validate the model us-
ing the data gathered from the conducted study.
Index Terms— Adaptive Media Playout, Inter-Destination
Media Synchronization, Crowdsourcing, Subjective Quality
Assessment, Quality of Experience
1. INTRODUCTION
Synchronizing the multimedia playback among geographi-
cally distributed users is one of the features enabled by Inter-
Destination Multimedia Synchronization (IDMS) [1]. A sys-
tem that claims to provide IDMS requires the four following
mechanisms:
• Session management of clients requiring IDMS.
• Signaling of timing and control information among
the participating clients within an IDMS session.
• Negotiation on a reference playback timestamp
deals with the selection of a playback timestamp within
an IDMS session to which clients have to synchronize
their playback.
• Carrying out the actual synchronization to overcome
the identified asynchronism by modifying the multime-
dia playback of each client.
In this paper we focus on carrying out the actual syn-
chronization. A naı̈ve strategy for carrying out the synchro-
nization is to pause the multimedia playback or skip multi-
media content. Recent subjective quality assessments have
978-1-4799-6536-6/14/$31.00 ©2014 IEEE
shown that the QoE decreases exponentially with an increase
in stalling events of the multimedia playback [2]. Therefore,
we propose to use Adaptive Media Playout (AMP) which
changes the playback rate adaptively. As stated in [3] and
[4], AMP was initially introduced for compensating the im-
pact of error prone communication channels on the smooth-
ness of the multimedia playback and avoiding buffer under-
/overruns. That is, the playback rate is increased/decreased in
order to maintain a particular playback buffer fill state. In the
literature, many AMP algorithms have been introduced which
assume an error prone communication channel and most of
them are based on the buffer fill state [5].
Other studies investigated the decrease of the frame rate
for short time periods for low bit-rate videos [6]. In [7] the
impact of reducing the frame rate for the entire duration of a
video sequence was investigated. Most of the discussed stud-
ies investigated AMP for video only, a playback rate lower
than the nominal playback rate, and for the full duration of
the multimedia content.
The aim of this paper is to study and quantify the impact
on the QoE of increasing and decreasing the playback rate for
randomly selected content sections with different length (in
time). Therefore, we present audio-visual metrics that allow
us to quantify these playback rate variations (Section 2). We
further verify our assumptions by correlating our metrics with
the results of a conducted subjective quality assessment using
crowdsourcing (Section 3). Finally, we analytically derive a
utility model that describes the correlation between the QoE
and the introduced metrics and we provide insights on how
the change in the playback rate affects the QoE (Section 4).
The results and contributions are summarized and concluded
in Section 5 including future work.
2. QUANTIFYING THE DISTORTION OF
MULTIMEDIA CONTENT IN THE TEMPORAL
DOMAIN
In the past decade many spatial quality metrics have been in-
troduced, especially for the video domain [8]. It has been
shown that the spatial quality metrics are able to represent the
impact on the QoE to a certain extent. Recent quality met-
rics aim to cover the temporal domain. For example, in [9] a
177
 2014 Sixth International Workshop on Quality of Multimedia Experience (QoMEX)

spatial-temporal quality metric for video has been introduced dvi may be any value in the interval of [−1, 1]. |F | depicts the
and subjectively evaluated. This metric relies on the image overall number of frames. For audio we followed the same
data of each video frame and quantifies its spatial-temporal principle as for video with the difference that we used the
distortion. In our case we aim on quantifying the distortion spectral energy of the audio frames for each audio channel by
caused by playback rate variations and, thus, we focus on tem- using the Fourier Transformation. The Fourier Transforma-
poral information only. tion for a single audio frame and for the i-th content section
Increasing or decreasing the media playback rate – de- is denoted by baci (cf. Equation 3), where c depicts the audio
noted as µ – results in a perceptual distortion in audio and/or channel. Equation 4 defines our distortion metric for audio.
video. This distortion depends on the actual multimedia con- Note that we take into account the audio channels.
tent, especially on the temporal features, for which the play- Mi
X jk
back rate is increased or decreased. Therefore, we propose aci (k) =
b e−2πi M fac (j) (3)
the following metrics for measuring the distortion caused by j=Ni
∼
modifying the playback rate of audio and video: C Fei
Sf Fei Sf
X X X X X
• Audio: the spectral energy of an audio frame for the si = (( |b
acu (k)| − |b
acu (k)|)) (4)
c=1 ∼
c-th channel is denoted by fac (x). u=Fsi
k=0 u=Fsi k=0

• Video: the average length of motion vectors between
two consecutive frames is denoted by fv (x).
These metrics allow us to quantify the distortion for audio
and video when increasing or decreasing the playback rate
for a specific content section. This is done by comparing how
much of each temporal feature has been experienced by the
user during the content section with and without the playback
rate change. We differentiate between increasing and decreas-
ing the playback rate when calculating our distortion metrics
because our hypotheses is that increasing the playback rate
may have a different impact on the QoE than decreasing the
playback rate.
In order to determine our metrics we calculate the last and
the first frame number for the content section for which the
playback rate shall be changed. Determining the first frame
for the i-th content section for which the playback rate is
changed is done by Fs (tsi , tei ) = btsi · f psµ0 c, where tsi , tei
denotes the start and end of the i-th content section in sec-
onds. f psµ0 represents the frames per second for the nominal
playback rate µ0 . For determining the last frame of the i-th
content section for which the playback rate has been increased
∼ 3.1. Participants, Stimuli, Methodology, and Assessment
or decreased we use the following function Fe (tsi , tei ) as de-
picted in Equation 1.
∼ For conducting our user study we selected the crowdsourc-
Fe (tsi , tei ) = bFs (tsi , tei ) + (tei − tsi ) · f ps∆µ c (1)
f ps∆µ denotes the frames per seconds for the changed play-
back rate ∆µ (∆µ = µ0 +δµ, where δµ denotes the change of
∼
the playback rate). With δµ equal to zero, Fe (tsi , tei ) yields
the last frame for the i-th content section without any play-
back rate changes which is denoted by Fe (tsi , tei ). In the
following, we introduce the metrics for the distortion in au-
dio (dai ) and the distortion in video (dvi ) for the i-th content
section. Equation 2 denotes the distortion metric for video.
∼
Fe (tsi ,tei ) Fe (tsi ,tei )
1 X X evaluation methodology used to conduct the study. The intro-
dvi = P|F | ( fv (j) −
fv (j)
j=1 j=Fs (tsi ,tei ) j=Fs (tsi ,tei )
C denotes the number of audio channels available. Sf de-
notes the highest frequency. Finally, dai denotes the distor-
tion in audio for the i-th content section.
1
dai = PC si (5)
c=1 sec
sec denotes the overall spectral energy of channel c. Again,
dai may be any value in the interval of [−1, 1]. If there are
more content sections for which the playback rate is changed
we use the average of the introduced metrics denoted by dv
and da , respectively.
3. SUBJECTIVE QUALITY ASSESSMENT USING
CROWDSOURCING
For validating our metrics and to investigate their correlation
with the QoE, we conducted a subjective quality assessment
using crowdsourcing in the following referred as study [10].
Therefore, we will first describe the key aspects of the study
followed by the screening/filtering of the participants and the
statistical analysis of the results.
Platform
ing platform Microworkers1 . Microworkers allows hosting
so-called campaigns to which subjects (called microworkers)
can subscribe. These campaigns include a detailed descrip-
tion of the task and asks each participant to hand in a proof in
order to verify their participation. The duration of the study
is approximately 15 minutes. We have found that the typi-
cal amount of money that is payed for a task with a duration
of about 15 minutes is approximately $0.20. Therefore, we
have set a slightly higher compensation of $0.25 as an ex-
tra motivation for each participant [11]. Figure 1 depicts the
fv (j))
duction explains the task and the test procedure. Furthermore,
(2) 1 http://www.microworkers.com, last access: July 2014.

178
 2014 Sixth International Workshop on Quality of Multimedia Experience (QoMEX)
Fig. 1. Methodology for the Study.
the participants are asked to agree to a disclaimer. The pre-
questionnaire allows us to gather demographic information
and helps us to check whether the participants are from the
countries we asked for at Microworkers. The training phase
should allow the participants to become familiar with the task
and the rating possibilities. In order to introduce AMP to the
participants and to its effects on the media playback we se-
lected the video sequence Babylon A. D. taken from [12]. The
training sequence is presented three times with three different
media playback rates µ ∈ {1, 0.5, 2}, i.e., µ = 1 corresponds
to the nominal playback rate µ0 , µ = 0.5 is half the nominal
playback rate µ0 , and µ = 2 denotes twice the nominal play-
back rate µ0 . The training sequence is presented for its whole
duration with the mentioned media playback rates.
After the training phase the main evaluation starts. For
the stimulus we selected a video sequence with audio and
duration of 51 seconds from the beginning of the open source
movie Big Buck Bunny2 . We annotated the content sec-
tions for which the media playback rate is increased or de-
creased. The playback rate changes were initially randomly
scattered throughout the whole video sequence with a cu-
mulative duration of 8.84 seconds, thus reflecting 17.3% of
the sequence’s duration. As indicated in Figure 1 we use
a single stimulus method as recommended in [13, 14]. We
use a continuous rating scale [0, 100] represented by a slider.
Furthermore, we randomly inserted a control question after
a stimulus presentation. In particular, this control question
asks the participants what they have seen in the previous
video sequence with three possible answers provided. In
total the sequence is presented nine times to the partici-
pants with the following configurations for the playback rate
µ ∈ {0.5, 0.6, 0.8, 1, 1.2, 1.4, 1.6, 1.8, 2}. The nominal play-
back rate µ0 = 1 depicts the hidden reference. Please note
that we only modify the playback rate of specific sections and
not for the entire video sequence.
At the end, we ask the participants to fill out a short post-
questionnaire which provides the possibility to give feedback
and to state whether subjects had already participated in a
similar study. The study is conducted by adopting an open-
source Web-based QoE assessment platform [15]. Please note
that the media player used by this platform uses the Wave-
form Similarity based Overlap-Add which tries to maintain
the pitch of audio when increasing or decreasing the play-
back rate [16]. In addition to its usability, the platform pro-
vides several mechanisms to track the behavior of participants
in terms of measuring the time of each stimulus presentation
2 http://www.bigbuckbunny.org, last access: July 2014.
179
and the possibility to ask control questions. The screening of
participants according to the mentioned mechanisms is dis-
cussed in the following section.
3.2. Screening and Filtering of Participants
Using crowdsourcing for subjective quality assessments is
gaining more and more attention due to reduced costs com-
pared to in-lab studies. However, crowdsourced studies are
unsupervised and adopting traditional screening methods –
such as outlier detection – are insufficient as participants may
try to cheat [17]. Therefore, we use additional data provided
by the Web-based QoE assessment platform for screening
and filtering participants as follows:
First, we use the above mentioned control question. An
incorrect answer leads to an exclusion of the participant from
the final statistical analysis because an incorrect answer may
indicate that the participant did not thoughtful take part in the
study.
Second, we try to detect participants who skipped, short-
ened, or paused the stimulus presentation. Therefore, we use
the F-test to test whether there exists a significant difference
between the variance of the theoretical playback time of each
stimulus presentation and the actual playback time of each
s2 2
participant represented by the statistics f = sX 2 , where sY is
Y
the sample variance of the nominal playback durations and s2X
the sample variance of playback times of a participant. This
ratio follows a F distribution with m − 1 and n − 1 degrees of
freedom.
Third, we investigate those participants who rated very
often extreme values. We define extreme values as the set of
values 0 − 5, 50, 95 − 100. This shall allow us to identify par-
ticipants that only moved the slider a few times from its initial
position (50) and those who just pushed the slider into the left
or right direction very often. According to the binomial distri-
bution with p = 12 the probability of rating an
 extreme value
Pk n j
or not is given by P(X ≤ k) = j=0 p (1 − p)n−j .
j
Therefore, the probability that a certain extreme value is se-
lected more than k-times is P(X > k) = 1 − P(X ≤ k). We
rejected the ratings of a participant if the probability for se-
lecting a certain extreme value more than k times was below
α = 5%, P(X > k) < α.
Without any filtering applied 80 persons participated in
the study. 11 participants have provided a wrong answer for
the control question. With the F-test on the variance of the
actual playback time of the stimulus presentations, five par-
ticipants could be identified that paused the playback or had
statistically significant lower playback times (i.e., skipped at
least one stimulus presentation). For the ratings nine partici-
pants have been screened that did not vote correctly (by con-
stantly voting 0-5, 50 or 95-100). Therefore, 55 participants
(48 male and 7 female) have been considered for the final sta-
tistical analysis of the results.
 2014 Sixth International Workshop on Quality of Multimedia Experience (QoMEX)
Fig. 2. MOS and 95% CI for (dv , da , µ).
3.3. Statistical Analysis of the Results
After screening the participants and their responses, the rat-
ings for each stimulus presentation were subject to statistical
significance tests. According to the Central Limit Theorem
we assume that the ratings of the participants are normally
distributed. Nevertheless, we have conducted a Shapiro-Wilk-
test to assess whether the ratings are not normally distributed.
The null hypothesis (H0), stating that no normal distribution
is present, was rejected for each configuration of playback
rates of the stimulus presentations. By the use of the metrics
introduced in Section 2 the distortion caused by increasing or
decreasing the playback rates for the selected content sections
are expressed by da for the average distortion in the frequency
domain for the audio channels and dv for the average distor-
tion of motion for video.
Figure 2 depicts for each triple (dv , da , µ) the assessed
Mean Opinion Score (MOS). It can be observed that play-
back rates near the nominal playback rate of µ = 1 cause only
a slight drop in QoE. A Student’s t-test supports this finding
by stating no significant difference in MOS between the ref-
erence of µ = 1 and the following playback rates: µ = 0.8
(p = 0.93, t = −0.083); µ = 1.2 (p = 0.92, t = 0.096);
µ = 1.4 (p = 0.81, t = 0.42); µ = 1.6 (p = 0.22, t = 1.23);
µ = 1.8 (p = 0.16, t = 1.41). These results indicate that the
users could not notice a significant difference for playback
rates µ ∈ [0.8, 1.8].
For the other playback rates it can be observed that the
QoE significantly degrades. A Student’s t-test revealed that
there exists a significant difference in the MOS from the nom-
inal playback rate of µ = 1 and the media playback rates with
following values: µ = 0.5 (p = 0.00, t = 4.5217); µ = 0.6
(p = 0.002, t = 3.2); µ = 2 (p = 0.03, t = 2.19). These
results provide the evidence that users perceived a significant
difference between the reference and the test conditions.
In the following we investigate how well our metrics cor-
180
relate with the MOS for the different playback rate configura-
tions. The Pearson correlation coefficient for dv and the QoE
ratings is ρ = 0.43. For da the Pearson correlation coefficient
is ρ = 0.679. If we take the absolute values of |da | and |dv |
for calculating the Pearson correlation coefficient we obtain
following values for the linear correlation between the met-
rics and the QoE scores. The Pearson correlation coefficient
for |da | and the QoE ratings is ρ = −0.5549 and for |dv | and
the QoE ratings ρ = −0.9565. For the metric in the audio do-
main both da and |da | show a low linear correlation with the
obtained MOS, respectively. For the metric in the video do-
main we have obtained contrary results. Taking dv we have a
low linear correlation between the metric and the QoE ratings
but if we take |dv | there exists a high negative linear corre-
lation. Nevertheless, if we want to retain the ability of dis-
tinguishing whether the playback rate has been decreased or
increased we have to use the signed metrics. The low values
between the signed metrics and the QoE scores show us that
assuming a linear relationship between the distortion metrics
and the assessed QoE may not be appropriate. Therefore, we
try to find a model that explains this correlation better than a
linear model which will be discussed in the next sections.
Finally, the results indicate that with an increase in |dv |
and |da | the QoE is reduced. Interestingly, the QoE does not
decrease linearly. For the playback rate changes in the range
of [0.8, 1.8] the QoE remains high compared to the reference.
Increasing or decreasing the playback rate further causes a
huge drop in the QoE.
4. QOE UTILITY MODEL FOR AMP
In [5] several AMP algorithms were assessed on their impact
on the QoE regarding QoS parameters such as the initial play-
back delay, loss rate, underflow time ratio, and the playback
rate by introducing cross traffic. The actual impact on the
playback of the content was not taken into account. Further-
more, there is the need for a model which can be easily com-
bined with other QoE metrics in order to assess the QoE of a
system that uses AMP. The presented results of the conducted
subjective quality assessment using crowdsourcing gave us a
first impression on how the QoE degrades with an increase or
decrease in the playback rate when selecting content sections
with a short time duration. With the knowledge that the Pear-
son correlation between the QoE scores and the audio/video
metric is low, we try to find a function which allows us to
approximate the QoE more precisely than a linear function
could do. Therefore, we need a function that allows to esti-
mate the QoE from the distortions for audio and video and re-
turns us the coefficient of degradation, like ζ : R×R → [0, 1].
Therefore, let
1 x−θ1 2 1 y−θ3 2
ζ(x, y)θ = e− 2 ( θ2 ) e− 2 ( θ4 )
be the two-dimensional function of degradation with the pa-
rameter vector θ that describes the relationship between the
 2014 Sixth International Workshop on Quality of Multimedia Experience (QoMEX)

degradation of the QoE and the quantitative metrics (dv , da ).

Furthermore, ζ(x, y)θ is logarithmic concave for (x, y). The
results of the subjective quality assessment indicated that
small distortions in the metrics, especially for audio, do al-
ready cause an impact on the QoE. Therefore, we formulate
our model as follows:
QoE(dv , da ) = QoEwo · ζ(dv , da )θ∗ , (6)
∗
where θ represents the optimal parameter vector such that
for a cost function f (θ) for each θ it holds that f (θ) ≥ f (θ ∗ ).
Thus, θ ∗ representing the optimal solution to a minimization
problem. The QoEwo states the QoE without any playback
rate changes. This QoEwo can be derived from QoS parame-
ters (e.g., bit-rate, resolution, delay, jitter, etc.) by the use of
existing models such as those in [5, 18]. The assessment of
the actual QoEwo is out of scope of this paper.
In order to instantiate our proposed utility model (cf.
Equation 6) we use the responses received by the conducted
subjective quality assessment using crowdsourcing. We fit Fig. 3. Fitted ζ for the received responses.
our model to the obtained data by using multiple instances of the actual data revealed that the instantiated model reflects
of the conjugate gradient method [19]. For the cost function 92.48% of the variability of the actual data. Furthermore, we
we used the Least-Square-Estimator or the squared l2 -norm conducted an F-test to test whether θ is the zero vector. The
given by ||ln(ζ θ (x, y)) − ln(z)||22 depicted in Equation 7. test revealed that the null-hypotheses can be rejected with p =
This function is neither convex nor concacve for θ, but at 2.49 · 10−4 and F = 27.84 for α = 5%. We have shown that
least twice differentiable. ζ(x, y) fits our purpose quite well and provides the possibility
to estimate the coefficient of degradation for different values
X
f (θ) = (ln(ζ(x, y)θ ) − ln(z))2 , (7)
(x,y,z)∈M
of (dv , da ).
QoE(x,y)
where M := {(x, y, z) ∈ R3 |z = max{QoE(x,y)} } is the set
x,y

of 3-tuples with z representing the QoE assessed for (da , dv ).
Therefore, we try to find the parameter vector θ ∗ that mini-
mizes our f (θ). For determining the conjugated search direc-
tions we use the method proposed by Polak-Ribiere [20]. In
order to find a near optimal vector θ we use multiple instances
of the conjugate gradient algorithm with starting points uni-
formly distributed in the interval of ]0, 1] for all θi . We se-
lected the θ that provided the lowest costs in terms of our cost
function f (θ).
By the use of the conjugate gradient method we fitted
ζ(x, y)θ to the responses received during our study discussed
in Section 3 and we found the following values for the param-
eter vector θ ∗ = (0.0011, 0.0482, −0.0004, 0.0184)T . Equa-
tion 8 depicts the fully instantiated utility model.
1 x−0.0011 2 1 y+0.0004 2
QoE(dv , da )θ∗ = QoEwo · e− 2 ( 0.0482 ) e− 2 ( 0.0184 )
(8)
Figure 3 depicts the QoE(dv , da )θ∗ by the use of the fitted
ζ. An interesting finding is that a distortion in audio impacts
the QoE more than the same amount of distortion in video.
This can be observed by comparing the second and fourth
component of θ or by taking a look at Figure 3.
To test how well our utility model fits the actual data, we
have conducted an analysis of variance (ANOVA) on how the
fitted model reflects the variability of the actual data. The
ratio of the sum of squares of the model and sum of squares
5. DISCUSSION AND CONCLUSION
As mentioned in Section 3 we use only the first 51 seconds of
Big Buck Bunny as multimedia content. This video sequence
is presented with different configuration of content sections
and playback rates for these content sections. Recency ef-
fects may occur and participants may unwittingly provide un-
reliable ratings. Even shuffling the stimulus presentation in
a random fashion does not avoid recency effects in this case.
Especially, when extreme conditions are presented consecu-
tively (e.g., first with a playback rate of µ = 0.5 and then with
µ = 1.4). An option is to use different video sequences. But,
this may have an influence on the actual task because partic-
ipants may like or dislike them and, therefore, this may have
an impact on the provided QoE rating. Therefore, we decided
to use only a single video sequence for this subjective qual-
ity assessment. Another possibility, that would have increased
the duration of the crowdsourced study, is to introduce dummy
video sequences which allow the participants to forget the last
real stimulus presentation. Nevertheless, further subjective
quality assessments have to be conducted in order to support
the findings presented in this paper (e.g., [21]).
The results of the subjective quality assessment using
crowdsourcing lead us to the hypotheses that the correlation
between the distortion in the video/audio domain and the
QoE of playback rate variations can be described by a non

181
 2014 Sixth International Workshop on Quality of Multimedia Experience (QoMEX)
linear model. The introduced model lead us to the finding that
audio plays an important role when increasing or decreasing
the playback rate. This is depicted in Figure 3 and denoted
in Equation 8. Comparing our results to the results obtained
by other subjective quality assessments which assess the QoE
of playback variations for video only [6, 7, 22] we can see
that altering the playback rate for the combination of audio
and video has a very different impact on the QoE. Please note
that for video only, the human perception is more tolerant for
playback rate variations. This is not the case for audio and
the combination of audio and video as shown in this paper.
The contribution of this paper is twofold. First, we have
shown that there is a significant difference between the im-
pact of increasing and decreasing the playback rate on the
QoE. An interesting finding is that increasing the playback
rate for specific content sections have a lower impact on the
QoE than decreasing the playback rate by the reciprocal of
the increase of the playback rate. Second, we have introduced
metrics that measure the distortion in audio and video caused
by increasing or decreasing the playback rate. With the use of
these metrics we derive a utility model.
Future work comprises the use of the obtained utility
model in order to determine content sections that minimize
the impact of playback variations on the QoE. Considering
the use case of carrying out the synchronization in IDMS, the
buffer contents may be used to determine appropriate content
sections for overcoming the identified asynchronism.
Acknowledgments: This work was supported in part by
the EC in the context of the SocialSensor (FP7-ICT-287975)
and QUALINET (COST IC 1003) projects and partly per-
formed in the Lakeside Labs research cluster at AAU.
6. REFERENCES
[1] M. Montagud, F. Boronat, H. Stokking, and R. Branden-
burg, “Inter-destination multimedia synchronization:
schemes, use cases and standardization,” Multimedia
Systems, vol. 18, pp. 459–482, 2012.
[2] T. Hossfeld, M. Seufert, M. Hirth, T. Zinner, P. Tran-
Gia, and R. Schatz, “Quantification of youtube qoe via
crowdsourcing,” in ISM, 2011, pp. 494–499.
[3] M. Kalman, E. Steinbach, and B. Girod, “Adaptive me-
dia playout for low-delay video streaming over error-
prone channels,” IEEE Transactions on Circuits and
Systems for Video Technology, vol. 14, no. 6, pp. 841–
851, 2004.
[4] M. Yuang, S. Liang, and Y. Chen, “Dynamic video play-
out smoothing method for multimedia applications,”
Multimedia Tools and Applications, vol. 6, no. 1, pp.
47–60, 1998.
[5] M. Li, “Qoe-based performance evaluation for adaptive
media playout systems,” Advances in Multimedia, vol.
2013, p. 7, 2013.
182
[6] Y.-F. Ou, Y. Zhou, and Y. Wang, “Perceptual quality of
video with frame rate variation: A subjective study,” in
IEEE ICASSP, 2010, pp. 2446–2449.
[7] Q. Huynh-Thu and M. Ghanbari, “Perceived quality of
the variation of the video temporal resolution for low bit
rate coding,” in Picture Coding Symposium, 2007.
[8] W. Lin and C.-C. J. Kuo, “Perceptual visual quality met-
rics: A survey,” Journal of Visual Communication and
Image Representation, vol. 22, no. 4, 2011.
[9] Y. Wang, T. Jiang, S. Ma, and W. Gao, “Novel spatio-
temporal structural information based video quality
metric,” IEEE Transactions on Circuits and Systems for
Video Technology, vol. 22, no. 7, pp. 989–998, 2012.
[10] B. Rainer and C. Timmerer, “Self-Organized Inter-
Destination Multimedia Synchronization for Adaptive
Media Streaming,” in 22nd ACM Multimedia, 2014.
[11] M. Hirth, T. Hossfeld, and P. Tran-Gia, “Anatomy of
a Crowdsourcing Platform - Using the Example of Mi-
croworkers.com,” in 5th IMIS, June 2011, pp. 322–329.
[12] M. Waltl, C. Timmerer, B. Rainer, and H. Hellwag-
ner, “Sensory Effect Dataset and Test Setups,” in 4th
QoMEX. IEEE, 2012, pp. 115–120.
[13] “Rec. ITU-R BT.500-11,” Tech. Rep.
[14] ITU-T Recommendation P.910, “Subjective video qual-
ity assessment methods for multimedia applications,”
International Telecommunication Union, Geneva,
Switzerland, Tech. Rep., Apr. 2008.
[15] B. Rainer, M. Waltl, and C. Timmerer, “A Web based
Subjective Evaluation Platform,” in 5th QoMEX. IEEE,
jul 2013, pp. 24–25.
[16] W. Verhelst and M. Roelands, “An overlap-add tech-
nique based on waveform similarity (wsola) for high
quality time-scale modification of speech,” in IEEE
ICASSP, vol. 2, April 1993, pp. 554–557 vol.2.
[17] T. Hossfeld, C. Keimel, M. Hirth, B. Gardlo, J. Habigt,
K. Diepold, and P. Tran-Gia, “Best Practices for QoE
Crowdtesting: QoE Assessment with Crowdsourcing,”
IEEE Transactions on Multimedia, 2013.
[18] F. Pereira, “A triple user characterization model for
video adaptation and quality of experience evaluatio,”
in Multimedia Signal Processing, 2005 IEEE 7th Work-
shop on, 2005, pp. 1–4.
[19] C. T. Kelley, Iterative Methods for Linear and Nonlin-
ear Equations, ser. Frontiers in Applied Mathematics.
SIAM, 1995, no. 16.
[20] E. Polak, Computational methods in optimization; a
unified approach [by] E. Polak. Academic Press New
York, 1971.
[21] B. Rainer and C. Timmerer, “A subjective evaluation us-
ing crowdsourcing of adaptive media playout utilizing
audio-visual content features,” in IEEE QCMAN 2014,
H. Lutfiyya and P. Cholda, Eds. IEEE, may 2014.
[22] Z. Lu, W. Lin, B. C. Seng, S. Kato, E. Ong, and
S. Yao, “Perceptual Quality Evaluation on Periodic
Frame-Dropping Video,” in IEEE ICIP, vol. 3, 2007, pp.
433– 436.