2015 2nd National Foundation for Science and Technology Development Conference on Information and Computer Science

Automatic Test Framework For Video Streaming

Quality Assessment
Dat Truong Tieu+, Binh Cong Nguyen+, Linh Manh Le+, Quan Minh Pham+, Quang Tri Nguyen* and Dung Trung Vo+
+
Faculty of Electrical and Electronics Engineering
University of Technology, VNU-HCM
Ho Chi Minh City, Viet Nam
*
Personify Inc
Chicago, Illinois, USA

Abstract— To understand better what happens when video
streaming takes place, this paper introduces a framework to
simulate a real-time video streaming over wireless channels. The
system is divided into many modules and is simulated with
different tools. DUMMYNET is used for the network simulator
while FFMPEG is for coder/encoder and sender/receiver module.
The video quality is measured by spatial (SSIM and PSNR)
metrics. To permit automatic characteristic, a system controller
with JENKINS is used to control the encoder, sender, network,
receiver and decoder. A script at this server can be run to
simulate automatically the system with many test cases for the
practical network setting. The simulation results show that the
simulated system can estimate accurately the quality of the
videos transmitted over wireless network.
Keywords— video streaming; quality assessment; FFMPEG;
DUMMYNET; JENKINS; PSNR; SSIM
I. INTRODUCTION
Video streaming plays a very important role in our daily
lives and has become very popular these days. There exists a
very diverse range of different video communication and
streaming applications from video-on-demand channels like
YouTube, Netflix to live sports broadcasting or interactive on-
line games and video conferencing like Skype. However, our
knowledge about those kinds of applications’ behaviors in the
real world is quite not much. Therefore, to understand video
streaming better, it is necessary to build a framework for
assessing their behaviors.
Videos which are delivered to the users by transmission are
suffered from the imperfect of the channel, such as package
loss, delay, bandwidth drop… For videos, their quality are
perceived by the viewers in a different scope than images’
quality. For images, the spatial quality is a concern. For
videos, the temporal quality of frames over time is also
considered. In a higher level, annoyance to video viewers can
be caused by one missing block or one low quality frame or
temporal quality inconsistent. Quality metric on video
streaming is thus the key measurement not only to let viewers
verify the video quality but also to permit video broadcasters to
enhance their network. Quality metrics on video streaming thus
play a very important role in video transmission.
In [1], the effect when blocks are missed in a single frame
during transmission is studied. That paper simulates a single-
hop wireless test-bed with IEEE802.11n WLAN configuration.
If blocks in an intra-frame are lost due to one or more packages
are lost, not only the quality of this intra-frame is affected but
also the quality of the inter-frames predicted based on this
intra-frame is affected. This case is studied in [2] where this
error propagation is investigated with package loss. If the
artifact happens at fixed location for a long time, it becomes
very annoying. [3] also studies and predicts the visibility of
package loss for MPEG-2 videos. If both the original and
received videos are available, the quality metrics can be
calculated by a frame-based method. Metric with frame-based
methods are PSNR [4] and SSIM [5]. Not many metrics are
applied in real framework for video streaming yet. If the
original videos are not available at the decoder, optical flow [1]
can be used to estimate the temporal quality of the videos.
Based on a framework with video quality assessment, a
specific solution to enhance their performances can be verified
and be compared to other solutions. One application example is
that if the transmission bandwidth is dropped or is not stable,
the codec can reduce the speed by down-sampling the frames
or increase the compression ratio. The video is thus in smaller
bandwidth and is easier to be transmission over the channel
without any interruption. For these purposes, a novel
framework is proposed in this paper to simulate the whole
process of video streaming in the real system. The system
includes different modules which are implemented by specific
tools. Especially, the quality assessment module is done for
spatial metrics.
The rest of the paper is organized as follow: section II
describes the architecture of the proposed framework for
simulating the process of video streaming, section III, IV, V
and VI will describe in more details all the components of the
framework. In section VII, measurement of the proposal
framework performances under some network conditions is
reported. Then, the conclusions and future works are
mentioned in section VIII.

978-1-4673-6640-3/15/$31.00 ©2015 IEEE 214

 2015 2nd National Foundation for Science and Technology Development Conference on Information and Computer Science

II. PROPOSED FRAMEWORK ß Initialize the sender:

Fig. 1 describes the proposed framework, which has 5 main rtmp://192.168.0.110/live/stream
components: Encoder/Sender module, Receiver/Decoder
module, Network Emulator module, Server module and Video ß Encode and send to server:
Quality Assessment module. ffmpeg -i input.mp4 -c:v libx264 -f flv
First, at the Encoder/Sender, the input video is encoded into rtmp://localhost/live/stream
a bit stream. Then the bit stream is chopped into packets and ß Receive and decode:
sent to the Server. After that, the Server continues streaming
the packets through the Network Emulator, which emulates a ffmpeg -re -y -i rtmp://localhost/live/stream live=1
plenty of network issues in the real world such as propagation output.flv

Fig. 1. Block diagram of the proposed framework

delay, packet loss, available bandwidth and multipath effect.

After going through the Network Emulator, the packets are
received and decoded into an output video by the
Receiver/Decoder. Then, the quality of the output video is
evaluated by comparing it to the input video at the Video
Quality Assessment module. The System Controller will be in
charge of controlling the whole system by sending some trigger
signals to inform which component to start and when to start
and handling errors if occur.
Detail of each module in the proposed framework will be
explained in the following sections. Section III discusses about
the Encoder/Sender module and Receiver/Decoder module.
Section IV mentions about the Network Emulator module.
Next, section V describles the tool used for Server-System
Controler module. Finally, section VI will detailize the Video
Quality Assessment Module.
III. ENCODER/SENDER – RECEIVER/DECODER
FFMPEG [6] is used to create both the Encoder/Sender and
the Receiver/Decoder at the two ends of the streaming system.
The reason FFMPEG is chosen because in fact it is a non-
charged and powerful software with a cross-platform solution
to record, convert and stream audios/videos. Besides, FFMPEG
supports many video formats and codecs as well as all the
common video streaming protocols such as: RTSP,
RTP/RTCP, RTMP, HTTP ...
At the Encoder/Sender, the input video is first encoded into
a bit stream. Then the bit stream is chunked into packets and
sent out to the Server. At the Receiver/Decoder, similarly,
FFMPEG does the inverse process of receiving the packets,
and decoding them into the video.
One example of the command to initialize the stream;
encode/send and receive/decode is
IV. NETWORK EMULATOR
There are plenty of tools and software for network
emulation. In this paper, a non-charged but powerful tool
called DUMMYNET [7] is selected due to its convenience and
very simple usage along with the capability of running on a
variety of operating systems.
The basic concept of DUMMYNET is to intercept the
network stack with an object called "pipe". All packets going
from the client to the server and vice versa have to enter these
"pipes" before reaching the internet. With these "pipes", we can
implement a variety of network attributes, such as propagation
delay, random packet loss, available bandwidth and multipath
effect.
Two steps for setting up network environments using pipes:
ß Step 1: Create pipes with parameters
ipfw pipe <number> config delay <var1>ms bw
<var2>Kbit/s plr <var3>
<number>: pipe identification.
<var1>, <var2>, <var3>: input parameters setting up
for delay, bandwidth and packet loss rate, respectively.
ß Step 2: Add pipe into the network stack
ipfw add prob <var4> pipe <number> proto
<protocol><down-up>
<var4>: the probability for packet flows go through
this pipe.
<number>: pipe identification.
<protocol>: protocol in the transport layer, TCP, UDP
or ICMP…

215
 2015 2nd National Foundation for Science and Technology Development Conference on Information and Computer Science

<down-up>: DUMMYNET defaults for both run jobs. More details on JENKINS as well as on the
downlink and uplink packets, or use in for incoming packets, framework establishment will be discussed in section VII.
out for uplink packets.
VI. VIDEO QUALITY ASSESSMENT MODULE
To remove pipe, the following commands can be used
Numerous video quality assessment methods and metrics
ß Remove pipe <number> have been proposed over the past years with varying
ipfw –q pipe <number> flush computational complexity and accuracy. Based on the
availability of the original and received videos, for this study,
ß Remove all pipes: two traditional full-reference video quality metrics are chosen:
ipfw –q flush PSNR (Peak Signal to Noise) and SSIM (Structural Similarity
Index Metric) for assessing the quality of received videos
Below is some examples for network emulator comparing to the original videos from the sender. Here we just
ß Delay: 50ms, TCP, downlink packets review the formulas for PSNR and SSIM computation, please
have a look at [10] and [11] for more details and specific
ipfw pipe 1 config delay 50ms implementation.
ipfw add pipe 1 proto tcp in The formula for computing PSNR:
ß Loss: Packet loss rate: 0.1 %, UDP, uplink packets
ipfw pipe 2 config plr 0.1
ipfw add pipe 2 proto udp out
ß Bandwidth: Bandwidth management: 200Kbit/s, ICMP, where are the pixel values of two frames, one
both uplink and downlink packets from the input video and the other from the output video. M
ipfw pipe 3 config bw 200Kbit/s and N are the horizontal size and the vertical size of the frame,
respectively. And n is the bit-depth, the number of bits
ipfw add pipe 3 proto icmp represent a pixel.
For two non-negative image signals x and y, the formula
V. SERVER – SYSTEM CONTROLLER for computing SSIM is:
Every video streaming system needs a streaming server.
For recent years, the Adobe Flash Media Server [8] has
emerged as a powerful and free software for providing where are the mean intensity of x, the mean
excellent functionality to create online video services. It intensity of y, the standard deviation of x, the standard
supports a plenty of live streaming protocols such as RTMP / deviation of y and the covariance between x and y,
RTMPE (Flash), HTTP DS (Flash), and HTTP LS (iOS). respectively. are the calibration constants.
Moreover, it also provides of high-performance multi-format
content output to the users and runs of additional content VII. EXPERIMENTS
protection services. Because of those reasons, we decided to
In this section, the set-up of our proposed framework and
use the Adobe Flash Media Server as the Server in our report measured its performances under some network
proposed framework and RTMP as the streaming protocol.
conditions are described in details. Fig. 2 shows the proposed
The Real-Time Messaging Protocol (RTMP) is used for framework set-up.
streaming multimedia content across a TCP/IP network. We
PC3
use ffmpeg to send and receive video.
Syntax of a command is:
rtmp://[username:password@]server[:port][/app][/
rtmp_live]
- Username: an optional username
- Password: an optional password
- Server: the address of the RTMP server.
- Port: the number of the TCP port to use (by
default is 1935). PC1 PC2
- App: the name of the application to access
- rtmp_live: live stream specification
For the System Controller, a tool called JENKINS [9] is
used. JENKINS is an award-winning application that monitors Fig. 2. The proposed framework set-up
executions of repeated jobs, such as building/testing software
projects continuously and monitoring executions of externally-

216
 2015 2nd National Foundation for Science and Technology Development Conference on Information and Computer Science
A. Framework Set-up
Three computers are used for the set-up. PC1 plays the role
of the Encoder/Sender as sending the input video out to the
Server. The Network Emulator is also installed in PC1 instead
of PC2, not like the illustration in the first place in Figure 1 for
getting some work load off PC2.
Fig. 3. Parameter setting for a muilti-job test
Fig. 4. JENKINS dialogue for spatial test
The Network Emulator, which is DUMMYNET, act as a
bridge between the Encoder/Sender and the streaming Server.
PC2 contains the module of Receiver/Decoder receiving the
incoming streaming video from the Server. The Video Quality
Assessment (VQA) module is also set up in PC2. PC3 plays
two roles as both the Adobe Flash Media Server in charge of
handling the streaming from PC1 and the JENKINS Server for
managing the whole system. JENKINS Server creates 3 jobs
to conduct the above system. The first job is responsible for
217
sending video from PC1 to PC3. The second one's task is to
activate The Network Emulator in PC1. The third job is for
PC2 to receive the video streaming from PC3 and then run the
VQA module afterward. All of the above jobs are managed
and automatically triggered by JENKINS Server. Fig. 3 shows
the parameter setting for a multi-job test and Fig. 4 shows the
JENKINS dialogue for spatial test in the simulated test frame.
B. Framework Measurement Results
Due to the main concern about the performance of video
conferencing systems, many different test cases are simulated
to investigate the performance of our framework. The choice
for the input videos is many head-and-shoulder videos
encoded at two different bitrates. One for high quality
compression is with target bandwidth of 500kbps. The other
for lower quality compression is with target bandwidth of
100kbps. For network emulation, two cases are considered
 2015 2nd National Foundation for Science and Technology Development Conference on Information and Computer Science

with available bandwidth and propagation delay. For the system. This framework can also be used to evaluate the
available bandwidth, two cases with bandwidth of 400kbps performances of video-streaming-based applications.
and 200kbps are investigated. For the propagation delay, two In the future, more test cases should be examined on the
cases with propagation delay of 150ms and 50ms are framework with a great variety of cases on network issues and
considered. The results are shown in the Table I. below for different types of input videos as well as with other streaming
both PSNR in dB and SSIM. protocols. Besides, for this study, the VQA module is still
rather simple. Only two popular full-reference metrics such as
TABLE I. MEASUREMENT RESULTS OF THE PROPOSED FRAMEWORK PSNR and SSIM are implemented for evaluating the output
videos. In future, the VQA module will be extended with more
Video Quality Metrics efficient and powerful metrics not only on spatial metrics but
Video
Bitrate on temporal metrics as well.
Network Test Cases
PSNR SSIM ACKNOWLEDGEMENT
This work was done during the time the authors were at
500kbps 35.105 0.94443 Personify Inc.
400kbps REFERENCES
100kbps 26.4432 0.80223
[1] An (Jack) Chan, Amit Pande, Eilwoo Baik and Prasant Mohapatra,
Available “Temporal Quality Assessment for Mobile Videos”, MobiCom’12,
Bandwidth 500kbps 35.105 0.94443 August 22–26, 2012, Istanbul, Turkey.
[2] Nick Feamster And Hari Balakrishnan, “Packet Loss Recovery For
200kbps Streaming Video”, 12th International Packet Video Workshop,
100kbps 26.4432 0.80223 Pittsburgh, PA, April 2002
[3] Kanumuri, S.; Cosman, P.C.; Reibman, A.R.; Vaishampayan, V.A.,
"Modeling packet-loss visibility in MPEG-2 video," Multimedia, IEEE
500kbps 35.105 0.94443 Transactions on , vol.8, no.2, pp.341,355, April 2006
doi: 10.1109/TMM.2005.864343
150ms [4] Q. Huynh-Thu and M. Ghanbari. Scope of validity of PSNR in
100kbps 26.4432 0.80223 image/video quality assessment. Electronics Letters, 44(13):800 –801,
Propagation 19 2008.
Delay [5] Z. Wang, L. Lu, and A. Bovik. Video quality assessment based on
500kbps 35.105 0.94443 structural distortion measurement. Signal Processing: Image
Communication, 19(2):121–132, February 2004.
50ms
100kbps 26.4432 0.80223 [6] https://www.ffmpeg.org/
[7] http://info.iet.unipi.it/~luigi/dummynet/
[8] https://www.adobe.com/support/flashmediaserver/downloads_updaters.h
Apparently, based on Table I above, it can be seen that the tml
network characteristics do not affect the quality of the output [9] https://jenkins-ci.org/
videos. All the PSNR values of the output videos with the [10] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli, “Image
quality assessment: From error measurement to structural similarity,”
same bitrate are similar. The same thing happens to the SSIM IEEE Trans. Image Processing, vol. 13, Jan. 2004.
values. The reason is because of the chosen streaming [11] S. Chikkerur, V. Sundaram, M. Reisslein and L. J. Karam, “Objective
protocol, which is RTMP. RTMP is a TCP-based protocol, Video Quality Assessment Methods: A Classification, Review, and
which obviously has flow control and error control. Therefore, Performance Comparison”, IEEE Trans. Broadcasting, vol. 57, No. 2,
June 2011.
the output videos almost have their best forms comparing to
[12] Winkler, Stefan, and Praveen Mohandas, "The evolution of video quality
the input videos. The only thing that affects the quality of measurement: from PSNR to hybrid metrics", IEEE Trans.
service here is just the video bitrate. Further check with the Broadcasting, vol. 54, No. 3, pp. 660-668, 2008.
PSNR and SSIM at the sender verifies the robustness of the [13] Piamrat, Kandaraj, et al., "Quality of experience measurements for video
PSNR and SSIM values calculated at the receiver using the streaming over wireless network.", Information Technology: New
proposed Video Quality Assessment (VQA) module. Generations, 2009. ITNG'09. Sixth International Conference on. IEEE,
2009.
VIII. CONCLUSIONS AND FUTURE WORKS [14] Seshadrinathan, Kalpana, et al. "Study of subjective and objective
quality assessment of video", IEEE Trans. Image Processing, vol. 19,
In this paper, a novel framework is proposed for simulating No. 6, pp. 1427-1441, 2010.
the real-world video streaming system. With this framework, [15] Chan, An Jack, et al., "Metrics for evaluating video streaming quality in
it can simulate a video streaming system in a real transmission lossy IEEE 802.11 wireless networks", INFOCOM, 2010 Proceedings
IEEE. IEEE, 2010.

218