IEEE International Conference on Wireless & Mobile Computing, Networking & Communication
A Design of Experiment (DOE) Analysis of the Performance of Uplink Real-Time
Traffic over a 3G Network
Ana Goulart and Wei Zhan
Department of Engineering Technology and Industrial Distribution, Texas A&M University
Abstract — The bandwidth and reliability of the reverse
link (or uplink) as the performance metrics of real-time
traffic are studied in this paper. A two-level full factorial
Design of Experiment (DOE) was conducted to identify the
impact on the bandwidth and reliability of four factors:
packet size, data streaming rate, time of the day, and buffer
size in a real network environment. Two extreme cases for
each factor were tested and statistical analysis is performed
to characterize how different factors and their interactions
may affect the bandwidth and reliability. The importance of
this study is illustrated in the optimal channel selection
mechanism where appropriate packet size, data streaming
rate and buffer size can be chosen to achieve the best result.
1. Introduction
The performance of real-time traffic over third-generation
(3G) cellular networks is a subject of growing interest
among network providers, application developers, and
end users in general. Bandwidth of the communication
system is a limited resource and can be quite different
from the advertised bandwidth. Moreover, reliability is
extremely important for mobile users exchanging critical
real-time applications over 3G cellular networks using the
UDP transport protocol.
For instance, digital ambulances, such as the DREAMs
ambulances [3], exchange high-quality video, vital signs,
voice and text communications in real-time with the
emergency room. By taking advantage of the widespread
cellular coverage in rural and urban areas, the ambulance
gets connectivity on its way to the hospital so that the
doctors can communicate in audio and video with the
emergency medical technicians (EMTs) and have
immediate access to the vital signs of the patient. To
978-0-7695-3393-3/08 $25.00 © 2008 IEEE
DOI 10.1109/WiMob.2008.110
Authorized licensed use limited to: Cork Institute of Technology. Downloaded on September 30, 2009 at 07:44 from IEEE Xplore. Restrictions apply.
achieve this, the ambulance uses a system of several 3G
wireless cards with different technologies and different
network providers [4]. Typically, there are four to six
wireless cards in the ambulance. As the ambulance
moves at high speeds, the pool of channels in use at a
given time varies.
In [5], the problem of optimally selecting the best channel
among the pool of channels available to the
communication system in the DREAMs ambulances was
discussed using modeling and simulation tools. However,
many parameters such as packet size and buffer size in the
optimal channel selection scheme can take different
values, which may affect the selection result. It is
therefore important to understand how different factors
(system and environmental) affect the user’s performance.
This is determined mainly by the uplink transmission
characteristics since the main source of real-time data is
the mobile node.
There are several studies on the performance of real-time
applications over 3G systems. However, they covered
different aspects, such as downlink transmissions [6, 8],
older wireless technologies [6, 7, 8], TCP traffic [12],
and/or simulation or test bed results [11,13]. Some of the
ideas in these papers such as performance evaluation may
be similar to our work, but there are significant
differences. For the uplink transmission of the UDP
transport protocol, the end-user does not have access to all
the information about the network, with limited amount of
feedback to the end user. This makes the on line
performance evaluation of the uplink a much more
challenging task and more critical to the overall system
optimization. The production network based on
CDMA/HDR (High Data Rate) [1] used for the
466
experiments also may have different characteristics
compared to other system or simulation models.
For any given wireless communication system, there are
many factors that impact the performance of the uplink
transmission, e.g., packet size, buffer size, data streaming
rate, location of the end user, weather condition, time of
day, etc. They can be divided into two groups: one is the
control parameters, and the other is the noise parameters.
A control parameter is one that can be set to certain value
by the end user. A noise parameter is one that the end user
has no control. The optimization of the system
performance is to select the control parameters such that
for any values of the noise parameters certain objective
function involving the uplink transmission rate and the
loss rate is maximized. To this end, it is necessary to
understand how each control parameter affects the uplink
transmission rate and the loss rate for different values of
the noise parameters. In this paper, a statistical analysis
tool called Design of Experiment (DOE) [10] is used to
analyze the impact of relevant variables on the
performance of the end user’s applications.
In DOE, certain combinations of extreme cases of each
variable are targeted for testing, with each experiment
repeated several times. The resulting uplink bandwidth
and loss rate are used to identify the impact of each
variable and the interactions between variables. This is
accomplished with the help of a statistical tool such as
Minitab. In this paper, a two-level full factorial DOE is
used. Two-level means for each variable two extreme
cases are tested. Full factorial means all combinations of
the extreme cases for each variable are tested. Therefore,
if there are N parameters, the total combinations will be
2N. To reduce the impact of random noise associated with
the testing, each experiment in DOE is repeated several
times and the average result will be used for analysis.
This paper is organized as follows: the experimental setup
for DOE is discussed in Section 2; Section 3 has the
analysis of the DOE results; Section 4 presents our
conclusions and future work.
2. Experimental Setup
In the DOE analysis, the network is considered as a
“black box”. The only known factor from the network is
the coverage map provided by the network provider
467
Authorized licensed use limited to: Cork Institute of Technology. Downloaded on September 30, 2009 at 07:44 from IEEE Xplore. Restrictions apply.
(Figure 1). A location of average (or “good”) coverage is
chosen for the experiments in College Station, Texas, a
city with a population of about 60,000.
The card used in our tests is a Sierra Wireless AirCard
597A, which is a 3G wireless data card based on
CDMA/HDR [1]. This technology is also known as an
evolutionary advancement for CDMA, and when used for
data only is called EV-DO. EV-DO systems were
designed for client-server Internet applications, such as
web browsing and file transfer. Thus, the forward and
reverse links were designed to provide asymmetric data
rates, with the bandwidth in the forward (i.e., downlink)
higher than in the reverse links (i.e., uplink). According to
[1], we can expect the downlink capacity to be three or
four times greater than the uplink capacity for data
applications.
The tests are performed using a network performance tool
called Iperf [4]. To emulate real-time traffic, the
experiments are performed using UDP traffic, at constant
bit rate. The reason for assuming constant bit rate is that
we are mainly interested in voice and remote monitoring
traffic. As shown in Figure 2, using the mobile station as
the client (i.e., the mobile station is the source of packets)
and a fixed Internet node as the server, the actual
bandwidth over a period of 10 minutes is measured in
each experiment, with the Iperf statistics generated every
2 seconds.
Experiments
Location
Best Good Fair
Figure 1. Location of experiments.
 Reverse channel Due to the fact that different routes provided even more
UDP packets (uplink) random results, the variable “location” is isolated in our
3G Core
Iperf Network tests, i.e., all experiments are tested in a fixed, indoor
3G location. Although this scenario has limitations for the
Data Card target application of the ambulances, it helps us to
Client (mobile station)
perform the DOE analysis more effectively.
Internet

Based on the above consideration, the DOE analysis is

narrowed down to four variables: packet size, data rate,
time of the day, and buffer size, with the bandwidth and
reliability used as the outputs. The resulting two-level full
Server factorial DOE is defined by Tables 1 and 2. With four
(fixed node) Iperf
variables, the full factorial DOE consists of 16
experiments (Table 2). Each experiment is repeated three
Figure 2. Setup for the assessment of uplink traffic.
times to reduce the effect of randomness, which leads to a
total of 48 experiments.
800 800 800
Table 1. DOE factors
Bandwidth (kbps)

600 600 600

Variable Value =1 Value = -1
400 400 400
Packet Size 100 bytes 500 bytes
200 200 200
Streaming Rate 1 Mbps 100Kbps
0 0 0
0 500 0 500 0 500 Time 6 am 6 pm
Buffer Size 100kbytes 0 bytes
2 40 100
Pkt Loss Ratio (%)

80
1.5 30
60 Table 2. Factorial DOE experiments
1 20
40
Mean
0.5 10
20 Packet Streaming Buffer Mean packet
0 0 0 Size Rate Time Size Band loss
0 500 0 500 0 500
(bytes) (kbps) (Kbytes) width rate
Figure 3. Experiments with a moving node for the same (kbps) (%)
route with packet size of 500 bytes. 1 500 100 6 pm 0 90.00 0.47
2 100 100 6 pm 0 83.05 14.44
Design of Experiment: The initial focus of the 6 pm 0
3 500 1000 121.22 0.37
experiments was to evaluate the bandwidth and reliability 1000 6 pm 0
4 100 112.49 16.49
of mobile users, such as the DREAMS ambulance [3]. 100 0
5 500 6 am 46.82 0.34
However, it was found out that the moving results were 100 6 am 0
6 100 77.15 16.42
very random and difficult to analyze. For a given route, 6 am 0
7 500 1000 156.10 0.37
three measurements were taken, under the same weather 1000 6 am 0
8 100 68.50 13.35
conditions, same speed of the vehicle, same time of the 100 100
9 500 6 pm 99.31 0.19
day, same packet size, same buffer size, and same data 100 6 pm 100
10 100 81.33 0.31
streaming rate. Figure 3 shows the bandwidth and 6 pm 100
11 500 1000 165.61 0.19
reliability results of one of these experiments. The x-axis 1000 6 pm 100
12 100 61.47 1.22
represents time in seconds; each test took 600 sec, i.e., ten 100 100
13 500 6 am 78.40 0.53
minutes. Note the variation in the bandwidth among the 100 6 am 100
14 100 99.61 0.03
three tests. As for the reliability, the packet loss rate 6 am 100
15 500 1000 200.50 0.09
results show different behaviors, with the first test being 1000 6 am 100
16 100 150.03 0.06
the most reliable one.

468

Authorized licensed use limited to: Cork Institute of Technology. Downloaded on September 30, 2009 at 07:44 from IEEE Xplore. Restrictions apply.
Note from Table 1 that small packet sizes were chosen in bandwidth. Of course, these conclusions are limited to the
the experiments. For instance, packet sizes can range from ranges between the -1s and 1s specified in Table 1.
less than 100 bytes for G.723.1 audio codecs, to close to
500 bytes for G.711 codecs. The time of the day ranged
from a time in which we assumed there is a small number
of users in the network (6 a.m.) to a rush hour time when
it is likely that there are more users competing for the
network resources (6 p.m.). The data rates are also chosen
as two extremes: data streaming rates at 1 Mbps are likely
to be more than the uplink transmissions can handle,
while 100 kbps represents a more reasonable value.
Finally, an extremely small buffer size (0 bytes) and an
extremely large one (100 Kbytes) are used. Although such
large buffers are not used in real-time applications, they
help us evaluate the effect of bursty packet losses
independently of the size of the packets transmitted in the
experiments.

3. Results of DOE analysis

Figure 4. Pareto Chart for bandwidth.

Based on the Design of Experiments, the means of the

bandwidth and packet loss rate are shown in Table 2. The
experimental results in Table 2 are analyzed using
Minitab to plot the Pareto Chart of the Effects, the Main
Effects, and the Interaction Plots for bandwidth and
packet loss rate.

Bandwidth -- There are different graphs that can be used

to illustrate the DOE analysis results. The Pareto Chart of
the Effects for bandwidth is shown in Figure 4. With a
confidence level of 95%, which is 1 - Alpha, the
streaming rate and the interaction between the packet size
and the streaming rate are identified as the significant
factors contributing to the bandwidth, as indicated by the Figure 5. Main Effects for bandwidth.
corresponding values for these factors over the threshold
of 2.571.
However, the Pareto Chart does not have enough detail on
how the main factors affect the bandwidth. Figure 5
shows the Main Effects Plot for bandwidth, where only
the four factors are considered without the interactions
between the factors. Recall that the -1 and 1 are the two
extreme cases defined in Table1. When each factor is
changed from -1 to 1, the Main Effects Plot shows that the
change of the streaming rate results in the largest change
in the bandwidth, which is expected. Packet size has the
second most significant impact on the bandwidth. The
Figure 6. Interaction Plot for bandwidth.
time of day has the least significant impact on the

469

Authorized licensed use limited to: Cork Institute of Technology. Downloaded on September 30, 2009 at 07:44 from IEEE Xplore. Restrictions apply.
The Main Effects Plot does not take interactions between
factors into consideration, which can lead to wrong
conclusions sometimes. Figure 6 shows the interactions
between all the factors. On the top of the plot, the values
-1 and 1 are indicated for each factor labeled in the
bottom. The two lines in each square correspond to the -1
and 1 value for the factor as identified on the right hand
side of the plot. The difference in the slopes of these two
lines is a measure of the interaction between the two
factors, same slope meaning no interaction and drastically
different slopes meaning strong interaction. For instance,
even though the two lines for the interaction between
streaming rate and buffer size indicate very different
results when the factors take the values of -1 and 1, the
slopes of these two lines are very close. Therefore, the
interaction between the streaming rate and the buffer size
is weak. On the other hand, the interaction between the
Figure 7. Pareto Chart for packet loss rate.
packet size and the streaming rate is strong because the
slope change is significant.

Reliability -- Similar analysis can be done for the packet

loss rate, which is a metric for reliability of the uplink
transmission. The Pareto Chart, the Main Effect Plot, and
the Interaction Plot for packet loss rate are shown in
Figures 7, 8, and 9, respectively.

From the Pareto Chart, with confidence level of 95% one

can conclude that the main factors for the loss rate are
packet size, buffer size, and the interaction between them.
The same conclusion obtained from the Pareto Chart can
be drawn from the Main Effects Plot: larger packet (-1)
and larger buffer (1) minized the packet loss rate. Also,
one can observe the experiments 1 through 8 (Table 2), Figure 8. Main effects for packet loss rate.
where the 100-byte packet transmissions showed higher
mean packet loss rates than that of the 500-byte packets.

The Interaction Plot reveals that the only strong

interaction is between the packet size and the buffer size.
Note that time of day and data streaming rate had small
influence on packet loss rate when compared to the
influence of packet size and buffer size.

4. Conclusions

This paper presents an innovative way to study the

simultaneous effects of different variables on the
throughput and packet loss rate using the DOE method.
Figure 9. Interaction Plot for packet loss rate.

470

Authorized licensed use limited to: Cork Institute of Technology. Downloaded on September 30, 2009 at 07:44 from IEEE Xplore. Restrictions apply.
The results help one to better understand the impact of
packet size, buffer size, data streaming rate, and the time
of day for uplink real-time transmission in 3G wireless
communication systems.
For example, based on the DOE analysis and the choice
of DOE parameters from Table 1, the reliability is
strongly influenced by the choice of buffer size and
packet size. The larger packet size (500 bytes) showed to
be more resilient to packet losses than the smaller packet
(100 bytes). The bandwidth is strongly influenced by the
data streaming rate and its interaction with packet size,
whereas time of the day had little influence.
[10] Meyer, R. and Krueger, D. “A Minitab guide to statistics,”
3rd Edition, Prentice Hall, 2005.
[11] Jalali, A. Padovani, R. and Pankaj, R. “Data throughput of
CDMA/HDR a high efficiency-high data rate personal
communication wireless system, In the Proc. of the IEEE
51st Vehicular Technology Conference, Vol. 3, pp. 1854-
1858, 2000.
[12] Lee, Y. “Measured TCP performance in CDMA 1x EV-DO
network,” In the Proc. Of the Passive and Active
Measurement Conference, 2006.
[13] Qi Bi, Vitebsky, S., “Performance analysis of 3G-1X
EVDO high data rate system,” In the Proc. of the Wireless
Communications and Networking Conference (WCNC),
2002.

In our future research work, a multi-level factorial DOE

can be deployed to evaluate additional values of packet
size, data streaming rate, and buffer size to derive a model
with more detailed information. Future work will also
include a DOE analysis using a different wireless service
provider and the implementation of the optimal uplink
channel selection [5] using the DOE analysis results.

REFERENCES

[1] Bender P., Block P., Grob M., Padovani R., Sindhushayana
N., and Viterbi A. “CDMA/HDR: A bandwidth-efficient
high speed wireless data service for nomadic users.” IEEE
Communications Magazine 2000; 8(7): 70-77.
[2] Iperf, http://dast.nlanr.net/projects/Iperf.
[3] Ewing R.E., Flournoy L., and Wall J., “Texas A&M
University Disaster Relief and Emergency Medical
Services (DREAMS).” Technical Report, November 2006.
[4] Rodriguez P., Chakravorty R., Chesterfield J., Pratt I., and
Banderjee S. “MAR: A Commuter Router Infrastructure for
the Mobile Internet.” In the Proc. of ACM MobiSys’2004.
[5] Sahai G., Goulart A., Zhan W., and Arnold R. “Optimal
Selection of Wireless Channels for Real-Time
Communication in Ambulances,” In the Proc. of the IEEE
Radio and Wireless Symposium, Jan 2008.
[6] Alexiou A., Bouras C., and Igglesis V. “Scalable rate control
for video transmission over UMTS.” International Journal
of Communication Systems, 2007; 20(12): 1315-1335.
[7] Basso, A., Kim, B-J., Jiang, Z., “Performance evaluation of
MPEG-4 video over realistic EDGE wireless networks.” In
the Proc. of the 5th International Symposium on Wireless
Personal Multimedia Communications, Vol. 3, pp.1118-
1122, October 2002.
[8] Schieder, A., Horn, U., and Kalden, R. “Performance
analysis of realtime applications in mobile packet switched
networks.” ITG-Fachbericht, European Wireless’99, 1999.
[9] Dulski, A., Beijner, H., and Herbertsson, H. “Rural W-
CDMA – Aiming for nationwide coverage with one
network, one technology, and one service offering.”
Ericsson Review – The telecommunications journal, Vol. 2,
2006.

471

Authorized licensed use limited to: Cork Institute of Technology. Downloaded on September 30, 2009 at 07:44 from IEEE Xplore. Restrictions apply.