Cross-Validation of DSRC Radio Testbed and NS-2
Simulation Platform for Vehicular Safety Communications
Gaurav Bansal, John B. Kenney
Toyota InfoTechnology Center, USA
[gbansal,jkenney]@us.toyota-itc.com
Abstract—One of the most important challenges for DSRC
deployment is maintaining high performance under heavy
channel load. The study of congestion control mechanisms will be
impractical if it requires hundreds of physical radios deployed on
vehicles. In this paper we demonstrate two important results that
together suggest a strategy for investigating congestion control.
First, we show that the NS-2 simulator accurately models the
MAC and PHY associated with an IEEE 802.11 hardware
implementation. This opens the door to using simulation as the
primary investigative tool. Second, we show that a technique in
which one radio emulates N vehicles can produce results quite
similar to the case in which a larger number of distinct vehicles
exist. This will allow selective validation tests with tens of physical
radios emulating hundreds of vehicles. The means by which we
demonstrate these results is a cross-validation technique. We
show that three test approaches produce consistently similar
results: physical radios with emulation, NS-2 with emulation, and
NS-2 without emulation. The close agreement among these results
validates both the radio emulation technique and the NS-2
implementation of the 802.11 MAC and PHY protocols. We use
this cross-validation technique to compare results for a wide
variety of transmit power, message rate, data rate, channel
bandwidth, message size and EDCA settings.
Keywords-DSRC, NS-2, IEEE 802.11, WAVE, Radio Testbed
I. INTRODUCTION
The US Federal Communication Commission (FCC) has
allocated 75 MHz of spectrum in the 5.9 GHz band for
Dedicated Short Range Communication (DSRC) among
vehicles, and between vehicles and roadside infrastructure [1, 2].
The primary purpose is to enable safety applications that can
prevent accidents. DSRC radio technology is standardized in
IEEE 802.11p [3, 4], IEEE 1609.3 [5], and IEEE 1609.4 [ 6].
In [7, 8, 9] techniques have been proposed to perform
congestion control in the DSRC safety channel. However the
existing work does not include radio tests to demonstrate the
severity of DSRC congestion. To study the congestion
performance, we configured a testbed with DSRC radios in a
laboratory setting. The purpose of the tests was to model the
channel loading in a stationary environment in which a large
number of vehicles (around 200) are transmitting at a nominal
rate (10 Hz). Further, as it is difficult to assemble 200 actual
radios for experiments, we developed an emulation technique
where each physical radio is used to emulate N vehicles by
transmitting at a higher rate. For example, channel loading
equivalent to 150 vehicles transmitting at 10 Hz was created by
978-1-4244-8327-3/11/$26.00 ©2011 IEEE
Aaron Weinfield
Denso International America Inc., USA
aaron_weinfield@denso-diam.com
using 30 radios with each transmitting at 50 Hz, using a
technique described in Section II. The tests were conducted for
various data rates, message sizes, Enhanced Distributed
Channel Access (EDCA) parameters, channel bandwidths,
transmit powers, and numbers of emulated vehicles.
Network Simulator 2 (NS-2) is a discrete event network
simulator, which is widely cited in wireless communication and
ad-hoc networking research. Version NS-2.33, released in 2008,
added improvements to make the simulator compatible with the
IEEE 802.11p standard MAC and PHY protocols for Wireless
Access in Vehicular Environments (WAVE) [10]. The key
modifications include cumulative SINR computation, capture,
and more accurate modeling of the IEEE 802.11 CSMA/CA
mechanism. In [15] the authors identify certain weaknesses of
NS-2, and they suggest some configuration options that might
allow it to perform on par with commercial simulator packages
for the special case of fixed length packets. However, we are
not aware of any study comparing NS-2 with real DSRC radios.
Testing with physical radios and testing with simulated radios
each have advantages compared to the other. For example,
physical radios provide a more realistic measure of performance
with respect to channel fading, frame capture, and vehicle
mobility in real time. On the other hand, simulations are much
more efficient when testing large networks with many system
variables. We believe there is a role for each in developing
solutions for DSRC channel congestion, with the bulk of testing
carried out via simulation and with selective use of radio testbed
experimentation.
We recreate a radio testbed topography in the NS-2 simulator.
When modeling M radios we use two techniques: a) M distinct
simulated nodes, and b) 30 distinct nodes each of which
emulates N = M/30 vehicles by transmitting at N times the
vehicle message rate. We then address two issues. First, we use
the simulator without emulation to validate the accuracy of the
radio emulation technique in the testbed. Second, we use the
radio tests to validate the accuracy of the NS-2 MAC and PHY
model. In this way, we use the radios and simulator to
cross-validate each other.
In this paper we show that performance results obtained by
NS-2 simulation, both with and without emulation, tend to
follow the radio results quite closely and in a consistent manner.
This implies that the goal of cross-validation is achieved, i.e. the
NS-2 simulator gives a good estimate of the performance results
of actual radios and the physical radio emulation technique
provides an accurate model of the congestion performance that
can be expected from a large number of vehicles. These results
show that testing of congestion scenarios can proceed on two
complimentary fronts: a) reasonably scaled tests using tens of
physical radios can emulate the performance of hundreds of
radios, and b) simulations can be used to assess various
algorithms, protocols, and parameters without incurring the
complexity of a physical radio test each time.
The paper is organized as follows. In Section II, we describe the
laboratory environment of the tests conducted with actual radios.
The NS-2 configuration is described in Section III. We present
comparative results of NS-2 simulations and radio tests in
Section IV. Finally the paper is concluded in Section V.
II. LABORATORY TESTS CONDUCTED WITH FIELD RADIOS
The radio testbed was constructed by placing DSRC radios
physically close to each other in an indoor office environment,
with 15 dual-radio DENSO wireless safety unit (WSU) systems
on the periphery of a rectangular area roughly 20 feet by 10 feet,
as shown in Fig. 1 [11]. While these laboratory tests were
conducted in indoor environments, similar performance has
been reported for outdoor environments as well [12]. To better
model the normal vehicle density, each radio antenna was
configured with an attenuator between 0 dB and 50 dB with a
goal of distributing the Receive Signal Strength (RSS) over a
range of -30 to -95 dBm at each receiver. Attenuating a radio
antenna affects both transmit and receive paths. Therefore, a
few radio pairs with high amounts of attenuation were
completely out of range (i.e. “hidden”) from each other, but still
able to communicate with radios having lower attenuation. In
each dual-radio WSU both radios were configured to transmit
and only one radio was configured to receive, so there were a
total of 28 x 15 = 420 transmitter-receiver pairs, or links
(excluding the radios in the same WSU from the calculation).
Of these, 8 links were classified as out of range. A histogram of
the link RSS values is shown in [11].
WSU 7 WSU 8
WSU 4 WSU 5 WSU 6 WSU 9 WSU 10
~ 8 feet
WSU 3
~ 10 feet
WSU 2
~ 20 feet
WSU 1 WSU 15 WSU 14
Fig 1: Physical WSU Layout in the Laboratory Testbed, each
WSU has two DSRC radios
A total of 101 tests were conducted for various combinations of
the following parameters: data rate (6 Mbps, 12 Mbps),
over-the-air (OTA) message size (300 bytes, 378 bytes, 464
bytes), EDCA parameters (both IEEE 1609.4 trial-use [13] and
IEEE 802.11p [4] defaults for access category indices AC0,
AC1, AC2, AC3), channel bandwidth (10 MHz channel 172, 20
MHz channel 175), transmit power (10 dBm, 20 dBm), number
of emulated vehicles (30, 60, 90, 120, 150, 180, 210, 240, 270,
300, 330, 360) and emulated vehicle message rate (5 Hz, 10 Hz).
III. NS-2 SIMULATOR CONFIGURATION
We attempted to configure the NS-2 simulator to model the
radio testbed environment as accurately as possible. Our
emphasis was on MAC collision modeling in the congested
network regime. For the PHY, we found that the simple Rician
fading model works quite well. It has a dominant line-of-sight
(LOS) component. The nodes in NS-2 were placed in the same
physical topology as in the radio testbed. We implemented the
attenuation of radios at the transmitter side by decreasing the
transmit power and at the receiver side by increasing the values
of carrier sense threshold and reception sensitivity threshold.
We implemented simulations with and without emulation. In an
emulated scenario we had 30 nodes transmitting. For example,
if each of the 30 nodes emulated five vehicles transmitting at 10
Hz, the emulated node transmitted at 50 Hz. The corresponding
non-emulated scenario used 150 nodes transmitting at 10 Hz,
with five nodes co-located at each physical radio location. The
carrier sense threshold value for 6 Mbps data rate was set to -92
dBm, and for 12 Mbps data rate it was set to -89 dBm. The
reception sensitivity threshold values are taken to be same as
carrier sense threshold values. These values were all determined
by calibration tests with the radio testbed. The noise floor was
set to -99 dBm in 10 MHz channel and -96 dBm in 20 MHz
channel. The preamble capture feature has been enabled with a
SINR capture threshold of 4 dB.
Some of the differences between the radio testbed and the NS-2
simulator were as follows:
1) Generation Time: This subsection describes how
messages are generated in the testbed and in NS-2. In NS-2
without vehicle emulation, when a vehicle generates a message
at time t it also schedules its next message at time t+T. If the
transmit rate is X Hz, then T is a uniform random variable over
the interval [1/2X, 3/2X]. In NS-2 with vehicle emulation, 30
simulated nodes each emulate N vehicles transmitting at X Hz.
When a node generates a message at time t it also schedules its
next message at time t+T, where T is a uniform random variable
WSU 11
over the smaller interval [1/2NX, 3/2NX]. This is illustrated in
the upper portion of Fig. 2. In the testbed a radio always
WSU 12 emulates N vehicles at X Hz. Each radio generates N messages
WSU 13
in each fixed interval of duration 1/X seconds, and each message
generation time is chosen independently and uniformly over the
entire 1/X interval. This is illustrated in the lower portion of Fig.
2. Note that when using the emulation technique it is not
possible for two messages from the same node to collide. The
results in Section IV, however, show this to be a minor factor.
2) EDCA: The EDCA mechanism is supported in 802.11
chips in the test radios, but is not fully implemented in NS-2.
We simulated different EDCA parameters, but restricted each
message in a given simulation to use the same access category
(AC). We set the NS-2 parameter CWMin to the desired CWMin
value for the AC. Similarly, we set the NS-2 DIFS parameter to
the desired AIFSN value. We observe in the simulations that
our results tend to match radio results more tightly in the cases
of IEEE 802.11p default AC2 (CWMin = 3, AIFSN = 3) and AC3
(CWMin = 3, AIFSN = 2). The match between simulations and
radio tests is still good, but somewhat looser in the cases of AC0
(CWMin = 7, AIFSN = 6) and AC1 (CWMin = 15, AIFSN = 9). It
may be that our modelling of EDCA by configuring CWMin and
DIFS is a better approximation for AC2 and AC3 than it is AC0
and AC1. We expect that when EDCA is fully implemented in
NS-2 (rather than just configuring the DIFS and CWMin
parameters), the results will match more tightly for all EDCA
mechanisms.
NS-2 Emulation: Divide 100 msec into N sub-intervals; one packet
per sub-interval 100 msec Message Interval
100/N msec subinterval
Each packet could collide with any of 29*N packets from other
radios in 100 msec
Radio Testbed Emulation: Send N packets per 100 msec.
Each packet time uniform over 100 msec, independent of others
100 msec Message Interval
Do not enter back-off until prior packet sent; no self-collision
Each packet could collide with any of 29 packets from other radios in
100/N sub-interval
Fig 2: Emulation Techniques employed in the testbed and NS-2,
assuming 30 radios and each emulating N=6 radios.
IV. RESULTS
The default parameters for the radio and simulation tests are
shown in Table I. These are representative of defaults used in
industry testing [14].
For each combination of parameters, the number of emulated
vehicles per node was varied from N = 1 to 6. For the 5 Hz
message rate tests the number of emulated vehicles was
extended to include N = 7 to 12. In general the tests consisted of
varying one variable at a time from the default set. The primary
performance metric is packet error rate (PER), defined as the
ratio of packets not received correctly to total packets
transmitted. PER can be measured over one or more links and
over a variety of time intervals. In these tests PER is reported for
the set of links with RSS within a 5 dB range, e.g. -80 to -85
dBm, and the first 30 seconds of each test are excluded from the
results to avoid transient conditions. Graphical results are
presented in the Appendix. Each graph includes PER results for
one or more 5 dB RSS bins. Within each bin the graph
compares three tests with identical parameters: radio testbed
(with emulation), NS-2 simulation with emulation, and NS-2
simulation without emulation.
TABLE I. DEFAULT PARAMETERS FOR DSRC RADIO TESTS
Parameter Value
Power 20 dBm
EDCA mechanism AC0 IEEE 1609.4 trial version
OTA message size 378 Bytes
Data rate 6 Mbps
Channel 172 (10 MHz channel)
Message rate 10 Hz
Fig. 3 illustrates the basic results using the default parameters in
Table I. Results are shown for two RSS bins: -45<RSS<=40
dBm, and -85<RSS<=80 dBm. As expected, due to the fading
effect PER is higher for the low power bin (-85 to -80 dBm) and
lower for the high power bin (-45 to -40 dBm). Within each
power bin, PER increases with the number of vehicles
(emulated or individually simulated). This is consistent with the
higher frame collision probability that is associated with higher
channel load [11].
The most important observation from Fig. 3 is that the three
curves associated with a given parameter combination and a
given power bin tend to agree with each other quite closely.
This implies two things. First, it implies that the NS-2 simulator
platform accurately models the MAC and PHY of actual radios.
Second, it implies that the emulation technique used in the radio
testbed leads to approximately the same frame collision
probability as would be observed without emulation.
Generally, the NS-2 results without emulation show slightly
higher PER than the NS-2 results with emulation. This is
intuitive; when a node emulates multiple vehicles, the frames
transmitted by those vehicles cannot collide with each other, so
the emulation technique leads to lower PER. The NS-2 results
without emulation might also be expected to show higher PER
than the radio testbed results (with emulation). In our tests,
however, the radio testbed PER is slightly but consistently
higher than the simulation PER. This might reflect a small
unmodeled source of error in the radio tests, e.g. multipath
fading slightly worse than the Rician model, or sensitivity
slightly worse than the modeled value. Even with this caveat,
however, the agreement among the three test results for a given
power bin is quite high, indicating that our goal of
cross-validation is achieved in the default case.
Figs. 4-9 demonstrate the same general trends and relationships,
reflecting a series of tests in which one default parameter is
modified. In all cases the agreement among the three test
approaches remains quite high. Fig. 4 represents the case where
transmit power is reduced from 20 dBm to 10 dBm, which is
often considered a lower bound for DSRC safety
broadcasts. Fig. 4 also includes an extra bin (-65<RSS<=60
dBm) to illustrate performance at medium power levels. Figs. 5
and 6 show the effect of modifying the OTA message size from
the default 378 bytes, to 300 and 464 bytes, respectively. DSRC
safety message sizes vary depending on options related to
security and message content; the sizes tested reflect the
approximate minimum and maximum expected in
deployment. Fig. 7 illustrates a reduction in the message rate to
5 Hz per vehicle. Note that in the NS-2 simulation with
emulated vehicles, a test using 10 Hz message rate and M
vehicles is identical to a test using 5 Hz message rate and 2M
vehicles. For NS-2 simulations without emulation, there are
small differences between comparable 10 Hz and 5 Hz tests; this
is true of the radio tests as well. Fig. 8 shows test results
obtained using a 20 MHz channel with 12 Mbps data rate (i.e.
the modulation and coding rate remain QPSK and ½,
respectively). Finally, Fig. 9 shows the effect of using the
higher priority EDCA access category AC3. We tested every
access category in trial use 1609.4 (now obsolete) and
802.11p. The results for AC0 (Fig. 3) and AC3 (Fig. 9) are
representative.
In every case there is close agreement among the results of the
three test approaches, suggesting we have achieved our
cross-validation goal for these non-default cases as well. The
trends in this group of figures are mostly as expected, e.g. all
other things being equal, PER increases with decreasing
transmit power, with increasing message size, with increasing
message rate, and with decreasing channel bandwidth. The
latter result might not be obvious, since the advantage of shorter
packet transmission times is partly offset by the higher noise
floor associated with the 20 MHz channel. Since most losses in
these tests are due to collisions, the shorter packet time appears
to be a more important factor. The performance comparison
between AC0 and AC3 is also not obvious. AC3 is a higher
priority category which uses a shorter contention window that
might lead to more collisions. We might expect AC3 to perform
worse, but in fact AC3 has lower PER. This is a complex
question to analyze and we are investigating it as future
research.
V. CONCLUSIONS
In this paper we demonstrate two important results. First, we
show that for an indoor office environment where LOS is
available, the NS-2 simulator accurately models the MAC and
PHY associated with an actual 802.11 implementation. Second,
we show that a technique in which one radio emulates N
vehicles can produce results quite similar to the case in which a
larger number of distinct vehicles exist. Collectively, these
results suggest a two prong strategy for investigating methods
of controlling DSRC channel congestion: 1) use simulation as
the primary tool for testing a wide variety of algorithms and
parameters, and 2) use the emulation technique in radios so that
tens of nodes can model hundreds of vehicles, and perform
selective tests to validate the effectiveness of the simulated
approaches. Among many potential extensions of this work,
we consider a study of EDCA effects, based on an accurate
NS-2 EDCA implementation, to be the most
important. Outdoor testing is also important, though the close
agreement between our indoor radio tests and prior outdoor
tests is encouraging.
REFERENCES
[1] US Federal Communications Commission, R&O FCC 03-324,
“Dedicated Short Range Communications Report and Order”, Dec. 17,
2003.
[2] D. Jiang, V. Taliwal, A. Meier, W. Holfelder, and R. Herrtwich, “Design
of 5.9 Ghz DSRC-based Vehicular Safety Communication”, in IEEE
Wireless Communications, vol. 13, no. 5, pp. 36-43, Oct. 2006.
[3] IEEE Std. 802.11-2007, “Wireless LAN MAC and PHY Specification”.
[4] IEEE 802.11p, July 2010, “Amendement to Wireless LAN MAC and
PHY Specification”.
[5] IEEE Std. for Wireless Access in Vehicular Environemnts-Networking
Services, IEEE Std 1609.3-2010, Dec. 2010.
[6] IEEE Std. for Wireless Access in Vehicular Environments-Multi-Channel
Operations, IEEE Std 1609.4-2010, Dec. 2010.
[7] C.-L. Huang, Y. P. Fallah, R. Sengupta, and H. Krishnan, “Adaptive
Intervehicle Communication Control for Cooperative Safety Systems”, in
IEEE Network, vol. 24, no. 1, pp. 6-13, Jan.-Feb. 2010.
[8] M. Torrent-Moreno, J. Mittag, P. Santi, and H. Hartenstein,
“Vehicle-to-Vehicle Communication: Fair Transmit Power Control for
Safety-Critical Information”, in IEEE Transactions on Vehicular
Technology, vol. 58, no. 7, pp. 3684-3703, Sep. 2009.
[9] A. Weinfied, J. Kenney, and G. Bansal, “An Adaptive DSRC Message
Transmission Interval Control Algorithm”, ITS World Congress 2011,
Oct. 2011, Accepted for Publication.
[10] Q. Chen, F. Schmidt-Eisenlohr, D. Jiang, M. Torrent-Moreno, L.
Delgrossi, and H. Hartenstein, “Overhaul of IEEE 802.11 modeling and
simulation in ns-2”, Proceedings of the 10th ACM Symposium on
Modeling, analysis, and simulation of wireless and mobile systems,
October 2007.
[11] A. Weinfield, “Methods to Reduce DSRC Channel Congestion and
Improve V2V Communication Reliability” in ITS World Congress,
Busan, Korea, 2010.
[12] US Department of Transportation, National Highway Traffic Safety
Administration, “Vehicle Safety Communications-Applications VSC-A,
Draft Final Report, Appendix I – Multiple OBE Scalability Testing
Results”.
[13] IEEE Std 1609.4-2006, “IEEE Trial-Use Standard for Wireless Access in
Vehicular Environments (WAVE)-Multi-channel Operation”.
[14] US Department of Transportation, National Highway Traffic Safety
Administration, Rep. DOT HS 811 073, “Vehicle Safety
Communications-Applications VSC-A, First Annual Report”, Jan. 2009.
[15] M. Takai, Y. Owada, and K. Seki, “A Comparative Study on Network
Simulators for ITS Simulation IEEE802.11 Medium Access Control
(MAC) Models” in ITS World Congress, Stockholm, Sweden, 2009.
APPENDIX
Fig 3: PER vs emulated vehicles for default parameters.
Fig 4: PER vs emulated vehicles when transmit power is 10 dBm Fig 7: PER vs emulated vehicles when message rate is 5 Hz

Fig 8: PER vs emulated vehicles while using channel 175 (20 MHz bandwidth)
Fig 5: PER vs emulated vehicles when OTA message size is 300 Bytes

Fig 6: PER vs emulated vehicles when OTA message size is 464 Bytes Fig 9: PER vs emulated vehicles when EDCA mechanism is AC3 (trial version)