Professional Documents
Culture Documents
Abstract—With an increase in the number of mobile users • Cost: The EPC components such as MME (Mobility
and traffic, mobile network operators are finding it difficult to Management Entity), HSS (Home Subscriber Server),
scale their radio and core networks. Further, hardware network SGW (Serving gateway) and PGW (Packet data network
appliances are expensive to procure and upgrade, and are difficult
to adapt and program for new services. These trends have gateway) are proprietary devices, with a number of com-
recently spurred several efforts to redesign various components of plex functionalities packed into a single box. This makes
mobile networks, including the LTE Evolved Packet Core (EPC). them expensive to procure, maintain, and upgrade with
Software Defined Networking (SDN) and Network Functions increasing traffic demands.
Virtualization (NFV) are two popular emerging networking • Flexibility: Most components of the EPC have all the con-
paradigms that aim to increase network flexibility and scalability,
while reducing the overall cost. With SDN, the control and data trol and data plane logic programmed into the hardware.
planes of the packet core can be separated, enabling cheaper Any modifications in the functionality to add new services
packet gateways in the data plane, and an intelligent core network requires replacement of the entire appliance. Further, it is
controller to handle the signaling and management functions. usually hard to integrate and incorporate functions from
With NFV, the various hardware components that comprise the different vendors into a single box.
packet core can be virtualized and run as software on a cloud,
enabling benefits such as elastic scaling and quick innovation. • Scalability: Hardware appliances typically scale verti-
While several proposals exist to use SDN and NFV to redesign cally, i.e., the complete appliance must be replaced by
the EPC, there is no common framework to compare the new a more powerful one in order to handle higher load.
designs on various performance metrics. This paper presents the However, telecom operators would prefer architectures
design and evaluation of two open-source implementations of the that scale horizontally, in order to elastically scale the
LTE EPC, one based on SDN principles and the other based
on NFV, and presents a performance comparison of the two network in response to load.
approaches. Experiments with our prototype show that an NFV- • Signaling overheads: The current architecture involves
based implementation is better suited for networks with high a lot of control signaling among the EPC components,
signaling traffic, because handling the communication with the incurring substantial amounts of overhead in terms of
SDN controller quickly becomes the bottleneck at the switches in bandwidth and processing time.
the SDN-based EPC. On the other hand, an SDN-based design
of the EPC is better suited for networks with high data plane
traffic, because SDN switches are often more optimized for packet
forwarding than virtualized software appliances. We believe that
our framework can be used to develop and compare several such
design alternatives, and can serve as a guide for future redesigns
of mobile data packet core networks.
I. I NTRODUCTION
header (IEEE 802.1Q) to store the TEID (Tunnel Endpoint CONTROL MME CONTROL HSS
GTP-C / UDP
CONTROL
MME (via the default switch) uses UDP unlike in the case of RAN DATA SGW GTP-U / UDP
DATA
PGW DATA
SINK
LTE which uses SCTP.
Fig. 6: NFV-based LTE EPC implementation.
V. NFV- BASED EPC
We now provide an overview of our NFV-based EPC imple- Multi-threaded SCTP Server Architecture: The SCTP
mentation, as shown in Figure 6. The core network functions server architecture consists of a single master thread and
of the EPC (MME, HSS, SGW and PGW) are built as software several worker threads. The master thread communicates with
modules, and are run on VMs hosted on a private cloud. each of the worker threads via a logical Unix pipe. The master
Much like in the case of the SDN-based EPC, we also built a thread listens for incoming client connections on a stream
RAN-simulator module that combines the functionalities of the socket, and creates an additional socket file descriptor for
Registration throughput (regs/sec)
every new SCTP client that is accepted. It then sends the new
3500
connection socket file descriptor to the worker threads in a
round-robin fashion, to balance load across all worker threads. 3000
The worker thread is responsible for servicing the request on 2500
the new connection, through all the steps required to handle 2000
the request. For example, the worker thread in the MME 1500
must implement the various steps required to process a UE 1000
attach request, blocking for responses at each stage from the
500
other nodes. The workers threads use the event-driven select
00 5 10 15 20
mechanism to multiplex the multiple requests being handled
by each of them concurrently. The number of worker threads in Number of concurrent UEs
this architecture can be sized to fully utilize all the processing Fig. 7: SDN EPC: Registration throughput vs. no. UEs.
cores of the VM.
Multi-threaded UDP Server Architecture: The UDP
server at the gateways consists of a single datagram socket of concurrent UEs and the rate at which each UE generates
being serviced by multiple server threads. All server threads traffic. We measure two performance metrics: (i) throughput,
try to read a UDP packet from the common shared socket. or the amount of data traffic successfully forwarded by the
The thread that succeeds in reading the packet is responsible EPC gateways to the sink per second, and (ii) round trip
for processing the request and sending a response back to latency, or the amount of processing overhead added by the
the downstream node. By fixing a suitable number of server EPC gateways in the data plane, as measured by a ping
threads in proportion to the number of system cores, this command during a load test. We ran all experiments for a
architecture provides good performance and scalability. duration of 300 seconds.
In both the server architectures, a global data structure is For the NFV-based EPC setup, all VMs were installed
used by the threads to store/retrieve necessary connection data on a private OpenStack-based cloud, and were provisioned
and state. This data structure is protected by locks to prevent resources as shown in Table I. We use two different setups
concurrent inconsistent access by the multiple threads. As when testing our SDN-based EPC. For control traffic exper-
part of future work, we plan to redesign our system using iments, where the bottleneck is likely to be the controller
lock-free data structures to eliminate contention between the CPU, we installed each of the components shown in Figure 5
threads for shared data structures. We also plan to explore on separate physical machines. All machines were connected
alternatives to the kernel-based socket communication model, by a 1Gbps LAN. However, for data traffic experiments, the
e.g., a userspace network stack built on top of kernel bypass network between the machines became a bottleneck. To avoid
techniques like Intel DPDK [19]. the network bottleneck, we installed the various components
within Linux containers (LXC) on the same physical machine,
VI. E VALUATION and connected them up using the Linux bridge. In all ex-
We now compare our two EPC implementations on vari- periments with the SDN EPC, all machines/containers of all
ous performance metrics under different traffic conditions, to components were provisioned with 2-core Intel i5-4440 CPU
assess the pros and cons of the designs. We begin with a @ 3.10 GHz, and 4GB RAM. Care was taken to provision
description of our testbed setup. similar resources in both the EPC implementations, in order
Setup. We test our EPC implementations using two types to be able to fairly compare the performance results.
of traffic: control traffic, consisting mainly of UE attach
(registration) requests generated by simulated users in the TABLE I: Configuration of NFV-based EPC testbed.
RAN simulator, and data traffic, consisting of TCP traffic
ENTITY CPU #CORES RAM OS
generated from the RAN simulator to the sink. We also ran
RAN 8 4GB
experiments with varying mixes of control and data traffic; Intel Xeon Ubuntu
MME, SGW, PGW, HSS E312xx @ 2 2GB 14.04
we do not report those results here due to lack of space. 2.6 GHz Server
For experiments with control traffic, we simulate a number Sink 4 2GB
of concurrent UEs in the RAN simulator, and make the UEs
continuously attach and detach from the LTE network, to
create a continuous stream of control traffic. We vary load A. Control plane performance
on the EPC by varying the number of concurrent UE threads. We now generate continuous control traffic to our EPC
We then measure the throughput, or the number of registration setups, and compare their performance along the metrics of
requests successfully completed by the EPC per second. throughput and latency. Figures 7 and 8 show the registration
For experiments with data traffic, we attach a specified throughput of the EPC implementations, as we increase the
number of UEs to the EPC from the simulator, and pump number of concurrent UEs in the RAN simulator. We find that
traffic from the RAN to the sink using iperf3 [20]. We vary the NFV-based setup is able to support a much higher control
the input data traffic into the EPC by varying both the number plane throughput (approx. 9000 reqs/s) than the SDN-based
45
140
Avg. CPU utilization (out of 200%)
4.5
switches also, in addition to processing overheads at the 4.0
controller, when processing control plane signaling messages. 3.5
Therefore, for networks with heavy signaling traffic, an NFV- 3.0
based EPC design might provide better performance than an 2.5
SDN-based design, where switches must pass all signaling 2.0
messages to the controller and install flow rules. 1.5
1.0
B. Data plane performance 0.5 0
10 101 102 103
We now load our EPC implementations with only data Input load generated (Mbps)
traffic and measure their performance. Figures 10 and 11 show Fig. 13: NFV EPC: Data plane latency.
the data plane throughput forwarded by the EPC gateways
to the sink in the SDN and NFV setups respectively, as we
increase the offered data plane traffic to the EPC. We find gateways (approx. 110 Mbps) which must send the traffic to
that the SDN EPC gateways that forward traffic directly from user space in the VM to make a forwarding decision. In fact,
the OpenvSwitch in kernel space are able to handle higher while our NFV-based EPC gateways hit a CPU bottleneck
amounts of traffic (approx. 45 Gbps) than the NFV EPC in our experiments, we were never able to fully saturate our
SDN-based gateways, as we were limited by the maximum other EPC architectures. We believe that such experimentation
network bandwidth for load generation (even in our container- will help us make informed choices in architecting the mobile
based setup), and not the forwarding capacity of the gateways. core in future standards. As part of our future work, we
Therefore, we believe their forwarding capacity is much higher plan to enhance our code by adding more features to make
than what our results show. our implementations more standards-compliant (though full
We believe that our SDN EPC will give much higher compliance and use in production settings are non-goals). We
performance if run on hardware SDN switches instead of are upgrading our code to use the latest high-performance
software switches. Similarly, we expect the performance of the network I/O frameworks such as Intel DPDK. We are also
NFV EPC gateways to also improve considerably if techniques working on distributed versions of the various NFV-based EPC
to bypass the kernel and improve network I/O for VMs, such software components, to explore the design choices in building
as Intel DPDK [19], were to be employed in the development scalable virtual network functions.
of our gateways. We are working on these improvements
R EFERENCES
currently. That said, with our current implementations, our
results show that an SDN-based EPC can give a much better [1] Cisco, “Cisco visual networking index: Global mobile
data traffic forecast update, 2015-2020.” [Online]. Avail-
data plane performance than an NFV-based EPC, simply able: http://www.cisco.com/c/en/us/solutions/collateral/service-provider/
because data plane forwarding can be done more effectively visual-networking-index-vni/mobile-white-paper-c11-520862.html
in a hardware or kernel switch than in a user-space VM. [2] “Connectem,” http://www.brocade.com/en/products-services/
mobile-networking.html.
Figures 12 and 13 also show the data plane processing round- [3] “OpenEPC,” http://www.openepc.com.
trip latencies of the SDN and NFV EPC setups. As expected, [4] “Virtual Mobile Network,” http://www.affirmednetworks.com/
the SDN EPC has a lower processing overhead than the NFV products-solutions/vepc/.
[5] “SDN LTE EPC,” https://github.com/networkedsystemsIITB/SDN
EPC for the same reasons discussed above. LTE EPC.
[6] “NFV LTE EPC,” https://github.com/networkedsystemsIITB/NFV
VII. C ONCLUSION AND F UTURE W ORK LTE EPC.
[7] A. Basta, W. Kellerer, M. Hoffmann, K. Hoffmann, and E.-D. Schmidt,
In this paper, we presented two implementations of the “A Virtual SDN-Enabled LTE EPC Architecture: A Case Study for S-
LTE EPC, one based on the principles of SDN, and the /P-Gateways Functions,” in Proc. of IEEE SDN for Future Networks and
other based on the idea of NFV. The SDN-based architec- Services (SDN4FNS), 2013.
[8] A. Basta, W. Kellerer, M. Hoffmann, H. J. Morper, and K. Hoffmann,
ture separates out the control and data planes in the EPC “Applying NFV and SDN to LTE Mobile Core Gateways, the Functions
components. The control functions run as applications on top Placement Problem,” in Proc. ACM SIGCOMM Workshop on All Things
of an SDN controller, while the data plane runs on general Cellular: Operations, Applications, Challenges, 2014.
[9] S. Ben Hadj Said, M. Sama, K. Guillouard, L. Suciu, G. Simon,
purpose SDN switches. The NFV-based architecture builds the X. Lagrange, and J.-M. Bonnin, “New control plane in 3GPP LTE/EPC
EPC components as software appliances running on virtual architecture for on-demand connectivity service,” in Proc. IEEE Inter-
machines in datacenters. Both the architectures eliminate the national Conference on Cloud Networking (CloudNet), 2013.
[10] L. E. Li, Z. M. Mao, and J. Rexford, “Toward Software-Defined
need for custom hardware equipment in deploying network Cellular Networks,” in Proc. European Workshop on Software Defined
functions, and provide great flexibility in managing resources. Networking, 2012.
While these architectures have been proposed before, we [11] X. Jin, L. E. Li, L. Vanbever, and J. Rexford, “SoftCell: Scalable and
Flexible Cellular Core Network Architecture,” in Proc. ACM Conference
have provided a complete implementation and performance on Emerging Networking Experiments and Technologies (CoNEXT),
comparison of both architectures across different types of 2013.
traffic. Our results show that an SDN-based EPC is a better [12] J. Cho, B. Nguyen, A. Banerjee, R. Ricci, J. Van der Merwe, and
K. Webb, “SMORE: Software-defined Networking Mobile Offloading
design choice when handling large amounts of data traffic, Architecture,” in Proc. ACM SIGCOMM Workshop on All Things
since it incurs lesser overhead when forwarding packets from Cellular: Operations, Applications, and Challenges, 2014.
the kernel or the switching hardware, as compared to an NFV [13] Brent Hirschman, Pranav Mehta, Kannan Babu Ramia, Ashok Sunder
Rajan, Edwin Dylag, Ajaypal Singh, and Martin McDonald, “High-
setup that makes forwarding decisions in user space. On the performance evolved packet core signaling and bearer processing on
other hand, an NFV-based EPC setup is better at handling general-purpose processors,” IEEE Network, vol. 29, no. 3, 2015.
large signaling load, because communicating every signaling [14] A. Banerjee, R. Mahindra, K. Sundaresan, S. Kasera, K. Van der Merwe,
and S. Rangarajan, “Scaling the LTE Control-Plane for Future Mobile
message with the controller and processing the resulting rule Access,” 2015.
updates quickly overwhelms the SDN switches. Therefore, the [15] X. An, F. Pianese, I. Widjaja, and U. Günay Acer, “DMME: A Dis-
optimal EPC architecture for a given network would depend tributed LTE Mobility Management Entity, journal=Bell Labs Technical
Journal, year=2012, volume=17, number=2,.”
on, among other things, the mix of control and data traffic [16] Y. Takano, A. Khan, M. Tamura, S. Iwashina, and T. Shimizu,
expected by the operator. We have released our entire code “Virtualization-Based Scaling Methods for Stateful Cellular Network
on Github [5], [6] as open-source, in order to spur further Nodes Using Elastic Core Architecture,” in Proc. IEEE International
Conference on Cloud Computing Technology and Science (CloudCom),
research in this direction. 2014.
While we have drawn some conclusions on the pros and [17] “Open Air Interface,” http://www.openairinterface.org/.
cons of various design options using our initial prototypes, [18] “Floodlight,” http://www.projectfloodlight.org/floodlight/, 2015.
[19] “Intel Data Plane Development Kit for Linux*: Guide,”
we envision this work to be the beginning, and not the end, http://www.intel.com/content/www/us/en/intelligent-systems/
of the discussion on EPC design choices. We invite other intel-technology/intel-dpdk-getting-started-guide.html.
researchers to extend the code to build and compare several [20] “Iperf3,” https://iperf.fr/, 2014.