Ain Shams Engineering Journal 9 (2018) 2079–2091

Contents lists available at ScienceDirect

Ain Shams Engineering Journal

journal homepage: www.sciencedirect.com

Electrical Engineering

Design and implementation of BGP novel control mechanism (BGP-NCM)

based on network performance parameters
M.T. Moubarak ⇑, Ashraf Diaa Elbayoumy, Mohamed Helmy Megahed
Communications Department, Military Technical College, Cairo, Egypt

a r t i c l e i n f o a b s t r a c t

Article history: Border Gateway Protocol (BGP) is the defecto standard routing protocol of the internet and has served as
Received 2 November 2016 backbone technology for logical routing tasks to provide global connectivity in the world. Unfortunately,
Revised 10 February 2017 there are limitations on network performance factors such as latency or link utilization which are not used
Accepted 27 February 2017
in BGP routing decisions. These limitations have a great effect on internet services, especially bandwidth
Available online 9 March 2017
sensitive application, where BGP does not automatically redistribute the traffic over different
Multihomed links based on link utilization or latency. In this paper, a novel control mechanism for BGP
Keywords:
(BGP-NCM) is proposed. BGP-NCM is a software controller which has been developed and injected in the
Bandwidth
BGP
Enterprise or internet service provider BGP multihomed network to force the traffic redistribution over
BGP-NCM the available links based on the link utilization or latency. The BGP-NCM has been developed using
Utilization Python programming language and it consists of three modules which are Monitoring, Traffic Calculation
Multihomed and Configuration modules. BGP-NCM will keep an eye on link utilization by checking the monitoring tool
Python log and when the link utilization reaches a predefined value, the BGP-NCM traffic calculation module will be
Quagga triggered to determine the amount of traffic and corresponding IP prefix as well as the target link with avail-
SLA able bandwidth to accommodate the redistributed traffic. Our results show that BGP-NCM solves the traffic
SNMP
redistribution problem without any change in both BGP protocol and running internet infrastructure.
Ó 2017 Ain Shams University. Production and hosting by Elsevier B.V. This is an open access article under
the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/).

1. Introduction titude of paths to increase the end-to-end network throughput.

For the last twenty years, internet network is a critical infras-
tructure for largest part of world’s population and its failure or
even weak performance may noticeably influence daily life of its
users which includes businesses, academia, entertainment and
other sectors [1].
Border Gateway Protocol (BGP) is the defecto standard routing
protocol of the internet and has served as backbone technology
for logical routing tasks to provide global connectivity in the world
[2].
Meanwhile, the hierarchical structure of the Internet is becom-
ing flatter [3], which leads to multiple paths between every pair of
Autonomous Servers (ASes). However, BGP as the dominant inter-
domain protocol fails to take advantage of the availability of a mul-
⇑ Corresponding author.
E-mail address: m_tawfik20@hotmail.com (M.T. Moubarak).
Peer review under responsibility of Ain Shams University.
Production and hosting by Elsevier
Instead, todays BGP routers only use a single, default path, which
when congested leads to even poor performance. While BGP is
being traffic agnostic, it fails to react to congestion and burstiness
in the traffic [4].
The Internet performance is affected due to over-utilization or
congestion of a certain path as well as DDOS attacks which has a
significant effect on link bandwidth. In such a case, internet appli-
cations will suffer from high delay which affects the quality of ser-
vice or it may lead to session failure between the source and
destination.
The relationship between a client and a network or service pro-
vider is governed by a Service Level Agreement (SLA), and SLA ful-
filment remains an open issue [5].
Multihoming is a technique to increase the reliability of the
Internet connection for an IP network using multiple links to the
internet from different Internet Service Providers (ISPs). To allow
links redundancy, the BGP routing protocol is used for traffic distri-
bution over these multiple links [6,7] but not automatically.
Standard inter-domain traffic engineering (TE) consists of out-
going traffic engineering and incoming traffic engineering outgoing
traffic engineering is relatively easy to optimize because routing
outgoing traffic is decided by the AS itself. On the contrary,

http://dx.doi.org/10.1016/j.asej.2017.02.008
2090-4479/Ó 2017 Ain Shams University. Production and hosting by Elsevier B.V.
This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/).
2080 M.T. Moubarak et al. / Ain Shams Engineering Journal 9 (2018) 2079–2091
incoming traffic engineering is difficult to optimize because rout-
ing incoming traffic is decided by other ASes in the Internet [8].
Latest BGP version has no direct implementation that supports
any Quality of Service (QoS) and it cannot meet SLA which is
important for the internet users. Majority of IP traffic forwarding
devices has their own control plane software with narrow view
on global path possibilities to destination IP network and no infor-
mation about chosen best path QoS characteristics [9].
In normal BGP operation, only after the best-path calculation is
complete, it is possible to measure and enhance the network per-
formance parameters such as latency, throughput, jitter, packet
loss and others. The lack of wider internet performance signaling
and the difficulties for pro-active measurements implementation
has made the internet as a hard-challenging network to use the
network performance factors such as link utilization and latency
in routing decisions [10].
There are limitations on network performance factors such as
latency or link utilization which are not used in BGP routing deci-
sions. These limitations have a great effect on internet services,
especially bandwidth sensitive application, where BGP does not
automatically redistribute the traffic over different Multihomed
links based on link utilization or latency.
In this paper, the design and implementation of BGP Novel Con-
trol Mechanism (BGP-NCM) has been proposed as an effective
solution to the previously mentioned BGP limitation, supports in
some the internet scalability challenges and internet users’
requirements such as:
 Network and service availability to be 100% up-time.
 Keeping Latency within the acceptable range.
 Keeping links usage below predefined commit levels.
 Guarantee the effective/proper use of available internet
resources.
The BGP-NCM has been designed and implemented to enhance
the network performance parameters as follows:
 Implementation of BGP-NCM for BGP control based on network
performance parameters such as link utilization and latency
without any change in BGP protocol or running Internet
environment.
 Redistribution of traffic based on link utilization over available
multihomed links for congested links offers optimum band-
width management to enhance the network performance and
achieves the required Service Level Agreement (SLA).
 Reduce the end to end time delay for time sensitive applications
by switching the application to other multihomed Link with
available bandwidth.
 BGP-NCM will handle BGP route updates in automatic and
dynamic manner based on network performance and cost
metrics.
The reset of this paper includes the followings: Section 2
describes the Related Work. Section 3 introduces the concept,
used tools and packages, implementation and operation process
of the BGP-NCM. Section 4 represents the research network
design. Section 5 shows the traffic performance measurements.
Section 6 presents BGP-NCM implementation challenges. Section 7
presents the conclusion and finally future work has been pre-
sented in Section 8.
2. Related work
In [10], the authors proposed the development of separate
latency management control panel with the following features
for end-users: first, the ability to configure measurement policies
including probe addresses and second, monitoring the measured
latency and the chosen path. Because of the limited community’s
information space, a new kind of extended communities could be
proposed for latency information sharing through BGP update
messages.
In [11], the authors proposed Multi-Path Inter-Domain For-
warding (MIFO) protocol, which enables AS border routers to adap-
tively offload outbound traffic from a congested default to an
alternate path with only data plane modification, rather than hav-
ing the load information communicated to the control plane and
using it to recognize alternate paths. Specifically, MIFO explores
multiple paths learnt by the control plane and uses load-aware for-
warding at the data plane over different outgoing AS paths, to keep
forwarding in line speed.
In [12], the authors proposed a novel inter-domain multipath
flow transfer mechanism based on SDN and multi-domain collab-
oration where they designed a routing detection method based
on hierarchical iteration. The authors take advantages of the SDN
strong controller and fine-grained computing abilities to improve
the detection capability as well as interdomain collaboration to
relieve the overheads of other proposed multipath solutions based
on BGP notification without changing the existing BGP protocol.
These protocols are such as Cisco multi-path selection mechanism
based on BGP, a route deflection mechanism based on labels, and
the multipath BGP (MBGP) mechanism.
In [13], the authors proposed a Multi-Dimension Link Vector
network view exchange mechanism (MLV). The exchanged domain
views provide fine-grained information including not only IP desti-
nation and AS path, but also, the AS links with IP flow 5-tuples
routing information and QoS attributes.
In [14], the authors proposed Routing As a Service (RAS)
which imports third-party entities for calculating routing paths
to support customized routing with high scalability and great
flexibility.
In [15], the authors proposed outsourcing the Routing Control
Logic as a new routing model. The outsourcing control logic is used
for inter-domain routing.
In [16], the authors focused on not only the network reliability
in terms of maintaining connectivity but rather on the networks
QoS and its ability to meet a certain SLA. Therefore, reliability anal-
ysis becomes a special case of this general model. Various param-
eters are used to realistically represent the properties of end-to-
end routes. These properties include the number and composition
of multipath, the degree of multihoming and the number of hops in
each serial path.
All previously mentioned related work tried to solve the prob-
lem using one of the following approaches:
a. Modifying the existing BGP protocol which leads to more
overheads to the protocol which affects the performance
and can’t be practically implemented due to the inability
to change the running internet environment.
b. Using of Software Defined Network (SDN) but using SDN
needs that all routers and switches over the internet support
the SDN which cannot be implemented in the real internet
environment. Also, routers and switches in the traffic path
should be under the same administration to receive the traf-
fic flow changes from the same SDN controller.
Accordingly, the proposed controller is designed such that the
key feature of the proposed mechanism is that it could be real time
implemented to solve the traffic redistribution problem without
any change in both BGP protocol and running internet
infrastructure.
 M.T. Moubarak et al. / Ain Shams Engineering Journal 9 (2018) 2079–2091
3. BGP novel control mechanism (BGP-NCM)
3.1. Concept of the mechanism
The proposed BGP Novel control mechanism (BGP-NCM) is a
software controller which has been developed and injected in the
BGP multihomed network to force the traffic redistribution over
the available links based on the link utilization, latency or any
other network performance parameters could be measured such
as packet loss.
On the following work, we will focus on link utilization because
the link congestion results in both packet loss and high latency as
well as packet loss may be due to faulty devices or cabling and this
can’t be solved by software mechanism.
The proposed mechanism will be used to automatically force
BGP policies over the enterprise or ISP routers to handle the traffic
based on the link utilization and this action will be carried out if
the utilization of the link reached a predefined value.
The designed BGP-NCM concept has been evolved based on the
Software Defined Network (SDN) [17,18]concept that there is a
controller which controls network switches and it forces the traffic
flows over this switches using the open follow protocol.
SDN needs that all routers and switches should support SDN
and must be under the same administration while the used BGP-
NCM will be implemented without any change in interdomain
routing system but only adjust the BGP attributes to change traffic
routes and accordingly link utilization will be changed.
BGP-NCM will be embedded in the network and just collects
network performance metrics from different tools and send new
routing policy through telnet or SSH sessions to the router and
doesn’t run BGP protocol and accordingly no traffic will be passed
or forwarded to it to avoid any performance degradation and no
more network overheads or security threats have been expected
in case of implementing the BGP-NCM and if it fails the BGP will
return to its normal operation finally expected performance and
security issues will be discussed on BGPNCM implementation chal-
lenges section.
3.2. Used tools and packages
The proposed BGP-NCM was developed using different software
packages as well as used several software packages to deploy the
routers, monitor the traffic utilization, calculate the traffic utiliza-
tion and finally force the router policy. These software packages
are as the following:
a. Community Enterprise Operating System (CentOS) [19,20],
CentOS is a Linux distribution that attempts to provide a
free, enterprise-class, community-supported computing
platform which aims to be functionally compatible with its
upstream source, Red Hat Enterprise Linux (RHEL). The Key
feature of CentOS that it has a repository which provides a
set of dynamic programming languages, database servers,
and various related packages.
b. Quagga Routing Software Suite [21], Quagga is a routing
software package that provides TCP/IP based routing ser-
vices with routing protocols such as OSPF, RIP and BGP-4.
Quagga also supports special BGP Route Reflector and Route
Server behavior. In addition to traditional IPv4 routing pro-
tocols, Quagga also supports IPv6 routing protocols. Quagga
provides routing protocol MIBs. Quagga uses an advanced
software architecture to provide a high quality, multi-
server routing engine. Quagga has an interactive user inter-
face for each routing protocol and supports common client
commands. Due to this design, a new protocol could be
2081
added to Quagga easily. Finally, Quagga is distributed under
the GNU Public License operating system like CentOS,
FreeBSD and NetBSD. Quagga daemons are shown in Fig. 1.
Where each routing protocol has it corresponding daemon
which should be enabled to allow the routing protocol over
Quagga router as well as the Zebra daemon which is the
master routing daemon.
c. The Multi Router Traffic Grapher (MRTG) [22,23], MRTG is a
graphical tool used to monitor the amount of traffic passing
through a network link, such as a router, switch or server.
Being written in Perl, it has successfully been ported to most
Unix platforms as well as Microsoft Windows. MRTG uses
data obtained from using Simple Network Management Pro-
tocol(SNMP) [24,25] to poll the network devices. Once the
data is obtained, it generates HTML pages containing images
based on the traffic load, allowing a graphical view of the
traffic load complete with timestamps and history. Sample
graph of the MRTG output is shown in Fig. 2.
d. Python programming language [26–29] is a suite of client–
server protocol implementations, C integration, and third-
party tools and libraries. Python is ideal for prototyping
and deploying network applications. Python’s runtime inter-
pretation drastically shortens the turnaround from modifi-
cation to execution, a significant advantage when most
changes are incremental and most problems appear at run-
time. Python’s clean syntax, dynamic typing, integrated
exception handling, and object facilities have made it a
widely used interpreted language. With the emergence of
large-scale network tools and applications, such as the
twisted framework, Bit Torrent, and Zope content manage-
ment system, Python has proven to be a legitimate platform
for network-oriented projects in a space that C has tradition-
ally dominated.
e. NetFlow was a protocol originally developed by Cisco Sys-
tems in 1996 to collect IP traffic information. It can provide
traffic statistics such as the top bandwidth users or IP pre-
fixes, applications they use and what percentage of traffic
they use and this information is sent to a collector [30,31].
Sample of NetFlow output is shown in Fig. 3.
NetFlow has many equivalents, where other vendors than Cisco
provide an equivalent technology on their routers and switches,
but use a different name for the technology Jflow or cflowd for
Juniper Networks, NetStream for 3Com/HP and Huawei, Cflowd
for Alcatel-Lucent Rflow for Ericsson, AppFlow Citrix, Traffic Flow
MikroTik and sFlow for other many vendors like Alaxala, Allied
Telesis, Arista Networks, Brocade, Dell, D-Link, Enterasys, Extreme,
Fortinet, Hitachi, IBM, Juniper, LG, Mellanox, MRV, NEC, Netgear,
Proxim Wireless, Quanta Computer, Vyatta, ZTE and ZyXEL.
Fig. 1. Quagga daemons.
2082 M.T. Moubarak et al. / Ain Shams Engineering Journal 9 (2018) 2079–2091

Fig. 2. MRTG traffic graph.

Fig. 3. NetFlow output sample.

3.3. BGP-NCM implementation
The proposed BGP-NCM consists of Internet network, link and
BGP-NCM software as shown in Fig. 4.
Internet network represents services, users, traffic and routers.
BGP routers are the most important part which has been imple-
mented using Quagga daemon over CentOS Linux machine.
Internet services like HTTP and FTP have been represented by
CentOS Linux machine with both HTTP and FTP service enabled
and has been configured to listen and respond to internet users’
requests.
Internet users have been represented by CentOS Linux machine
with Linux script used to generate the traffic over the network.
Each ISP has been implemented by CentOS Linux machine with
Quagga daemon. BGP is the deployed routing protocol over all
Quagga routers.
MRTG package has been deployed as link monitoring tool and
SNMP configuration has been applied between MRTG and Quagga
routers to get each link utilization statistics instantaneously and
build link utilization data base.
BGP-NCM software has been developed using Python program-
ming language and the BGP-NCM consists of three modules.
3.3.1. BGP-NCM link monitoring module
a. It will monitor each link utilization by periodically checking
the MRTG Log.
b. MRTG polls each router interface using SNMP to get the uti-
lization statistics and accordingly updates each link log file
every 5 min.
c. Each record in the log file is time stamped and the monitor-
ing module will keep track with log file updates to check if
the threshold value is reached.
d. When the threshold value has been reached, the module will
continue to check the log and for more 5 consecutive times
to confirm that the threshold value has been reached and
link utilization is continuously exceeding the threshold.
e. The previously mentioned multiple check action for utiliza-
tion used to avoid network instability where the utilization
increase may be instantaneously and returns to its normal
 M.T. Moubarak et al. / Ain Shams Engineering Journal 9 (2018) 2079–2091
Fig. 4. BGP-NCM components.
level again and accordingly no need for traffic
redistribution.
f. BGP-NCM Traffic Calculation Module will be initiated based
on the confirmation of that the link utilization has exceeded
the threshold value as per the previous step.
g. Threshold value for each link will be determined based on
the services offered through this link and its required SLA.
3.3.2. BGP-NCM traffic calculation module
a. It will be triggered when the predefined threshold is
reached.
b. For this module, we get the traffic statistics based on Net-
Flow or Internet Protocol Flow Information Export (IPFIX)
reports.
c. Traffic Calculation module will analyze the Net-Flow report
and generate table of utilization, Prefix pairs in descending
order from highest utilization Prefix to lowest utilization one.
d. After determining the utilization Prefix Pairs, the suitable
pair will be selected to reroute the traffic over another link
based on available bandwidth over another links.
e. Finally, if there is no bandwidth available, it sends an alarm
to the network administrator otherwise the BGP configura-
tion module will be initiated.
3.3.3. BGP-NCM configuration module
a. BGP differs from the other routing protocols by neither being
classified as a distance-vector protocol or a link state-
protocol. BGP bases its routing decisions on attributes. These
attributes are divided into groups of well-known mandatory
and discretionary [32].
b. Well known mandatory must appear in every UP-DATE mes-
sage and must be supported by every BGP software
implementation.
 AS-PATH: The AS-PATH attribute is a list of all the ASes
that route must take to reach its destination. Separated
by spaces, this list can contain several ASes if the distance
is big enough.
2083
 ORIGIN: The origin attribute indicates how BGP learned
about a route. Allowed values are IGP, EGP or
Incomplete.
 NEXT-HOP: The EBGP next-hop attribute is the IP address
that is used to reach the advertising router.
c. Well known discretionary May or may not appear in every
UPDATE message, but must be supported
 LOCAL-PREF: Used by a BGP peer to prefer a specific exit
point from the local AS if there are several to choose
from. This information is propagated to peers in the
UPDATE message.
 ATOMIC-AGGREGATE: Alerts BGP speakers along the
path that some information has been lost due to route
aggregation
d. BGP policies could be applied to control both Inbound and
Outbound traffic and accordingly the BGP-NCM control
mechanism can be deployed to both scenarios.
e. BGP AS Path attribute will be used to control the inbound
traffic by using AS Path prepending technique as well as
the BGP local preference attribute which will be used to con-
trol the outbound traffic.
f. AS Path prepending technique used to make BGP routers to
insert AS numbers to the Path to become more longer and
then to preferred by BGP routers and select other shortest
Paths for this Prefix if available.
g. BGP local preference attribute will be used to select the out-
bound traffic where route with highest local preference
value will be preferable.
h. BGP-NCM configuration module will select the suitable BGP
policy to be applied to reroute the selected prefix over the
new link by adjusting the above BGP attributes.
i. Before applying the new configuration, a penalty timer will
be checked to avoid Route dampening penalty.
j. Route dampening penalty is used to help service providers
prevent one customer’s router or circuit problems from
affecting the stability of the provider’s network by with-
drawing problem BGP routes.
k. Route dampening penalty could be applied due multiple
routing updates which caused by multiple configuration
2084 M.T. Moubarak et al. / Ain Shams Engineering Journal 9 (2018) 2079–2091
changes or link flapping so penalty timer has been imple-
mented in the BGP configuration module.
BGP-NCM operation process could be summarized as follows:
(1) Monitoring Module Loop if utilization threshold isn’t
reached then check MRTG Log.
(2) Initiate Traffic Calculation Module.
(3) Get NetFlow Statistics for the target Link.
(4) Calculate Traffic per each Prefix.
(5) Build Prefix- Bandwidth Utilization Table.
(6) Select Highest Prefix and Bandwidth Utilization Pair.
(7) Check MRTG log for other links for bandwidth availability
and (If there is no bandwidth available then return to (6)
and select the 2nd highest prefix bandwidth pair) and repeat
(7) if no available bandwidth then sends Alarm.
(8) Issue BGP configuration policy.
(9) Check penalty timer if no then wait and recheck else send
configuration, reset the timer and return to (1).
BGP-NCM operation process is shown in Fig. 5 as a system flow
chart to present the previously mentioned steps as well as the
three modules operation has been represented.
Fig. 5. BGP control mechanism operation process.
4. Research network design
The designed BGP network has been divided into three sections
Enterprise, Internet service providers (ISP) and internet users as
shown in Fig. 6.
The enterprise section represents the service part where it
offers different internet services to the internet users. Enterprise
uses three BGP Multihomed links of 10 Mbps bandwidth to main-
tain its service reliability, availability and off course its SLA.
ISP section represented tier 1 or tear 2 ISP where the internet
traffic transits through their networks from the internet users to
enterprise and vice versa.
Internet users’ sections represented the tier 3 service provider
which offers internet access to the internet users for normal inter-
net activities.
IP prefixes has been assigned to each network section and
accordingly BGP configuration has been applied to each router in
the network to allow end to end traffic flow from users’ section
to access the services in enterprise section.
MRTG has been installed and SNMP configuration (Server IP,
SNMP community) has been applied over the MRTG server and
enterprise Quagga router to allow utilization monitoring for the
three multihomed links of the enterprise.
This network model represents real time network scenario with
the connections between users, enterprises and ISPs.
5. Traffic performance measurements
Link utilization has a significant effect on network performance
parameters delay and jitter especially when the link becomes fully
utilized or over utilized.
In normal BGP operation traffic may increase till the link
becomes over utilized with no action done to redistribute the traf-
fic over any link has available bandwidth and accordingly the delay
and jitter will be increased and affect the offered services.
In the following experiment, Network performance will be mea-
sured by an End-to-End delay and jitter over 10Mbps links and
simulation has been done using OP-Net Modeler [33,34]. The per-
formance is compared for different traffic loads with 50%, 75%,
100% and 120% percent of link capacity to focus on the effect of link
over utilization on the network performance. Fig. 7 shows that for
link loads (50% and 75%) the delay varies from 1 ms to 2 ms and for
(100%) it increased to reach about 3.5 Sec where link is fully uti-
lized and finally when link has been over utilized with load
(120%) the delay significantly increased and it reaches 22.5 s which
means that unacceptable service level has been reached.
Fig. 8 shows that for loads (50%, 75%and 100%) that jitter varies
from about 0 to 3 ms but when link is over utilized with (120%)
load it reaches to 246 ms.
As shown in this experiment the traffic over link should be kept
within the link capacity and link should not be over utilized and
BGP-NCM will be deployed to redistribute the traffic and keep
the traffic load within link capacity.
The network requirement represented by delay, jitter and
packet loss for some services are shown in Table 1 which confirms
how these services will be affected by link over utilization [35].
In this scenario BGP-NCM will be deployed at the enterprise to
maintain enterprise internet links utilization under 90% of link
bandwidth and maintains the enterprise offered services quality
or its SLA. The enterprise 1st internet link connected to ISP1 has
about 4 Mbp traffic load and then starts to increase to reach about
9.5 Mbps as shown in Fig. 9.
The enterprise 2nd internet link connected to ISP 2 was config-
ured as a backup for both 1st link and the 3rd link with zero traffic
load as shown in Fig. 10 and the enterprise 3rd internet link con-
nected to ISP3 has about 5 Mbps traffic load as shown in Fig. 11.
 M.T. Moubarak et al. / Ain Shams Engineering Journal 9 (2018) 2079–2091 2085

Fig. 6. BGP-NCM network.

Table 1
Voice & video network requirements.

One way latency Jitter Loss

Desktop share <1000 ms <100 ms <0.05%
Video conferencing <150 ms <30 ms <0.10%
IP telephony <150 ms <30 ms <0.10%

All links utilization graphs have been obtained from link moni-
toring tool that monitors enterprise links utilization. In BGP normal
operation enterprise 1st link traffic load will be increased till the
enterprise services offered through this link will be affected due
to the link becomes fully or over utilization.
In case of deploying BGP-NCM, when the 1st link utilization is
increased to exceed the 90% of link bandwidth (9.5 Mbps as shown
in Fig. 9) the BGP-NCM traffic calculation module has been trig-
Fig. 7. Packet end-to-end delay. gered and select the required IP prefix to be rerouted as well as
the enterprise 2nd internet link has been selected to carry this
amount of traffic because it is the link with less traffic load and
has an available bandwidth to accommodate the redistributed
traffic.
Fig. 12 shows the difference in enterprise router BGP configura-
tion before and after BGP-NCM configuration module has sent the
new BGP policies to redistribute the traffic where six route maps
have been configured two per each link and applied to carry out
the required traffic redistribution over available internet links.
After BGP routing policies has been applied, the traffic over the
1st link has been decreased to reach to about 4Mbps as shown in
Fig. 13 and the ellipse and arrow highlights the traffic change in
the graph.
The decreased traffic load over the enterprise 1st link has been
instantaneously appeared on the 2nd enterprise link and its traffic
loads becomes about 6Mbps as shown in Fig. 14. And accordingly,
1st link utilization has been kept under 90% of link bandwidth and
accordingly enterprise maintain its offered service level.
The previous experiment shows that the BGP-NCM has
Fig. 8. Packet jitter. enhanced the network performance parameters delay and jitter
2086 M.T. Moubarak et al. / Ain Shams Engineering Journal 9 (2018) 2079–2091

Fig. 9. ISP1 initial traffic load.

Fig. 10. ISP2 initial traffic load.

and offers proper utilization of available internet links compare
with normal BGP operation.
Table 2 shows the significant enhancement of BGP-NCM in per-
formance parameters delay and jitter compared with normal BGP
operation where in normal BGP operation the traffic load could
be increased to reach the 100% of link capacity and exceeds the link
capacity as shown in the above table resulting in unacceptable val-
ues of delay and jitter that accordingly affects any offered service
through this link while applying BGP-NCM keeps the traffic load
within the link capacity and keeps the delay and jitter values with
the acceptable ranges and accordingly maintains network perfor-
mance level for the offered service.
Table 3 shows the traffic load over the ISPs links for normal BPG
operation compared with BGP-NCM operation where in normal
BGP operation the traffic is increasing over the ISP1 link while
the ISP2 link has no traffic and all services offered over ISP1 link
will be affected when link becomes fully utilized or over utilized
while when applying BGP-NCM the traffic of ISP1 link traffic will
M.T. Moubarak et al. / Ain Shams Engineering Journal 9 (2018) 2079–2091 2087

Fig. 11. ISP3 initial traffic load.

Fig. 12. Enterprise BGP router configuration.

2088 M.T. Moubarak et al. / Ain Shams Engineering Journal 9 (2018) 2079–2091

Fig. 13. ISP1 redistributed traffic load.

Fig. 14. ISP2 redistributed traffic load.

Table 2 Table 3
BGP-NCM link performance parameter comparison. ISPS links load.

Protocol load Delay Jitter Protocol Load ISP1 ISP2 ISP3

BGP (100%load) >3000 ms >3 ms BGP >9.5 Mbps 0 Mbps <5 Mbps
BGP (>100%load) >3500 ms >240 ms BGP-NCM <4 Mbps <6 Mbps <5 Mbps
BGP-NCM (<100%load) <3 ms <1 ms
 M.T. Moubarak et al. / Ain Shams Engineering Journal 9 (2018) 2079–2091
be redistributed when the predefined threshold has been reached
and accordingly ISP2 link received an amount of ISP1 traffic to
maintain the services performance.
6. BGP-NCM implementation challenges
In this section, we will discuss some of expected challenges to
real implementation of the proposed BGP-NCM because of the nat-
ure of BGP protocol operation that already results in many perfor-
mance and security issues.
6.1. BGP-NCM performance challenges
Deploying BGP-NCM will issue a new BGP routing policy in case
of link over utilization or the utilization reached predefined value
to maintain the service level and accordingly BGP routers will send
update messages to their BGP neighbors to update their routing
tables and accordingly network performance may be affected
because as follows:
a. Multiple Network changes due to route updates or link fail-
ure leads to many BGP update messages this may result in
network instability and accordingly performance degrada-
tion due one of the following:
 BGP slow Convergence Where Convergence time is the
time required to router to start forwarding packets after
failure or change happens, as long as router received
update messages there is no traffic will be forwarded so
the increase of convergence time will decrease the net-
work performance.
 Route dampening penalty could be applied by ISPs on a
BGP neighbor due multiple routing updates received
form it due to multiple configuration changes or link flap-
ping to avoid the network instability problem.
BGP-NCM deployed the penalty timer to be checked before
applying any new configuration to control and prevent the contin-
uous update messages and accordingly avoid both slow conver-
gence and route dampening penalty problems.
b. BGP update messages has a great effect on some real-time
service as Voice over IP (VOIP), when a BGP update happens,
voice quality falls to an unintelligible level where call can’t
be established by caller for the time needed by router to con-
verge [36].
Based on the above BGP update messages will affect the voice
call quality and accordingly deploying the BGPNCM may affect
the voice calls or any similar service but also the link congestion
or link over utilization has same effect on voice calls and this effect
will be continuous as long as the link is congested or over utilized
not only for the time needed for router to converge.
BGP-NCM will enhance the performance where it is offered to
solve the link congestion or over utilization problems and its effi-
ciency is that the time of service degradation will not exceed the
convergence time which already can be maintained as minimum
as possible by applying consistent BGP configuration.
c. The 3rd challenge is concerning to AS Path prepending
(ASPP) technique, which is a popular technique for BGP
inbound traffic engineering and used by BGP-NCM to control
the inbound traffic but selecting the route based on AS path
length is not guaranteed 100% as well as it may cause rout-
ing loops so to minimize this mentioned risk network one of
the following can be done:
2089
 Simply network admin can check his prefixes path length
by consulting public route servers and accordingly deci-
des needed path length to ensure that new routing policy
issued by BGP-NCM will be effective and accordingly
BGP-NCM will be initialized by these values correspond-
ing to each prefix and that is what we considered in this
version of BGP-NCM.
 Despite the ASPP approach has been extensively prac-
ticed by many ASes, but still there are many researches
done to propose algorithms to allow more robust use of
ASPP such as a Greedy ASPP Search Algorithm for ISPs
to practice ASPP systematically mentioned in [37]or algo-
rithm for computing the optimal padding strategies given
multiple neighboring links in [38]and accordingly BGP-
NCM can use any one of the previous algorithms as well
as we plane to work on our own algorithm to enhance the
BGP-NCM in the future work.
6.2. BGP-NCM security challenges
BGP relies on TCP as its transport protocol. BGP is susceptible to
the same attacks that target any TCP-based protocol as well as
because BGP is an application is vulnerable to various threats
and the most popular attacks are as the following:
a. BGP Route Manipulation attack occurs when the contents of
the BGP routing table has been altered by a malicious device,
which can prevent traffic from reaching its intended destina-
tion without acknowledgement or notification.
Route manipulation attack can be mitigated by enabling the
MD5 neighbor authentication mechanism, to ensure that only
authorized peers can establish this BGP neighbor relationship,
and that the routing information exchanged between these two
devices has not been altered [39]
b. BGP Denial of Service (DoS)-occurs when unwanted BGP
traffic has been sent by a malicious host to a victim in an
attempt to utilize all available BGP or CPU resources, which
results in a lack of resources for valid BGP traffic processing.
BGP Denial of Service (DoS) attack can be mitigated by Time to
Live (TTL) security check when external BGP session (eBGP) is con-
figured, the IP header TTL for all neighbor session packets is set to
1. This setting was originally assumed to be useful because it pre-
vents the establishment of an eBGP session beyond a single hop.
The BGP TTL security check is not required nor is it considered use-
ful for internal BGP sessions (iBGP) [40].
c. BGP Route Hijacking-occurs when a rogue BGP peer mali-
ciously announces a victim’s prefixes to reroute some or all
traffic to itself for untoward purposes.
BGP Route Hijacking could be minimized by using both BGP
Neighbor Authentication with MD5, BGP Time to Live Security
Check Beside some other configuration actions like Configuring
Maximum Prefixes, Filtering BGP Prefixes with Prefix Lists, Filter-
ing BGP Prefixes with Autonomous System Path Access Lists and
AS Path Length Limiting to ensure that received routes are from
trusted neighbors [41].
Finally, BGP-NCM design has considered the above-mentioned
threats as follows:
 BGP-NCM controller doesn’t run BGP protocol or has any active
BGP session with BGP routers on the network.
2090 M.T. Moubarak et al. / Ain Shams Engineering Journal 9 (2018) 2079–2091
 BGP-NCM controller will be connected to the enterprise or ISP
private network and accordingly will not be exposed on the
internet to avoid DDoS attack or to be compromised and used
to change the routing tables and forwarded the traffic to other
destinations.
 BGP-NCM controller could communicate with BGP routers to
send the new BGP policy configuration over secured channel
based on IPSEC tunnel to add more security layer.
7. Conclusion
This paper presents a novel BGP control mechanism to be
engaged with the ossified inter-domain routing protocol (BGP)
[42]to be able to fulfill and manage the internet scalability
demands of bandwidth, services, applications, technologies and
of course users.
One of the key features that should be offered by the proposed
intelligent mechanism is to handle the internet traffic in smarter
manner to automatically redistribute the traffic over the Multi-
homed connections based on link utilization as well as without
any change or disturbance for existing internet system.
The results of deploying BGP-NCM control mechanism is to
enhance the overall BGP network performance where this paper
proposed traffic redistribution based on link utilization. This paper
offers the followings:
 More efficient use of available bandwidth over existing links.
 Keep the traffic within the link capacity.
 Maintains network performance parameters within acceptable
ranges.
 Improve performance and reliability with real-time optimiza-
tion for all IP traffic.
 Providing reliable routing quality without overprovisioning.
 Achieving the Service Level Agreement (SLA) of an enterprise or
ISP.
 Socio-economic impact where it saves bandwidth and accord-
ingly money to enterprise, ISP and then internet users.
8. Future work
 Link monitoring tool could be adjusted to monitor the latency
over the link instead of utilization and accordingly the thresh-
old value will present the maximum allowed latency over the
link based on the service offered over this link.
 Latency threshold is recommended to be used with real time
applications where the link could have normal traffic load but
suffers from high latency due to congestion in the other link
in the source to destination path.
 BGP-NCM could be deployed as Distributed Denial of Service
(DDoS) mitigation tool where in case of DDOS attack detection,
BGP-NCM will issue a corresponding routing policy to the BGP
neighbors to drop the DDoS traffic directed to the target IP
(Victim).
 We hope that BGP-NCM will be implemented and fine-tuned then
deployed as a service offered from tier one ISPs to tier two, tier
three and enterprises where BGPNCM servers will be hosted by
the tier one ISPs then other ISPs could register their multihomed
links over it. ISPs may accept traffic redistribution automatically
based on their predefined criteria such as link utilization or
latency. ISPs may receive updates concerning their network paths
congested links and they will act accordingly which leads to the
enhancement of the overall internet performance.
 BGP-NCM will be tuned for effective use of ASPP technique by
including his own ASPP algorithm for controlling inbound
traffic.
References
[1] Flash T. Reducing Web Latency: the Virtue of Gentle Aggression. ACM
SIGCOMM 2013 conference 2013;43(4):159–70.
[2] Rekhter Y. A Border Gateway Protocol 4 (BGP-4); 2006. [Online]. Available:
<http://www.ietf.org/rfc/rfc4271.txt>, [Accessed 22 10 2016].
[3] Gill P, Schapira SGM. Let the market drive deployment: a strategy for
transitioning to BGP security. Proceedings of the ACM SIGCOMM 2011
conference 2011;41(4):14–25.
[4] Jiang H, Dovrolis C. Why is the Internet traffic bursty in short time scales?
Proceedings of the 2005 ACM SIGMETRICS international conference on
Measurement and modeling of computer systems 2005;33(1):241–52.
[5] Santos-Boada G, Amazonas JRdA, Solé-Pareta J. Quality of network economics
optimisation using service level agreement modelling. Trans Emerg
Telecommun Technol 2016;27(5):731–44.
[6] Fujinoki Hiroshi. Analysis on ideal network structures to improve reliability by
multi-path and multi-homing BGP routing in the Internet. In: Ultra Modern
Telecommunications & Workshops, 2009. ICUMT ’09. International Conference.
[7] Fujinoki Hiroshi. Improving reliability for multi-home inbound traffic: MHLB/I
packet-level inter-domain load-balancing. In: IEEE Proceedings of the
International Conference on Availability, Reliability and Security ARES. p.
248–56.
[8] Zhang Y, Wang Y, Pei D, Yuan J. Multi-AS cooperative incoming traffic
engineering in a transit-edge separate internet. Comput Netw
2014;73:112–27.
[9] Sharma S. Demonstrating resilient quality of service in Software Defined
Networking. In: IEEE Proceedings of the International Conference on Computer
Communications INFOCOM. p. 131–4.
[10] Arins Andis. Latency factor in worldwide IP routed networks. In: IEEE
Proceedings of the 2nd Workshop on Advances in Information, Electronic
and Electrical Engineering (AIEEE). p. 1–4.
[11] Zhu M, Li D, Liu Y, Pei D, Ramakrishnan KK, Liu L, Wu J. MIFO: multi-path
Interdomain Forwarding. In: IEEE Proceedings of the 44th International
Conference on Parallel Processing (ICPP). p. 180–9.
[12] You L, Wei L, Junzhou L, Jian J, Nu X. An inter-domain multi-path flow transfer
mechanism based on SDN and multi-domain collaboration. In: IEEE
Proceedings of the International Symposium on Integrated Network
Management (IM). p. 758–61.
[13] Chen Z, Bi J, Fu Y, Wang Y, Xu A. MLV: A Multi-dimension Routing Information
Exchange Mechanism for Inter-domain SDN. In: IEEE Proceedings of the 23rd
International Conference on Network Protocols (ICNP). p. 438–45.
[14] Lakshminarayanan K, Stoica I, Shenker S, Rexford J. Routing as a Service
Technical Report 2004. Computer Science Division, University of California;
2004. <https://www.cs.princeton.edu/~jrex/teaching/spring2005/reading/
ras04.pdf., California>.
[15] Kotronis V, Dimitropoulos X, Ager aB. Outsourcing the routing control logic:
better internet routing based on SDN principles. In: ACM Proceedings of the
11th Workshop on Hot Topics in Networks. p. 55–60.
[16] Mohasseb Y, Moubarak M. A QoS oriented analytical model for BGP
multihomed networks. In: IEEE Proceedings of the 16th International
Conference of Mediterranean Electrotechnical (MELECON), Yasmine
Hammamet, Tunisia.
[17] Astuto B, Nunes A, Sophia-Antipolis, Mendonca M, Nguyen X-N, Obraczka K. A
survey of software-defined networking: past, present, and future of
programmable networks. IEEE Communications Survey & Tutorials 2014;16
(3):1617–34.
[18] Heller B, Sherwood R, Mceown Nick. The controller placement problem. In:
Proceedings of the First ACM Workshop on Hot Topics in Software Defined
Networks HotSDN. p. 7–12.
[19] CentOS Linux, Community Enterprise Operating System (CentOS), [Online].
Available: <https://www.centos.org>, [Accessed 22 October 2016].
[20] Schroder C. Linux Networking Cookbook. O’Reilly Media; 2007.
[21] Quagga Routing Suit, [Online]. Available: <http://www.nongnu.org/quagga/>,
[Accessed 22 October 2016].
[22] Oetiker’s T. The Multi Router Traffic Grapher, [Online]. Available: <http://
www.oss.oetiker.ch/mrtg/>.
[23] Oetiker T. Monitoring your IT gear: the MRTG story. IT Professional IEEE
Computer Society 2001;3(6):44–8.
[24] Mauro D, Schmidt K. Essential SNMP. 2nd ed. O’Reilly Media; 2005.
[25] Harrington D, Presuhn R, Wijnen B. An Architecture for Describing Simple
Network Management Protocol (SNMP) Management Frameworks. The
Internet Society; 2002 [Online]. Available: <http://tools.ietf.org/html/
rfc3411> [Accessed 22 October 2016].
[26] Paython, [Online]. Available: <https://www.python.org/>, [Accessed 22
October 2016].
[27] Gordon M. An introduction to network programming the Python way. IEEE
Distributed Systems Online, vol. 6, no. 10; 2005. p. 5–7.
[28] Chen Z, Chen L, Zhou Y, Xu Z, Chu WC, Xu Baowen. Dynamic Slicing of Python
Programs. In: Proceedings of the IEEE 38th Annual Computer Software and
Applications Conference (COMPSAC). p. 219–28.
[29] Redondo JM, Ortin F. A comprehensive evaluation of common python
implementations. IEEE Software IEEE Computer Society 2015;32(4):76–84.
[30] Hofstede R, Celeda P, Trammell B, Drago I, Sadre R, Sperotto A, Pras A. Flow
Monitoring Explained: From Packet Capture to Data Analysis with NetFlow
and IPFIX. IEEE Communications Surveys & Tutorials 2014;16.
 M.T. Moubarak et al. / Ain Shams Engineering Journal 9 (2018) 2079–2091
[31] Cisco. Introduction to Cisco IOS NetFlow - A Technical Overview. Cisco; 2012
[Online]. Available: <http://www.cisco.com/c/en/us/products/collateral/ios-
nx-os-software/ios-netflow/prod_white_paper0900aecd80406232.html>
[Accessed 22 October 2016].
[32] Rekhter Y, Li T. A Border Gateway Protocol 4 (BGP-4); 1995. [Online].
[Accessed 22 October 2016].
[33] Li H, Lin X. OPNET-based 3-tier network simulation architecture. IEEE
Communications and Information Technology 2005;2:793–6.
[34] Siraj S, Gupta A, Badgujar R. Network simulation tools survey. Int J Adv Res
Comput Commun Eng 2012;1(4):199–206.
[35] Houle JD, Ramakrishnan KK, Sadhvani R, Yuksel M, Kalyanaraman S. The
Evolving Internet - Traffic, Engineering, and Roles. In: The 35th TPRC
Research Conference on Communication, Information and Internet Policy,
Arlington.
[36] Kushman N, Kandula S, Katabi D. Can you hear me now?! it must be BGP. ACM
SIGCOMM Comput Commun Rev 2007;37(2):77–84.
2091
[37] Wang J, Chiu D, Lui J, Chang RKC. Inter-as inbound traffic engineering via ASPP.
Transactions on Network and Service Management 2007;4(1):62–70.
[38] Zhang Y, Tatipamula M. Characterization and design of effective bgp as-path
prepending. In: Network Protocols (ICNP) 19th IEEE International Conference.
p. 59–68.
[39] Touch J, Mankin A, Bonica R. The TCP Authentication Option. Internet
Engineering Task Force (IETF); June 2010 [Online]. Available: <https://tools.
ietf.org/html/rfc5925> [Accessed 22 October 2016].
[40] Gill V, Heasley J, Meyer D. The Generalized TTL Security Mechanism
(GTSM). The Internet Society; 2004 [Online]. Available: <https://tools.ietf.
org/html/rfc3682> [Accessed 22 October 2016].
[41] Protecting Border Gateway Protocol for the Enterprise, Cisco, [Online].
Available: <http://www.cisco.com/c/en/us/about/security-center/protecting-
border-gateway-protocol.html#7>, [Accessed 22 October 2016].
[42] NetVolution, European Research Council (ERC); 2014. [Online]. Available:
<http://netvolution.eu/scientific-approach.html>, [Accessed 22 October 2018].