Computers and Electrical Engineering 84 (2020) 106634

Contents lists available at ScienceDirect

Computers and Electrical Engineering

journal homepage: www.elsevier.com/locate/compeleceng

A Distributed Software Defined Networking Model to Improve

the Scalability and Quality of Services for Flexible Green
Energy Internet for Smart Grid Systems
Kashif Naseer Qureshi 1, Raza Hussain 1, Gwanggil Jeon 2,3,∗
1
Department of Computer Science, Bahria University, Islamabad, Pakistan
2
Department School of Electronic Engineering, Xidian University, Xi’an, China
3
Department of Embedded Systems Engineering, Incheon National University, Incheon, Korea

a r t i c l e i n f o a b s t r a c t

Article history: Green energy Internet is a new concept for future power systems based on high-level in-
Received 31 December 2019 terconnection among different systems for energy efficient delivery and manage energy
Revised 29 February 2020
resources. Smart grid is one of the core energy management system based on multiple
Accepted 9 March 2020
routing and data transferring services for energy management, distribution, pricing, and
Available online 21 April 2020
demand services. The traditional energy networks have suffered from high and complex
Keywords: computing operations and lead to degradation of services and the overall performance of
Software defined networking all operations. Software-Defined Networking (SDN) is one of the platform to enhance the
Internet of Energy flexibility and efficiency by decoupled the control plane from its data plane. A detachment
Internet of Things of these two-planes improved dynamic configuration, controller programmability, central-
Quality of services ized and decentralized network operations. In this paper, a distributed SDN approach is
Scalability used to address the scalability and robustness issues and improve the overall energy ef-
Network ficiency in smart grid systems. The SDN controller logically centralized but it works on
multiple nodes that physically distributed. Distributed nodes are communicating with each
other and improve the network performance to assist data flow. When flow arrives through
end devices to any data plane in distributed SDN, the proposed approach supports the
data plane to increase the controller response time. Also, the controller checks, whether
the data needs to prioritize flow to send it locally or globally with the help of Elephant
or Mice flow. The proposed approach is used to reduce the controller workload, response
time, and manage Quality of Services (QoS) and enhanced energy efficiency among con-
trollers. This approach also improves the green energy utilization in smart grid systems.
The experimental results indicate that the proposed approach is more suitable and feasi-
ble for smart grid systems.
© 2020 Elsevier Ltd. All rights reserved.

1. Introduction

With the passage of time, the usage of electrical elements increases the energy demand and more powerful and inte-
grated smart grids. The new integrated power systems provide and offer energy-efficient solutions, energy storage capa-
bilities and distributed into different scales. Large scale energy management systems lead to various challenges including

∗
Corresponding author.
E-mail address: ggjeon@gmail.com (G. Jeon).

https://doi.org/10.1016/j.compeleceng.2020.106634
0045-7906/© 2020 Elsevier Ltd. All rights reserved.
2 K.N. Qureshi, R. Hussain and G. Jeon / Computers and Electrical Engineering 84 (2020) 106634

scalability, energy consumption, and degradation of services. Different types of energy control and management architec-
tures have been designed for smart grids to address these challenges. One of the popular terms is Internet of Energy (IoE)
which has gained considerable attention to support the interconnection of distributed energy resources on a large scale [1].
Energy routers or power flow routers are the key components of IoE networks based on energy-efficient data routing solu-
tions. In these networks, Software Defined Network (SDN) is the immerging approach which drives the network operations
independently with the help of detaching control and forwarding plane. On one side, the flow forwarding device turns out
to be extremely well organized and programmable such as data plane. On the other hand, an intelligent forwarding machine
acts like a single entity to perform and control all network traffic like control plane. SDN infrastructure has a great impact
on networking applications especially for IoE networks, most specifically for energy consumption, load balancing, energy
transmission, processing, and storage capabilities [2-4]. End devices are connected and generate a massive amount of data.
These processes make a great impact on network capacity and energy utilization. The underlying network requires flexible
capabilities and parallel processing for data handling [5]. Distributed computing requires massive growing network-based
applications. Controlling network devices and individual processing of SDN is the best approach for software-based appli-
cations for IoE systems. We presented an efficient use of the emerging SDN platform powered by the Industrial Internet of
Things (IIoT) platform for the smart grid paradigm, which is currently considered as one of the challenge. To overlay the
path towards the SDN employed IIoT integrated smart grid, the use of OpenFlow protocol to integrate all the layers and
development of efficient control algorithms with low complexity is another challenge. In this regard, the new SDN platform
along with a developed algorithm is proposed to adopt dynamic change in a real-time smart grid systems for reliable and
flexible operations to enable the control layer by providing real-time monitoring [6]. To verify the feasibility of the devel-
oped system, different use cases are presented (e.g. expansion and fault scenarios) in the smart grid paradigm to evaluate
the data traffic flow and forwarding behavior. According to the demonstrative scenarios, this paper shows the potential of
the SDN controller for supporting the resilience of smart grids, even under fault circumstances.
The Internet is a combination of data units in IoE networks for data processing. To avoid data delay or error in flow traffic,
a large data center requires a suitable network configuration. Traditional network architecture is not efficient to provide all
facilities and handle the delay issues. A combination of two planes can be done by the southbound application program
interface and controller to application layer interaction which can be done by a northbound application program interface as
shown in Figure 1. Network state monitoring deals with two main aspects, real-time traffic, and abnormal node connection.
Real-time traffic involves the adjustment of network statistics and well-timed detection of network congestion. When the
controller controls more than one network domain, it should notice abnormal statistics. This abnormal node helps to adjust
the network topology structure like an abnormal node connection. The OpenFlow protocol controller helps to data plane
to maintain a forwarding table. However, sometime the controller may not have all flow information to route the network
because when topology change in SDN, link discovery protocol must be modified. Figure 1 shows the grid resilience for SDN
based IIoT.
Underlying topology always dynamic and never be fixed. The method proposed to add some threshold limit on the switch
to manage controller response time. Without a threshold switch, the data plane can reduce controller outcome. Nevertheless,
the reactively coming flow rate is unpredictable. Data out rate of switch differs because reactively traffic patterns are hard
to predict. Sometimes large data flow increases by the limit that can affect the controller performance. The topology of
the network can be changed in case of large data failure or to provide high availability of the resource [7]. Fixed topology
can be helpful but not viable completely. Controller response time and delay are dynamically changed in load conditions.
The main reason behind this is the different traffic models that are difficult to fulfill the QoS requirement and leads to
energy consumption issues. To achieve a better response time, the controller requires flow resources where the response
time between traffic flows should be minimum [8]. Because of this, we need many controllers to facilitate the smart grid
networks requirement. Many controllers will increase cost but they will facilitate flow. High traffic and maintenance requires
the addition and deletion of the controller in the network. The research has been done related to controller capacity but
the controller response time is more important to transfer the flow. As in [9], the authors explained that end-to-end delay
is a more important factor. We cannot compromise with performance when data flow arrives on the controller, its basic
need is to route the traffic in time. Controller response time should be minimizing to perform QoS requirements as focused
on controller capacity. Underlying topology always dynamic and never be fixed. The proposed methodology is to add some
threshold limit on the switch to manage controller response time [10]. Without a threshold switch, the data plane can
reduce controller outcome. However, the reactively coming flow rate is unpredictable. Data out rate of switch differs because
reactively traffic patterns are hard to predict. Sometimes large data flow increases the limit of threshold that can affect
the controller performance. The topology of the network can be changed in case of large data failure or to provide high
availability of the resource. Fixed topology can be helpful but not viable completely. Resource integrate accordingly control
plane because traffic flow and size behavior change regularly. Controller response time and delay are dynamically change in
load condition. The main reason is different traffic model to fulfill the QoS requirement.
The main apprehension is to deals with SDN and Controller Selection Problem (CSP). The controller selection problem
is more important than the controller placement problem [11]. When flow arrives at any data plane from end devices in
distributed SDN, there should be a specific limit on data-plane flow to increase controller response time. Data plane checks
different IP addresses in the table to select a controller based on trust level, distance, and bandwidth. The flow arrives in the
controller and check, whether data need to prioritize to send locally or globally. Based on the flow and QoS requirements,
the engine prioritizes each flow. Different algorithms have been proposed like Mice and Elephant. When prioritize process
 K.N. Qureshi, R. Hussain and G. Jeon / Computers and Electrical Engineering 84 (2020) 106634 3

Figure 1. Grid Resilience for SDN based IIoT

assigns ranked for flow in the controller, it decides the batter flow to serve the QoS requirements. For interconnectivity and
exchange massage between peers, controllers can be done by SDN inter message through Border Gateway Protocol (BGP).
Peer controller update and collects with the help of the state collection control function. When flow arrives on the controller,
it traffics the flow on time.
The single controller in the network has limited scalability and computing power. In large networks, computing oper-
ations and traffic flow are very high. A single controller faces several concerns like scalability, robustness, and bottleneck.
The proposed method introduces the topology independent framework to enhance the control plane with manipulative the
best number of controllers to decrease the capacity and improve the QoS. This work is an enhanced version of our previous
research in [12]. To achieve resiliently and scalability in communication smart grid with SDN is proposed in various liter-
ature. Authors in [13], handle the bundle of demand from IIoT SDN which can provide seamless data transmission for the
smart grid environment. The authors explained that the SDN control plane is flexible to handle smart grid demand in bulk.
In smart grid environment, it can sustain properly. In case, where the smart grid with SDN can be used for shifting and load
4 K.N. Qureshi, R. Hussain and G. Jeon / Computers and Electrical Engineering 84 (2020) 106634

balancing, smart grid commands to control the path of routing for adjusting the dynamic traffic, scheduling of critical smart
grid traffic flows, self-healing and fast failure detection need to discuss more. Furthermore, SDN is one of the demanding
method to serve and the full need for smart grid.
This study main objectives are as follows:

• Divide single SDN into distributed network domains to achieve scalability for smart grids networks.
• All controllers are connected to each other and responsible to manage their domains, to update global state.
• Comparison between SDN controllers with a threshold.
• To improve the energy efficiency by using proposed SDN based architecture for IoE networks.

The rest of the sections are divided into related work, the proposed model, experimental results and in the last paper
concludes with future direction.

2. Related Work

In this section, we discuss existing literature on SDN networks. Authors in [14] presented a heuristic base algorithm
for shifting a load of controller dynamically, through the switch to the controller and between data plane for average flow
request. But the author didn’t explain statistics, rules, and flow between controllers. In another study [15], the authors pro-
posed a general approach for distributed controllers for achieving scalability. But they didn’t explain the challenges related
to controller placement, number of optimal controllers, and distribution of load between controller. Authors in [16] showed
a data collection system for vehicle networks. They focused on the collection and delivery when there are a large number
of data. Based on a mixture of cooperation between mobile networks and ad hoc networks. The bottleneck problem is the
main concern in traditional networks. Because when data arrive in a traditional network, it is difficult to address massive
data transmission. They highlighted some important points like limited knowledge related to SDN and large volume. SDN
can provide QoS when a large volume of data flow arrives in the network to achieve QoS for large data applications and
assert analytics.
In another study in [17], the authors explained two major domains data analytics and SDN characteristics for introducing
the central network system. This scheme provides the decisions base knowledge from the third party. The analytical module
combines with SDN control plane together data from the network. This scheme facilitates all network information with help
of distributed message gathering system called Apache Kafta and distributed Database with Apache Cassandra. Knowledge-
base decision facilitates data with the help of data analytics. In another study [18] authors presented an application layer
centralized the logic for call interface provide between control networks. Resource allocation turned to provide by SDN in
the network. Network resource becomes more delegate capable and increases network flexibility. The authors in [19] pro-
posed controller placement problems in SDN with the help of a mathematical model. This model is used to determine the
controller type, location of the controller, and an optimal number of controllers for a specific set of switches. The authors
also presented the method of minimization of elements cast for the network.
More work has done in [20] to optimized the model and used for the bound algorithm as the optimizer. But it causes the
accuracy and consistency of the model due to adding multiple factors. The proposed social TV for data analytics with the
help of the SDN platform. The main goal behind this is to find a TV program that accepts the audience’s requirement with
distributed domains. They used Hadoop and cluster statistics execution. Authors combined SDN and big data with social
TV programs. Microblog data-based program is used to collect people thoughts, opinion, and perception. But in this model,
it is difficult to distinguish structured and unstructured data. Detection algorithms in the uncontrolled domain, in which
one willing to provide anomaly explicit support. DNA methodology used to provide data networks to the controller and
highlights the statistics generation. When communication takes place in the controller, it required high bandwidth and it
also cases overhead controller for each process. Data resource configures to provide local networks to support data analytics,
in which they have used an agent-based approach.
Authors [21] proposed a scheme to reduce transmission delay for smart devices. Smart devices generate huge amounts
of data and they faced the reduction of dimensionality problem. The scheme used reconstruction error minimization with
the help of the Frobenius norm. But authors didn’t explain how to handle large data from a single SDN controller. When
a single controller becomes overloaded with the large flow. Bottleneck and scalability issues are not highlighting in their
study. In another study [22], the authors presented an approach in traffic engineering to facilitate the SDN. In this scheme,
the network optimization and configuration in SDN controllers to provide quality of services is presented. Big data in the
computing domain is based on a large scale is MapReduce. TCP/IP based approach is used to provide a map phase for
reducers. The main issue is, when large data arrived, high-speed case problem to manage optimization and speed cannot
accommodate in this condition. In another study [23], a Pythia model is proposed for MapReduce to enable communication
for predicting the system application. It also allocated the optimized bandwidth for network application and configuration.
But application-aware and MapReduce approaches have been designed multiple times for the SDN system. For services
management, authors in [24] proposed the AWESoME prototype to facilitate the SDN. Originating services explained in detail
for the concept of DNS and blog domains for the initial packet association. Grained control defined for marginal load and
accurate flow enabling. So, the SDN data plane and control plane work practically. Table 1 shows the literature review on
SDN.
 K.N. Qureshi, R. Hussain and G. Jeon / Computers and Electrical Engineering 84 (2020) 106634 5

Table 1
Comparison of SDN Schemes

Scheme SDN Architecture Objective Limitations

Distributed controllers [15]
Preco [16]
Tensor Based Model [17]
Seer [18]
AAN-SDN framework [19]
Mathematical model [20]
SDN Social TV analytics [20]
Clustered distributed SDN [23]
BASS [22]
Pythia [23]
AWESoME [24]
Floodlight SDN controller Distributed controllers for achieving Didn’t explain Challenges of Distributed
scalability controllers
Software Defined Predictive routing for Ad hoc relay mode Repeated flow traffic blocking when
Vehicular Networks multi-hop large data evolve
T-SDN Network traffic prediction and QoS Minimal work on exploiting the
provisioning properties of SDN
ONOS SDN SDN & Big data framework provide Holistic method to deploy every module
intelligence in network with scale up
Centralized SDN SDN and AANs network underlying Large scale deployment issue in test
function, forwarding logics with M/R case in big data analytics
Controller Placement Determine type of controller, location of Causes the accuracy and consistency of
Problem controller and optimal number of model
controllers
SDN Micro blog data extracts the knowledge Challenges for unstructured or
and public perception in TV program structured data
Distributed controller To improve network performance, its Throughput rate, for flow setup when
scalability and its reliability No, of cluster are large
Cluster centrally Bandwidth-Aware Scheduling with SDN Huge amount of data still takes a lot of
controlled SDN in Hadoop time to transfer.
SDN Optimize bandwidth allocation lacks of clear and comprehensive SDN
design with MapReduce
SDN To facilitate web services to important Security for application, service
traffic, prioritize and identify accounting

2.1. Discussion

In large smart grid networks, organizations and data management is one of the main concerns. SDN is one of the best
approaches to fulfill network demands. SDN can benefit end devices by managing structured and unstructured data like
online streaming, online gaming, and social media. All end devices have data features to facilitate SDN. SDN can perform
intelligent networking to manage the device’s data with a more customizable and scalable manner. The main apprehension
is to deal with SDN and Controller Selection Problem (CSP). The controller selection problem is more important than the
controller placement problem. When flow arrives at any data plane from end devices in distributed SDN, there should be
a specific limit on data-plane flow to increase the controller response time. The Data plane checks different IP addresses
in the table to select the controller based on trust level, distance, and bandwidth. Flow arrives in controller and checks,
whether data need to prioritize to send locally or globally. Based on the flow and QoS requirement engine prioritize each
flow. Different algorithms have been proposed for this like Mice and Elephant. When prioritize process assign ranked for
flow in the controller. It decides what is batter for flow to serve and fulfill the QoS requirement. For interconnectivity and
exchange massages between peer controllers which can be done by SDN inter message through BGP. Underlying connected
data plane and peer controller have used inter SDN modules for communication. Peer controller updates and collects with
the help of state collection control function. When flow arrives on the controller its basic need is to traffic the flow on time.
Controller response time should be minimum to perform the QoS requirement. The existing schemes are not able to provide
the aforementioned QoS requirements.

3. Proposed Architecture for Smart Grids

This section presents the proposed model for Distributed SDN for smart grids. Many attempts have been done related
to controller capacity, but the controller response time is more important to transfer the data flow. We cannot compromise
with performance but cost and resources are affordable. When data flow arrives on controller its basic need is to traffic the
route on its time. Controller response time should be minimizing to perform QoS as focused on controller capacity. SDN can
control smart grids network inefficient way with the help of centralized and decentralized discipline of the control plane.
Therefore, SDN is a programmable solution for data centers to facilitate the bulk of data from the smart grids network.
When traffic flow and data managed by different platforms so, it’s one of the solutions for grid management with the help
of SDN in data centers. To the best of our knowledge, there is a limited work of IIoT in a smart grid based SDN platform to
monitor, natural disasters, electric outage, and fault events. Unlike existing approaches, our model comprehensively captures
the traffic flow of a typical smart grid in 5G networks employing the SDN platform. Thus, we tackle the above problems by
presenting a new vision of the SDN platform for smart utilities to provide real-time data monitoring by performing control
functionality that can give support to IIoT objects in case of fault and outage scenarios.
6 K.N. Qureshi, R. Hussain and G. Jeon / Computers and Electrical Engineering 84 (2020) 106634

Table 2
Used Symbols and Definitions

Symbols Definitions

k Total access switches

μ Processing average flow rate of controller
λ Average new flow arrival rate in each switch
S Processing average flow time of controller
T Service time average flow
N Queue average size

3.1. Mathematical Model

This study discusses the system modeling with the help of a mathematical model where analyzes the response time
to present the resource requirement utilized by the proposed model. The proposed model is simulated with the help of
Opendaylight, Mininet, BGP, and MATLAB. Some previous models have evaluated the controller response through M/M/1
network model and extend through the M/M/m discipline to evaluate Distributed SDN performance mathematically [25].
All SDN communication deployed in hierarchical architecture. Some other studies also discussed that M/M/m is the op-
timal way to show distributed results of the SDN process including memoryless and additive properties [26]. The flexibility
and capabilities of each entity in SDN to manage the data flow can make it a great fit under IIoT scenarios. In comparison to
our work, several works have elaborated on monitoring systems and fault recovery in grid network-based SDN architecture.
Different from existing methodologies, our system model comprehensively considers both smart grid networks and the SDN
platform for the traffic flow scenarios. For example, function management on SDN switches is assumed in the smart grid
environment by considering a centralized SDN controller employed communication protocol messages to detect the fault
or outage link and trigger re-establishment activities. A similar idea of local outage in the smart grid network to be re-
established has been presented by applying flow table admissions to support the fault link for recovery. So, with the help
of background mainstream, we can use mathematical Equation 1 for analysis to prove the strategy for Distributed SDN. In
which, incoming flow packets are considered and fallow the distributed Poisson constraint which has two processes given
justification for different time scales [27]. Furthermore, we are also considering exponential distribution for each controller
for service rate, this is normal behavior to analyze the queuing model and existing models. Table 2 shows the symbols and
their definition.

T (t ) = k + 1/2(μ − λk )M/M/1
 
T (t ) = p0 ∗ (m ∗ p)m/m !(1 − p) M/M/m
       −1k  −11−1
0
p = 1+ m−1 k = 1 ∗ (mρ ) k/k
! + (∞ ) (k = m) ∗ (mρ ) ! ∗ 1/ m − m
k/k k
−m (1)

3.2. Proposed Algorithm

For control the congestion of SDN/OpenFlow network, the passive detection is used where the SDN presents the con-
gestion level for OpenFlow switches such as whenever the link utilization is higher than a 90% threshold then the message
sends to SDN controller [28]. After receiving the messages, the controller is parsed to contain the following information:

• Switch ID (data path identification or DPID) for congestion location.

• Byte counters for Big flows in the congested link.

After this, the SDN evaluates the counter for the flow meter of congested switch to search the occupied flows in link
utilization or for video flows. The proposed algorithm is summarized where it detects the aggregated elephant flow for
traffic engineering.
If congestion detected, then the switch sends the congestion notification message to the controller through the Packet_in
messages in the OpenFlow protocol. In Algorithm 1, it is essential to sort all the flows in the flow set of the switch flow
table. The flows are sorted based on their size or importance. Only important video flows may be considered for simplicity
or in case of limited network the resource. Then the algorithm detects a single flow or flows as an aggregated elephant flow
when their flow is Bytes ≥ THRESHOLD.

3.3. Proposed Network Model

The network model is based on three main steps. The first step is implemented in MATLAB concerning the m/m/1 the-
orem which has 30 switches and one controller to understand the working of the real scenario of the network. MATLAB is
one of the best tools to analyze SDN networks. Single controller response time effected by maximum switches. To overcome
this issue, we are using a Distributed SDN controller strategy to scale the network. For this scenario, we are using M/M/m
 K.N. Qureshi, R. Hussain and G. Jeon / Computers and Electrical Engineering 84 (2020) 106634 7

Algorithm 1
Aggregated elephant flow detection algorithm

Input: The flow Table entry set (OF Flow Stats Entry) of the congested switch which is collected
through the Statistics Collector module of the SDN controller.
Output: An aggregated elephant flow:
ret=1 when a single flow as an elephant flow;
ret=2 when aggregated flows as an aggregated elephant flow.
The flow set F in the aggregated elephant flow.
1: ret=0;
F=null;
Flowentryset=StatisticsCollector.OF FlowStatusEntry;
Flowentryset.sortbybytesize;
2: FOR flowj .€ flowentryset do
3: IF (flowj .bytes ≥ THRESHOLD) THEN
4: RETURN ret=1;
5: FOR flowi € flow entryset and flowj € flowentryset do
6: IF (flowi.srcDPID=flowj .srcDPID)
or flowi .dstDPID=flowj .dstDPID THEN
7: F.add(flowi ) and F.add(flowj );
8: IF(F.bytes≥THRESHOLD)
TEHN
9: RETURN ret=2;

Figure 2. Network Topology

theorem to explain the real environment simulation with the help of the graph. The second step is utilizing elephant or
mice flow to prioritize the flow. Because of elephant flow, we can detect network topology. After that when flow arrives in
the controller and it checks, whether data need to prioritize to send locally or globally. Also, it reduces controller response
time as well. In the third step, we are using SDNi messages between the controllers to improve the performance with the
help of border gateway protocol. In which, we use the desired topology to simulate or particular situations as shown in
Figure 2.

3.4. Proposed Method

The proposed method deals with SDN and Controller Selection Problem (CSP). The controller selection problem is more
important than the controller placement problem [29]. When data flow arrives at any data plane from end devices in Dis-
tributed SDN, a limited data plane that should be set to increase controller response time. Data plane checks different IP
addresses in the table to select the controller based on trust level, distance, bandwidth, etc. Flow arrives in the controller
and it will check, whether data need to prioritize to send locally or globally. Based on the flow and QoS requirement engine
prioritizes each flow. Different algorithms have been proposed for this like Mice and Elephant.
When prioritize process assigns ranked for flow in the controller. It decides about better flow to serve and fulfill the QoS
requirements. For interconnectivity and exchange massage between peer controllers which can be done by SDN inter (SDNi)
massage by using BGP. Underlying connected data plane and peer controller used inter SDN module for communication. Peer
controller update and collects with the help of state collection control function as shown in Figure 3. When flow arrives on
the controller its basic need is to traffic the route on time. Controller response time should be minimum to perform the QoS
requirement. QoS and dynamic workload requirement needs for logically adding and deleting controllers in the network. The
methodology is supportive for invoke and revoke several logical controllers.
8 K.N. Qureshi, R. Hussain and G. Jeon / Computers and Electrical Engineering 84 (2020) 106634

Figure 3. The block diagram of proposed solution

3.5. Process Model

In this section, the physical model presents to achieve the research objectives. This study’s main concern is evaluating
the real-datasets and estimate the analytical theory to confirm the findings. It provides a virtual image of the data plane and
control plane as well. We can configure both planes according to the study requirement. We can configure many devices as
we want to emulate an SDN network with the help of MATLAB. For sustainability, the multiple controllers are able to control
fault tolerance of controller, scalable, have load balancing capabilities, and reduces the latency to its closest controller.
Furthermore, this study analyzes the output of multiple control planes. To provide modeling the flow set-up requirements
through data plane to control plane for the group of coming process M/M/m with m control planes, the transition diagram
is showed in Figure 4.

3.5.1. SDN Controllers Communication

Distributed SDN can be achieved by using the vertical or horizontal method. In which we are focusing on a horizontal
approach. Distributed SDN controllers can be established with the help of peer-to-peer controller communication, as can be
seen in Figure 5. Any controller wants to request for connections or information through neighbors, in its domain controllers
request from other domains in the network. It can be achieved east-west interface SDNi by using BGP that is simulated
in Packet Tracer. The horizontal method is more feasible and preferable for geographies ascending across the network, as
control planes in this model can communicate with the help of BGP to each other by standard and friendly protocol. This
 K.N. Qureshi, R. Hussain and G. Jeon / Computers and Electrical Engineering 84 (2020) 106634 9

Figure 4. State diagram of M/M/m

Figure 5. Inter-SDN Horizontal Approach for Controller Communication

method retains the SDN controller independent in the network, with path setup and distinct policies to relate to the network
elements in its control, while preserving a confederation with the adjacent networks domain.
BGP is a famous routing protocol that required to share the routing material through two autonomous systems in the
network. BGP is the model protocol and it can be used to adapt for controller communication in the inter-SDN environment.
Whereas, many SDN network domains have control and access to policies, QoS, and parameters that are useful for the SDN
control planes. BGP has one of the features that are used for controller negotiation through the east-west interface for SDN.
It can transmit the reachability capability and information as a message portion to their format. It is one of the most feasible
and standard protocol for any peer to peer information to be replaced.

3.5.2. Flow scheduling System Framework

Traffic engineering and traditional flow are based on IP. SDN/OpenFlow with the numerous forms of granularity like
source and destination data plane, application types, and VLAN. Hedera introduced, elephant flow in each single and large
flow completed a certain (10% of the link) duration and size. That also identified with the help of the same 10-tuple in
the network. The whole account for internet video flows as for the most Cisco VNI network traffic as the report predicts.
Whereas, all flow is much fewer in the video than 10% inflow to the capacity of the link. Elephant flow as different to the
individual, this study used the SDN/OpenFlow to aggregate big data flow into a labeled aggregated for elephant flow where
they flow into one aggregated elephant when flows shared the congested links and shared its path. Secondly, when the big
flows (video flows) are bigger than 1% of the congested link.

4. Experimental Results

Firstly, we discuss the simulation setup. In the first step, the proposed method implemented in MATLAB concerning
m/m/1 theorem. Single controller response time effected by maximum switches.

4.1. Assumption and Limitations

The following assumptions are taken into account while considering the entire scenario of the proposed scheme.

• This work does not consider any real application traffic.

• In random topologies, no optimization of the controller placement has been performed.
• The results are based on abstract topologies and no real internet topology has been considered.

4.2. Simulation Setup

The first step is evolving around network topology. In which we have centralized SDN that is implemented on the Open-
daylight platform with coordination of Mininet. We are assuming the topology that is independent of the network. The main
purpose is to explain how elephant flow evolves in any SDN topology. In this phase, we have re-implemented the work to
test load balancing technique to manage the elephant flows in data centers using SDN technology. We have compared their
approach of effectively assigning the link weights to adjust the data flow amongst the links with the Single Shortest Path
Routing technique. We used the OpendayLight Controller in OpenFlow testbed and observed that Weighted Multipath Rout-
ing outperforms the Single Path Routing, especially in the case of Elephant Flows. We observed that their approach performs
10 K.N. Qureshi, R. Hussain and G. Jeon / Computers and Electrical Engineering 84 (2020) 106634

Figure 6. ODL-SDNi working

Figure 7. SDNi Rest API Output

in case of heavy flows, effectively overcoming the network congestion and improving network utilization. Thus, it will pro-
vide better QoS to the customers. We only added the optional feature once which are permanently part of this controller
and run whenever it starts. Now, we present a network simulation with the help of pf Opendaylight and Mininet. All steps
of the simulation setups are as follows:
Step 1: In first step, the installation of Opendaylight and Mininet is conducted.
Step 2: With the collaboration of both tools we assign ip to SDN mininet controller that call in Opendalight
Step 3: After installing features of Opendaylight and mininet topology, we set the topology in web browser.
Step 5: After running the topology and getting its all statistics, we can Capturing OpenFlow Messages with the help of
Wireshark.
Step 6: In last step, we have MATLAB graph to demonstrate behavior of response time to control the multiple switches.

4.3. SDNi Distributed Approach

OpenDaylight is one of the popular controller designed by Cisco. It has an open-source SDN controller for ODL commu-
nity as shown in Figure 6. This controller also has a architecture based on Services Abstraction Layer (SAL) concept. As per
our knowledge, we use OpenDaylight first time as SDN controller. Figure 7 shows the SDNi Rest API Output.
Wrapper Features:
• SDNi Wrapper utilizes the existing ODL-BGP Plugin.
• Enhanced the NLRI update message (of BGP) for capability data
 K.N. Qureshi, R. Hussain and G. Jeon / Computers and Electrical Engineering 84 (2020) 106634 11

Figure 8. BGP working

Figure 9. Wireshark capture of an open message between R1 and R2

• This data to be exchanged available through the RestAPIs that are developed.
• Wrapper to read and store this data in a database (SQLite).
• Each controller to have peer data for the controllers in a session over real-time.
• The data exchanged can be restricted (based on security)

4.3.1. BGP Protocol

After implementing centralized SDN topology, we can implement Distributed SDN with the help of Packet Tracer. Due
to the limitation of the Mininet tool, we cannot implement communication between two controllers. So, we are assuming
controller to controller communication with the help of two routers that is connected with the BGP protocol to share in-
formation. Also, there are multiple switches they are acting like data planes. Furthermore, we also have end dives they can
communicate with each other as shown in Figure 8.
BGP protocol is one of the core routing protocol for Internet and considered a path vector protocol which does not use
any traditional IGP metrics such as RIP, OSPF and EIGRP. BGP uses path, rule or policies for routing decisions and main-
tains the IP table for network reachability among Autonomous Systems (AS). In this packet tracer activity, BGP has limited
functionality features specially to configure the complex networks. During this experiments, we also observed that OSPF
and EIGRP have multiple areas. We also learn the redistributed routes from the Static OSPF and EIGRP into BGP. In last,
the connectivity is verified of the networks where BGP uses variety of messages to established the connections, exchanging
the route information and check the remote BGP is still their or notifying the remote side if any errors occur. In order to
perform the process, BGP uses four messages including Open, Update, Keep alive and Notification messages.

4.3.2. Open Message

Figure 9 shows the Wireshark screen shot for open message from R1 to R2 and also presents the BGP version, AS number,
hold time, BGP ID and the optional parameters (MP-BGP and route refresh). The marker field on top is used to indicate MD5
authentication. When it’s filled with 1’s then not considering authentication.

4.3.3. Update Message

In update messages, the Figure 10 shows the message from R1. No routes are withdrawn and there are a couple of BGP
attributes. The ORIGIN, AS_PATH, and MULTI_EXIT_DISC (MED) and some of the flags are highlighted. The AS_PATH attribute
is transitive while MULTI_EXIT_DISC is optional. At the bottom, the NLRI information presented with a prefix.
12 K.N. Qureshi, R. Hussain and G. Jeon / Computers and Electrical Engineering 84 (2020) 106634

Figure 10. Capture an update message from R1

Figure 11. μ and the number of switches without Threshold

 K.N. Qureshi, R. Hussain and G. Jeon / Computers and Electrical Engineering 84 (2020) 106634 13

Figure 12. μ and the number of switches with Threshold

5. Result

To provide a virtual image of the data plane and control plane. We can configure both planes according to requirements.
We can implement network topology with the help of MATLAB. We can configure many devices as we want to emulate the
SDN network.
T ( t ) = k + 1 /2 ( μ − λ k) (2)
In the first analysis, we are comparing two graphs to obtain the comparison between SDN controller response without
any threshold and with a threshold. With the help of graphs, we can conclude that controller response time effected by
the switch flow requests from the SDN controller. In the first graph, we have set controller response time μ =10 0 0 0 and 10
switches. Their flow rate towards controller is λ =100, λ =200 and λ =300. The first graph shows that controller response
time increases when there is no threshold. In the second graph, we set the threshold value of 150. We analyze that controller
response time is better against the first graph. Figure 11 and 12 show the μ and the number of switches with or without
threshold.
Two analyses are performed in MATLAB. As Figure 14 explained with the help of Equation (1), that described the coordi-
nation between in one controller to S switch and flow setup time to the controller. Whereas, with the help [30], the results
are presenting the statistics that are very close to real-world examples, such as the arrival of packet rate (λ pps) and rate of
service (μ). As Figure 15 explains, there is a relationship between controller and switches. While 10 switches are connected
with one controller and time for each flow required 1ms setup is at fixed λ = 100 and μ = 10,0 0 0. On the other hand, to
manage at μ = 30,0 0 0 of according to execute 10 switches and also take care of changing of μ time for flow setup time.
Figure 13 shows the relationship between μ and the number of switches with regards to flow-setup time.
In this analysis concerning Figure 14, for observing the difference between the time for flow setup μ of the switches data
rate (200 packets) changed. It is seen that in both figures, at μ = 10,0 0 0 to manage switches the controller capacity changed
as well. Figure 14 shows Flow-setup time relationship with μ and the number of switches.
In results, the analysis showed that the capacity of controllers significantly affects the QoS in the context of applications
requirement. Whereas, to guarantee the application of the QoS requirement should have some limits or certain boundaries
for underlying switches or need to be distributed approach. Also, it required dynamic demands for flow balancing. A glance
at the graphs reveals that the capacity of controller rate change when the underlying switches rate increased. So, because
of this, we need to add more resource in the network that increases the cast but that will increase work efficiency as well.
Figure 15 shows the relationship between p0 and extra required resources (m), by ρ .
This experiment explained, the control plane capacity expressively affects the QoS application requirements. As a signif-
icance, to maintain its QoS requirements specific application, we have to put some boundary on switches that should be
under limited or certain bound, and dynamic flow is necessary for flow balancing. by Equation (2) where the M/M/1 model,
Figure 16 exposes that the number of flows in the queue fluctuates as traffic intensity (ρ = λ/μ) varies with probability.
14 K.N. Qureshi, R. Hussain and G. Jeon / Computers and Electrical Engineering 84 (2020) 106634

Figure 13. Relationship between μ and the number of switches with regards to flow-setup time

Figure 14. Flow-setup time relationship with μ and the number of switches

Figure 16 shows The relation of p queueing and more required resources (m), by (ρ
 
T(t ) = p0 ∗ (m ∗ ρ )m /m!(1 − ρ ) M/M/m (3)

Now we have Equation 3 representation of M/M/m, that indicated by advancing additional resources in the network
domain, queue decreases the probability of data flows considerably, as shown in Figure 14 Relating Figure 15 and 16 at given
ρ = 0.7, we can analyze that queue decreases with probability streams of flow that when we deploy the extra resource.
 K.N. Qureshi, R. Hussain and G. Jeon / Computers and Electrical Engineering 84 (2020) 106634 15

Figure 15. The relationship between p0 and extra required resources (m), by ρ

Figure 16. The relation of p queueing and more required resources (m), by ρ

6. Conclusion

The SDN domain is one of the prominent areas of research. In this paper, we proposed a distributed SDN based model
for the smart grid domain for better QoS especially decreases the controller response time whenever the flow reaches in
switches state. To improve the scalability, we used distributed SDN to avoid bottleneck significant problem, centralized SDN
robustness, bottleneck, and scalability problems in traditional smart grid networks. Whereas load distribution, their place-
ment, and an optimal controller are always there, the proposed model provided a guarantee in SDN and provide service
on time which is the main concern for network design. The proposed SDN model also improves energy consumption and
provide green energy utilization in smart grid IoE networks. Furthermore, this research also fascinated other network direc-
16 K.N. Qureshi, R. Hussain and G. Jeon / Computers and Electrical Engineering 84 (2020) 106634

tions. Firstly, we have a different distribution for analyzing the model and analysis of further general theoretical statistics.
The next consideration is capturing multiple data plane hopes significantly in the network domain, so, it will become easy
to deploy the Jackson-feedback concept that facilitates extended model. Thirdly, a more rigorous real-time prototype needs
to compare the next model that should be established. For the future, the study will be proposed to analyze M/M/1/c and
M/M/m/c methodology that can be more representative of the distributed domain, especially for smart grid areas. Further-
more, we are focusing to deploy this research in the SDN network in real-time strategy, later it can modify and utilize a
cloud-based SDN network. This work will attract many areas to focus SDN and try to deploy it in a network environment
like smart IoE enabled networks.

References

[1] Mahmud K, Khan B, Ravishankar J, Ahmadi A, Siano P. An internet of energy framework with distributed energy resources, prosumers and small-scale
virtual power plants: An overview. Renewable and Sustainable Energy Reviews 2020;127:109840.
[2] Han W, Dong M, Ota K, Wu J, Li J, Li G. SD-OPTS: Software-defined on-path time synchronization for information-centric smart grid. In: GLOBECOM
2017-2017 IEEE Global Communications Conference. IEEE; 2017. p. 1–6.
[3] K. N. Qureshi et al., "A Dynamic Congestion Control Scheme for safety applications in vehicular ad hoc networks," vol. 72, pp. 774-788, 2018.
[4] Qureshi KN, Din S, Jeon G, Piccialli F. Link quality and energy utilization based preferable next hop selection routing for wireless body area networks.
Computer Communications 2020;149:382–92.
[5] Goto Y, Ng B, Seah WK, Takahashi Y. Queueing analysis of software defined network with realistic openflow–based switch model. Computer Networks
2019;164:106892.
[6] Benamrane F, Mamoun MB, Benaini R. New method for controller-to-controller communication in distributed SDN architecture. International Journal
of Communication Networks Distributed Systems 2017;19(3):357–67.
[7] Rehmani MH, Davy A, Jennings B, Assi C. Software Defined Networks-Based Smart Grid Communication: A Comprehensive Survey. IEEE Communica-
tions Surveys & Tutorials 2019;21(3):2637–70.
[8] Wang C, Zhang G, Chen H, Xu H. An ACO-based elephant and mice flow scheduling system in SDN. In: 2017 IEEE 2nd International Conference on Big
Data Analysis (ICBDA); 2017. p. 859–63.
[9] Liu Z, Gao D, Liu Y, Zhang H. An Enhanced Scheduling Mechanism for Elephant Flows in SDN-Based Data Center. In: 2016 IEEE 84th Vehicular Tech-
nology Conference (VTC-Fall); 2016. p. 1–5.
[10] A. Aliyu et al., "Towards video streaming in IoT Environments: Vehicular communication perspective," vol. 118, pp. 93-119, 2018.
[11] M. Prasad et al., "A new mechanism for data visualization with TSK-type preprocessed collaborative fuzzy rule based system," vol. 7, no. 1, pp. 33-46,
2017.
[12] Hussain R, Bashir MU. Model to Improve Scalability and Quality of Services in Software Define Networking. In: 2019 2nd International Conference on
Communication, Computing and Digital systems (C-CODE). IEEE; 2019. p. 28–33.
[13] Benamrane F, Mamoun MB, Benaini R. New method for controller-to-controller communication in distributed SDN architecture. International Journal
of Communication Networks Distributed Systems 2017;19(3):357–67.
[14] Lange S, et al. Heuristic Approaches to the Controller Placement Problem in Large Scale SDN Networks. IEEE Transactions on Network and Service
Management 2015;12(1):4–17.
[15] Sood K, Yu S, Xiang Y, Peng S. Control layer resource management in SDN-IoT networks using multi-objective constraint. In: 2016 IEEE 11th Conference
on Industrial Electronics and Applications (ICIEA). IEEE; 2016. p. 71–6.
[16] Jiao Z, Ding H, Dang M, Tian R, Zhang B. Predictive big data collection in vehicular networks: A software defined networking based approach. In: 2016
IEEE Global Communications Conference (GLOBECOM). IEEE; 2016. p. 1–6.
[17] Sideris K, Nejabati R, Simeonidou D. Seer: Empowering software defined networking with data analytics. In: 2016 15th International Conference on
Ubiquitous Computing and Communications and 2016 International Symposium on Cyberspace and Security (IUCC-CSS). IEEE; 2016. p. 181–8.
[18] Zhao S, Medhi D. Application-aware network design for hadoop mapreduce optimization using software-defined networking. IEEE Transactions on
Network Service Management 2017;14(4):804–16.
[19] Sallahi A, St-Hilaire M. Optimal model for the controller placement problem in software defined networks. IEEE communications letters
2014;19(1):30–3.
[20] Hu H, Wen Y, Gao Y, Chua T-S, Li X. Toward an SDN-enabled big data platform for social TV analytics. IEEE network 2015;29(5):43–9.
[21] Kaur D, Aujla GS, Kumar N, Zomaya AY, Perera C, Ranjan R. Tensor-based big data management scheme for dimensionality reduction problem in smart
grid systems: SDN perspective. IEEE Transactions on Knowledge Data Engineering 2018;30(10):1985–98.
[22] Qin P, Dai B, Huang B, Xu G. Bandwidth-aware scheduling with sdn in hadoop: A new trend for big data. IEEE Systems Journal 2015;11(4):2337–44.
[23] Neves MV, De Rose CA, Katrinis K, Franke H. Pythia: Faster big data in motion through predictive software-defined network optimization at runtime.
In: 2014 IEEE 28th International Parallel and Distributed Processing Symposium. IEEE; 2014. p. 82–90.
[24] Trevisan M, Drago I, Mellia M, Song HH, Baldi M. AWESoME: Big data for automatic Web service management in SDN. IEEE Transactions on Network
Service Management 2017;15(1):13–26.
[25] Liu M, Dou W, Yu S, Zhang Z. A Decentralized Cloud Firewall Framework with Resources Provisioning Cost Optimization. IEEE Transactions on Parallel
and Distributed Systems 2015;26(3):621–31.
[26] Ahmad A, Paul A, Rathore MM, Chang H. Smart cyber society: Integration of capillary devices with high usability based on Cyber–Physical System.
Future Generation Computer Systems 2016;56:493–503.
[27] Kleinrock L. Theory, Volume 1, Queueing Systems. Wiley-Interscience; 1975.
[28] Li L, Zhang L, Fu B. Nimble: A fast flow scheduling strategy for openflow networks. China: China Science Publishing & Media Ltd; 2015.
[29] Sallahi A, St-Hilaire M. Optimal Model for the Controller Placement Problem in Software Defined Networks. IEEE Communications Letters
2015;19(1):30–3.
[30] Benamrane F, Mamoun MB, Benaini R. New method for controller-to-controller communication in distributed SDN architecture. International Journal
of Communication Networks and Distributed Systems 2017;19(3):357–67.

Kashif Naseer Qureshi (Member, IEEE) received the Ph.D. degree from the University of Technology Malaysia (UTM) in 2016. He
is currently a Senior Assistant Professor with Bahria University, Islamabad. He is a Cisco and Microsoft Certified Network Profes-
sional. His research interest focuses on Internet of connected vehicles (IoV), electronic vehicles charging management planning
and recommendation (EV), and the IoT use cases implementation in wireless sensor networks. He has been a reviewer for various
reputable academic journals. He has been leading two Malaysian and Higher Education funded projects and Learning as a PI. His
active engagement in research has resulted in over 100 publications.
 K.N. Qureshi, R. Hussain and G. Jeon / Computers and Electrical Engineering 84 (2020) 106634 17

Raza Hussain received the the master’s degree from the Department of Computer Science, Bahria University, Islamabad, Pakistan,
under the supervision of Dr. K. N. Qureshi. He is currently a Computer Scientist.

Gwanggil Jeon received the B.S., M.S., and Ph.D. (summa cum laude) degrees from the Department of Electronics and Computer
Engineering, Hanyang University, Seoul, Korea, in 20 03, 20 05, and 20 08, respectively. From 2009.09 to 2011.08, he was with
the School of Information Technology and Engineering, University of Ottawa, Ottawa, ON, Canada, as a Post-Doctoral Fellow.
From 2011.09 to 2012.02, he was with the Graduate School of Science and Technology, Niigata University, Niigata, Japan, as an
Assistant Professor. From 2014.12 to 2015.02 and 2015.06 to 2015.07, he was a Visiting Scholar at Centre de Mathématiques et
Leurs Applications (CMLA), École Normale Supérieure Paris-Saclay (ENS-Cachan), France. From 2019 to 2020, he was a Prestigious
Visiting Professor at Dipartimento di Informatica, Università degli Studi di Milano Statale, Italy. He is currently a Full Professor
at Xidian University, Xi’an, China and at Incheon National University, Incheon, Korea. Dr. Jeon was a recipient of the IEEE Chester
Sall Award in 2007 and the ETRI Journal Paper Award in 2008.