Communication Networks with Multiple SDN
Controllers
Peter Trúchly, Jakub Kubica
Faculty of Informatics and Information Technologies,
Slovak University of Technology in Bratislava, Ilkovičova 2, 842 16, Bratislava, Slovakia
peter.truchly@stuba.sk
Abstract—Software-defined networks are slowly but surely
coming into production. Currently, there are several different
SDN architectures. As SDN solutions continue to evolve, there is
room to improve current solutions in networks with multiple
SDN controllers. One of the researched aspects of the SDN
network is the deployment of multiple controllers to improve
reliability, load balancing, and efficiency. Each controller will
manage its SDN domain. In this paper, we proposed an efficient
SDN architecture based on a flat multicontroller multidomain
network that provides efficient load balancing in the network.
We demonstrated and verified its performance via simulations in
Mininet environment with Ryu controllers.
Keywords—SDN; Multiple SDN controllers; Load balancing
2023 International Symposium ELMAR | 979-8-3503-2512-6/23/$31.00 ©2023 IEEE | DOI: 10.1109/ELMAR59410.2023.10253918
I. INTRODUCTION
Software-defined networks (SDN) represent a computer
networking concept that promises to simplify network
management and administration as well as it supports
innovation through network programmability. SDN networks
are a perspective form of networking that separates the control
plane from the data (infrastructure) plane and provides a
software centralized control element otherwise known as a
controller [1]. Due to this separation of the control plane and
the data plane, network switches become simple devices whose
main task is to forward packets. Making decisions is moved to
the controller.
There are currently several different SDN architectures that
take into account their centralized or distributed, physical or
virtual, flat (horizontal) or hierarchical (vertical), and static or
dynamic realization [2], [3]. Centralized single controller
architecture is based on only one controller that performs all
control functions over the entire infrastructure layer network.
This architecture suffers from a single point of failure and
scalability problems. In case of distributed (multicontroller)
SDN architectures, the flat model has all controllers positioned
on one level (of control plane), connected with each other, they
handle global network information, and control their own
subnetwork (domain). However, controllers in the hierarchical
model form a hierarchy of several levels and have different
responsibilities. The root controller maintains global network
information, whereas local controllers (connected with the rot
controller) take control decisions for their subnetworks. The
flat model provides reduced load per controller, higher
scalability of the network, and higher resilience to failures but
must cope with controller’s management. The hierarchical
979-8-3503-2512-6/23/$31.00 ©2023 IEEE
65th International Symposium ELMAR-2023, 11-13 September 2023, Zadar, Croatia
Authorized licensed use limited to: NATIONAL INSTITUTE OF TECHNOLOGY TIRUCHIRAPALLI. Downloaded on November 18,2023 at 18:12:41 UTC from IEEE Xplore. Restrictions apply.
model can simplify managing controllers, but is again exposed
to problems with a single point of failure. There are also hybrid
distributed SDN architectures.
SDN solutions have already found their application in real
network environments such as wide-area networks and data
centers [4]. However, there are still a lot of challenges to be
investigated so that SDN technology could be deployed
efficiently. Those challenges include scalability, controller
placement, performance, security, interoperability, and
reliability. Networks with multiple SDN controllers belong to
perspective SDN solutions because they are based on
deployment of several control elements, in order to improve
reliability, load distribution and efficiency, as well as the
management of load distribution between individual OpenFlow
devices. Distributed flat SDN architectures eliminate the global
single-point-of-failure problem, but intercontroller coordination
must be efficient. In this paper, we proposed an efficient SDN
solution based on flat a multicontroller architecture that
provides effective load balancing in the network.
The paper is organized as follows. Section 2 describes
related works. The proposed solution (architecture) is presented
in Section 3. Section 4 demonstrates the proposed scenarios
used to test our new architecture. And finally, Section 5
concludes the paper.
II. RELATED WORKS
The way of solving the load distribution in SDN networks
was presented in the work [5]-[8].
The author in [5] implemented SDN controllers in a
horizontal approach. Individual controllers communicated
using the SDNi protocol. The author implemented his own
algorithm similar to the Multiple Spanning Tree Protocol. To
calculate the shortest route, the Dijkstra’s algorithm was used
and the sFlow was used for obtaining the current link
utilization as well.
In [6] a vertical approach is followed. The author suggests
having a central controller over all SDN domain controllers.
Using a central controller can cause a single point of failure
and a scalability issue. The central controller creates an abstract
topology at the SDN domain level and calculates a tree to
remove the loops. Based on the results, the Spanning tree
blocks those ports that are not part of the tree.
29
 The authors in [7] implemented a module for the SDN
controller. To achieve high performance and low latency, they
introduce a load balancing solution for data center networks
based on the OpenFlow protocol. The authors implemented a
dynamic routing algorithm.
Z. Lafit et al. in [8] created the DOLPHIN solution. The
authors proposed a dynamic solution for SDN control that
dynamically distributes the flow load between multiple lines
across multiple SDN domains. It also provides better use of re-
sources. It proposes a complete path calculation and topology
extension system (horizontal and vertical) in the environment
of several SDN domains.
III. THE PROPOSED SOLUTION
The proposed solution was developed using the Ryu SDN
controller, which can communicate with another SDN
controller and balance load traffic between multiple SDN
domains. The Ryu controller was integrated and tested in
Mininet environment.
The Mininet environment was used to emulate OpenFlow
switches and computers that will communicate in several SDN
networks. Individual SDN networks are interconnected using a
GRE (Generic Routing Encapsulation) tunnel. This tunnel was
opened on the OpenFlow switch. GRE encapsulated tunnels are
not encrypted.
As mentioned, the Ryu SDN controller was used in our
solution. This SDN controller is centralized and our proposed
solution requires a distributed controller. Therefore, a new
communication module was added to this SDN controller.
A. Obtaining data from topology
The overall topology consists of several SDN domains,
which are managed by SDN controllers themselves. Each SDN
controller discovers OpenFlow switches, links, and computers
in its SDN network. When the controller is turned on in the
OpenFlow network, the switches announce themselves that
they are active in the network.
When the pingall command is entered into the Mininet
environment, computers appear on the controller. The
controller detects them and saves the following data about
them: MAC and IPv4 address, identifiers (dpid) of switches
they are connected to, and the ports on these switches they are
connected to.
SFlow technology was used to obtain additional
information about the current values of link utilization in the
network. In the sFlow environment, each SDN domain acts as
an sFlow agent, which sends data to the sFlow collector. The
SFlow collector collects these statistics and stores them in its
database, which we can access from individual controllers
using the API.
In our solution, API calls to the sFlow collector are used to
obtain data from the topology, specifically, which switches
with which interfaces are located in the SDN network and
which links between switches exist. Data on the current link
utilization are acquired from the topology. For each interface,
65th International Symposium ELMAR-2023, 11-13 September 2023, Zadar, Croatia
30
Authorized licensed use limited to: NATIONAL INSTITUTE OF TECHNOLOGY TIRUCHIRAPALLI. Downloaded on November 18,2023 at 18:12:41 UTC from IEEE Xplore. Restrictions apply.
Figure 1. Operation of the proposed algorithm.
the ifoutoctets parameter is monitored, which means how many
bits leave that interface.
B. Operation of the proposed algorithm
Fig. 1 shows a diagram of the actions performed in our
proposed solution. Each SDN controller discovers computers
(end nodes) on its network. If two computers want to
communicate with each other, they first send ARP messages.
Computer 1 sends an ARP request in which it looks for the
MAC address of the target computer (computer 2).
This message is sent by the OpenFlow switch (to which the
computer 1 is connected) directly to the SDN controller.
OpenFlow switches have a default rule which sets that if the
device does not know what to do with a given packet, it sends
it to the controller. When this message arrives at the controller,
the controller retains information about that computer. When
computer 2 is not discovered, it will not allow it to
communicate on the network. After discovering computer 2 in
the network, the controller sends an ARP response.
When both computers have an ARP response, they start
communicating with each other, e.g. TCP data flow.
Upon arrival of a TCP SYN message on the controller, the
controller evaluates the shortest path between them over the
least utilized links. The shortest path is defined as the path
composed of the links with the highest available throughput.
The controller then sends OpenFlow rules to devices (switches)
along the way. If these computers are on the same network
(SDN domain), mutual communication can start.
If computer 2 is in the different SDN domain and is
connected to domain which is managed by controller 2, TCP
communication takes place between the controllers (of both
domains) and they exchange the discovered computers via the
message hosts. If an ARP request comes from computer 1 to
controller 1, the controller evaluates the shortest path and sends
OpenFlow rules within its SDN domain; it means following the
GRE interface. The ARP request does not cross between SDN
domains due to a blocking rule. This controller then prepares a
flow message and sends it to controller 2. Upon receiving this
message by controller 2, OpenFlow rules are distributed to
each switch along the path contained in the flow message
within its SDN domain. Subsequently, everything is ready for
communication between those two computers.
C. Communication between SDN controllers
Communication between several SDN controllers in
different SDN domains is performed using the TCP protocol.
 Each controller has its own database in which it stores
information about its network. TCP communications are based
on a client-server model. The SDN controller in the client role
sends synchronization messages to the SDN controller in the
TCP server role regularly every second. In our proposed
solution, the following types of messages will be used:
• Keep-alive messages are used to maintain a TCP
connection to the TCP server. TCP clients regularly
exchange these messages with the TCP server.
• Host messages contain information about the newly
discovered computer.
• Flow messages inform another controller that it is
necessary to load individual OpenFlow rules to
switches in the given SDN domain. This message
contains the path of the connection with information
about the destination port and the type of protocol.
This message also contains important information
about the source and destination computers.
D. Uploading OpenFlow rules
The OpenFlow rules are sent from the SDN controller to
the OpenFlow switches and are divided into two types.
The first type of rules that are uploaded to the switches
represents the initialization rules. In a Mininet environment,
they consist of rules for blocking LLDP messages, blocking
ARP messages on the GRE interfaces of individual switches,
and also a rule that states that if the switch does not know what
to do with a given packet, it forwards it to the controller. These
OpenFlow rules do not expire over time and remain in the
OpenFlow table on individual switches at all times.
The second type of rules (uploaded to switches) is designed
to ensure data communication. For the Mininet environment,
this type of rule can be divided into two subtypes: intra-domain
and inter-domain upload rules.
When uploading rules, the so-called idle timeout function
was implemented in the rules ensuring communication. This
parameter is set to 100 s. This means that if the data flow ends
and then 100 seconds pass, the rules are automatically deleted,
and the switches inform the controller about their deletion.
Figure 2. General view of proposed solution.
65th International Symposium ELMAR-2023, 11-13 September 2023, Zadar, Croatia
Authorized licensed use limited to: NATIONAL INSTITUTE OF TECHNOLOGY TIRUCHIRAPALLI. Downloaded on November 18,2023 at 18:12:41 UTC from IEEE Xplore. Restrictions apply.
E. General view of the proposed solution
The overall view of our proposed solution is shown in Fig.
2. The SDN network is divided into several SDN domains; the
individual SDN domains are connected by GRE tunnels and
contain OpenFlow switches and computers.
Each controller contains three main parts, namely the
shortest path calculation application, the sFlow database, and
the communication module. The communication module is
used to ensure communication with other controllers in the
network through the TCP protocol. Communication consists of
three types of messages: keep-alive, hosts, and flows.
The shortest path calculation application contains an
efficient Dijkstra algorithm. This solution also distributes
OpenFlow rules to switches, which the Dijkstra algorithm
evaluates as being on the shortest route through the least
utilized links.
The controller also contains a connection to the sFlow
database, from which it determines the current link utilization.
IV. TESTING SCENARIOS
The functionality of our implementation was demonstrated
in test scenarios.
The basic concept of load balancing was focused on in a
simple topology consisting of three switches and three
computers shown in Fig. 3. The yellow lines represent pings
sent between all end node pairs to make them visible in the
network. Then the H1 node started TCP communication with
H2 (red line). After a short time, H2 started UDP transmission
to H1 (blue line). Different paths for these data transmissions
acknowledge the deployment of load balancing in the network.
To ensure real control of the load balancing for all flows that
have ended and elapsed over some time, namely 100 s, they
will be removed from the OpenFlow tables. If users want to re-
establish the connection after this time, they will have to go
through the process of finding the shortest path again through
the least busy links. This initial testing clearly demonstrated the
functionality of our proposed load-balancing solution within a
single SDN domain.
Figure 3. Load balancing in simple SDN network.
31
 Figure 4. Multi-domain final path of flows.
In the next test scenario (shown in Fig. 4), the
communication and load balancing between the three
interconnected L2 networks were verified. Blue and red lines
represent data communications with calculated paths for them
in the network via UDP and TCP protocol respectively. Once
each controller knows all the relevant data related to the
topology information, the data flows are simulated.
Retransmissions have also occurred in this TCP flow
simulation, which are caused by the delay caused by sending
flows messages. Then communication starts. With three
domains, the delay can reach a maximum of 2-3 seconds in the
worst-case scenario.
In this test scenario, intra-domain and inter-domain
communications were controlled. The resulting flows are also
shown in Fig. 4. The proposed algorithm chose the least
utilized and at the same time the shortest paths. The TCP
protocol reliably provided communication between SDN
domains. The calculated shortest paths created the best path so
that the maximum capacity of the links was used.
V. CONCLUSION
In this paper, we described our work on load balancing in
multicontroller SDN networks. The prototype of our solution is
based on a new communication module integrated into the
originally centralized-based controller. In the network,
individual controllers exchange information about computers in
individual domains and information about when and to which
switches to send OpenFlow rules. Our proposed solution is able
to determine the actual utilization of current links using the
sFlow protocol. Subsequently, our solution is able to calculate
the shortest path through the local domain of the controller or
the entire L2 network. Unlike other approaches, no central
controller is necessary. The proposed and implemented
solution was tested and its functionality was demonstrated in
several different scenarios. The delay caused by the delivery of
the TCP sFlow message proved to be a low disadvantage.
OpenFlow switches cannot send unknown broadcast packets,
so they send such packets to the controller, and the controller
drops them because it is waiting for a TCP flows message. This
65th International Symposium ELMAR-2023, 11-13 September 2023, Zadar, Croatia
32
Authorized licensed use limited to: NATIONAL INSTITUTE OF TECHNOLOGY TIRUCHIRAPALLI. Downloaded on November 18,2023 at 18:12:41 UTC from IEEE Xplore. Restrictions apply.
delay is on the order of tens of milliseconds for the two SDN
domains. With three controllers, it is on the order of a few
seconds, a maximum of 4 s. The TCP protocol coped with this
minimal delay by performing the required retransmissions.
While communicating with the UDP protocol, the initial
packets are dropped on the SDN controller, and only when the
TCP flows message arrives this connection can work.
ACKNOWLEDGMENT
This publication has been written thanks to Slovak National
research projects: APVV-19-0401, APVV-20-0346, and
KEGA 025STU-4/2022.
REFERENCES
[1] D. Kreutz, F. M. V. Ramos, P. E. Verissimo, C. E. Rothenberg, S.
Azodolmolky, and S. Uhlig, “Software-defined networking: a
comprehensive survey,” Proceedings of the IEEE, vol. 103, no. 1, pp.
14–76, 2015.
[2] O. Blial, M. B. Mamoun, and R. Benaini, “An Overview on SDN
Architectures with Multiple Controllers,” Journal of Computer Networks
and Communications, 2016.
[3] S. Saraswat, V. Agarwal, H. P. Gupta, R. Mishra, A. Gupta, and T.
Dutta, “Challenges and solutions in software defined networking: A
survey,” Journal of Network and Computer Applications, vol. 141, pp.
23–58, Sep 2019.
[4] B. Sokappadu, A. Hardin, A. Mungur, and S. Armoogum, ”Software
Defined Networks: Issues and Challenges,” 2019 Conference on Next
Generation Computing Applications (NextComp), pp. 1-5, 2019.
[5] T. Ofčarovič, ”Application of software defined networks with multiple
controllers,” MSc. thesis, Slovak University of Technology, 2019.
[6] A. K. Selvakumaran, ”Software-defined inter-domain switching,” MSc.
thesis, Ryerson University, 2016.
[7] Y. Li and D, Pan, “OpenFlow based load balancing for fat-tree networks
with multipath support,” 12th IEEE International Conference on
Communications (ICC’13), Budapest, Hungary, 2013.
[8] Z. Latif, K. Sharif, F. Li, M. M. Karim, S. Biswas, M. Shahzad and S.P.
Mohanty, “DOLPHIN: Dynamically Optimized and Load Balanced
PatH for INter-domain SDN Communication,” IEEE Transactions on
Network and Service Management, vol. 18, no. 1, pp. 331-346, March
2021.