A Hash-based Distributed Storage Strategy of

FlowTables in SDN-IoT Networks

Wei Ren,Yan Sun
Beijing Key Laboratory
of Intelligent Telecommunications
Software and Multimedia
Beijing University of
Posts and Telecommunications
Beijing 100876, China
Email: rw478463266@gmail.com,
sunyan@bupt.edu.cn
Tin-Yu Wu
Department of Computer
Science and Information
Engineering
National Ilan University
Yilan, Taiwan, ROC
E-mail:tyw@niu.edu.tw
Mohammad S. Obaidat
Fellow of IEEE and Fellow of SCS,
Computer & Information Science Dept.
Fordham University,
Bronx, NY, USA
E-mail:m.s.obaidat@ieee.org or
msobaidat@gmail.com

Abstract—Nowadays the integration of IoT and SDN has been
a research hotspot which attracts significant attention. However,
as resources are relatively limited in IoT, direct application
of SDN will cause some challenges, one of them is that IoT
forwarding nodes cannot store massive complex FlowTables like
a traditional OpenFlow switch. To solve the problem, this paper
proposes a hash-based distributed storage strategy. Specifically,
we present a multi-dimension selection method to decide the
optimum distributed storage location. And then a hash space is
formed by using the content of data flow in IoT, it is the basis of
FlowTables deployment and data forwarding in distributed stor-
age. Moreover, we introduce a new FlowTables search mechanism
which is built on the principle of the binary tree. Experimental
results demonstrate that our strategy efficiently improves the
FlowTables storage capacity with small performance loss in IoT.
Index Terms—IoT, SDN, Distributed storage, OpenFlow,
FlowTable
I. I NTRODUCTION
With the outstanding progress of 4G, WSN and other tech-
nologies [1], Internet of Things (IoT) penetrates incessantly
in various social areas, including industry, medical treatment,
power grid, etc. But there are many problems along with its
development. For example, it lacks an effective mechanism to
manage complex heterogeneous IoT devices [2]. Many meth-
ods have been proposed to resolve these problems. Software
Defined Network (SDN) is one of these solutions and achieves
excellent results [3]. It is a new network design paradigm
which decouples control plane and data plane [4]. Thus SDN
brings convenient management, programmability and flexibili-
ty to IoT. The most widely used protocol in SDN is OpenFlow
(OF) [5]. It defines the communication interface between the
control and forwarding layers. The FlowTables defined by OF
are the foundation to control data flow forwarding. Whereas
to achieve complex and fine-grained flow control, the size
and number of FlowTables in OF-switches become extremely
large. For instance, there are as many as 1227 bits in the match
fields of OF 1.3 FlowTables [6].
The massive and complex FlowTables stored in a switch
may cause many problems, especially in a case that the
resources of switches are limited. For example, a switch
can delete the plethoric FlowTables, which would further
exacerbate the network delay and controller load. To mitigate
this problem, some people optimize the timeout to delete
overflow FlowTables, some simplify the FlowTable storage
structure, and others concern the extension of hardware storage
space in a OF-switch. Huan Ma et al. [10] proposed a
FlowTables distributed storage framework to solve the exces-
sive FlowTables storage problem in traditional SDN network.
They think all forwarding nodes should be connected to each
other and jointly store an entire FlowTables. However, since
the capacity of IoT forwarding nodes are more inadequate,
traditional extensive FlowTables are highly unlikely to be
stored in a single IoT node. What’s more, existing techniques
are unsuitable for IoT owing to its unique features. For
example, forwarding nodes may not directly communicate
with the IoT SDN controller owning to the constraints of
wireless communication and power. Therefore, we propose a
hash-based distributed storage strategy to address the problem
of abundant FlowTables storage in the SDN-IoT network.
Our strategy focus on the cluster SDN-IoT network. In
this architecture, some IoT nodes play the roles of controller
or OF-switch [7] [8] [9]. Generally, the role of nodes is
determined according to their abilities. We assume that all
nodes can be divided into three categories: the SDN Lo-
cal Centric Node (SDNLCN), the SDN Local Switch Node
(SDNLSN) and the IoT Perception Node (IPN). They are
exhaustively explained in the section III. In our method, when
the storage space of a SDNLSN is exhausted, other multiple
SDNLDNs with large remaining storage space will store the
FlowTables of this SDNLSN. These nodes work as a group to
store FlowTables and cooperate mutually to complete network
function. There are two kinds of distributed storage strategies:
one is the distributed storage strategy of SDNLSNs in a region,
and the other is the distributed storage strategy of SDNLCNs
inter regions.
Fig. 1 shows the proposed distributed storage of FlowTables
in a horizontal view. The SDNLSNs in an aggregation region

978-1-5090-5019-2/17/$31.00 ©2017 IEEE

 FlowTables
### ### ### ###
FlowTables ... ... ... ...
FlowTables
###
...
###
...
###
...
###
...
... ... ... ...
Agg ### ### ### ###
... ... ... ...
Region1 FlowTables Agg ...
...
...
...
...
...
...
...
###
...
...
###
...
...
###
...
...
###
...
...
Region2
Overflow
Overflow Node2 Overflow
Node1 Node3

Agg
Region3

FlowTables
Agg ###
...
###
...
###
...
###
...
Overflow
Region4 ... ... ... ...
Node5
Overflow
Node4 FlowTables
### ### ### ###
... ... ... ...
... ... ... ...

### ### ### ###

...
...
...
...
...
...
...
...
Distributed FlowTables SDN Local Centric Node(SDNLCN) SDN Local Switch Node(SDNLSN) IoT Perception Node(IPN)

Fig. 1: Distributed storage of FlowTables in a horizontal view

need to report their own information to the SDNLCN when limited capacity. In the deployment of FlowTables, we build
their states or abilities change, hence the SDNLCN has global hash value space on the basis of data content that are sensed
network information. When the storage space of a SDNLSN location, sensed time and data type.
is insufficient, the SDNLCN will discover this condition and (3) We systematically evaluate this strategy under a sim-
carry out the distributed storage strategy. The main problems ulation test platform, test results show that the distributed
which we launch are: (1) storage location selection; (2) storage can provide more FlowTables storage space with small
FlowTables deployment method; (3) FlowTables lookup and performance loss.
data flow forwarding. The specific solutions are as follows: The rest of the paper is organized as follows. We review
∙ For the distributed storage location, we constructed a the background and related work in Section II. Section III
multi-dimension location selection scheme. The factors introduces the architecture of SDN-IoT network. Hash-based
involve the distance from the SDNLCN, remaining stor- distributed storage strategy is presented in Section IV. Section
age space, whether storing other SDNLSNs FlowTables V reports on the experimental. Finally, we conclude this paper
or not (repeatability) and the communication distance with future work in Section VI.
from the current storage SDNLSN.
∙ In the FlowTables deployment, aiming to facilitate II. R ELATED W ORK
FlowTables management, storage balance load and data
forwarding, we introduce a hash-based deployment strat- In this section, we summarize previous approaches that are
egy of FlowTables. As IoT is typically data-centric, the related to our work.
hash value space is built on the basis of IoT data content. FlowTables are generally stored in TCAM of OF-switches in
∙ As for efficient FlowTables lookup and data flow forward- traditional SDN network, so the ability of TCAM is the main
ing, this paper presents a binary tree lookup and routing constraint of FlowTables storage. Paul Congdon et al. [11]
method which can reduce the resource consumption. proposed a method to optimize the classification and lookup
Finally, we build a simulation environment based on Open- of FlowTables in TCAM. Linlian Zhang et al. [12] efficiently
vSwich and test the proposed distributed storage strategy. utilized the timeout field to delete useless FlowTables. A
The experimental results reveal that our strategy effectively virtual compression approach was introduced by S Veeramani
achieves FlowTables distributed storage and data forwarding et al. [13] to diminish the size of FlowTables stored in TCAM.
with acceptable performance loss. Compared with the existing These studies focus on storage of a single switch.
methods, our main contributions lie in the following three Another research direction is to reduce the storage space
aspects: occupied by FlowTables. Jingguo Ge et al. [14] presented
(1) To the best of our knowledge, our proposed distributed a heuristic storage space optimization algorithm (H-SOFT)
storage strategy is the first solution to the problem that IoT for FlowTables. H-SOFT degraded the complex and high-
forwarding nodes with limited resources may hardly store dimensional fields of a FlowTable into multiple ones with
massive FlowTables. simple and low-dimensional fields. Chiba et al. [15] replaced
(2) Our distributed storage strategy take the special features the action field of FlowTables with simple bits to lessen the
of IoT into account, such as data-centric forwarding and FlowTable size. To a certain degree, these work can alleviate
 SDN Controller
Internet
IoT Gateway
IoT Controllerr
Agg ...
Region3
Agg
Agg Region2
Region1
R
Region1
SDN Local Centric SDN Local Switch
Node(SDNLCN) Node(SDNLSN)
Fig. 2: The cluster SDN-IoT architecture
the problem of massive FlowTables storage. However, they are
useless when numerous FlowTables exit in a switch.
Considering the above deficiencies and unique character-
istics of IoT, we propose a hash-based distributed storage
strategy. It can store more fine-grained FlowTables without
huge consumption for IoT forwarding nodes.
III. T HE ARCHITECTURE OF SDN-I OT NETWORK
In this section, we mainly introduce the cluster SDN-IoT
network architecture and explain three different function nodes
in detail.
As we all know, there are many diverse SDN-IoT solutions,
one of them is the cluster SDN-IoT network which can
efficiently manage complex heterogeneous devices. It usually
consists of three layers of SDN network as described in Fig.
2. The upper network is composed of traditional SDN, its
controller manages all lower OF-switches, including an IoT-
gateway. The IoT-gateway is also an IoT-Controller in the
intermediate layer network. It is responsible for controlling
cluster head nodes of aggregate regions (Agg Region) in IoT.
The third layer network is IoT Agg Region comprised by
IoT nodes that can communicate mutually. The strongest node
of an Agg Region is the controller (SDNLCN). It is capable
of controlling the network of a region, data forwarding inter
regions, data perception and data process. The weaker node
is the SDNLSN whose functions include data forwarding in a
region, data perception and simple data process. In addition,
the IPN is the weakest node in an Agg region, it is mostly
useful for the date perception. SDNLCN, SDNLSN and IPN
are all perception nodes of IoT, their differences lie in their
distinct abilities, such as power, compute and storage. In this
way, some of SDNLSNs have sufficient capacity to store entire
FlowTables, while others may need some assistance to store
their own FlowTables. The SDNLSNs which need distributed
storage are called overflowNode for the convenient description.
IV. D ISTRIBUTED STORAGE STRATEGY OF F LOW TABLES
This section mainly introduces the basic principles and
procedures of proposed hash-based distributed storage of
FlowTables. There are three key problems to be addressed:
selection of storage location, FlowTables deployment and data
flow forwarding. And the first is the basis of FlowTables
deployment and data forwarding. Moreover, a well-designed
deployment method can increase the efficiency of the data flow
forwarding. These will be explained in detail later.
FlowTables are stored in SDNLSNs and SDNLCNs which
Agg
... Region4 are respectively responsible for data forwarding in a region and
inter regions. When the storage space occupied by FlowTa-
bles exceeds a threshold, the overflowNode begins to story
them in a distributed way. There are two types of methods
to calculate the threshold: static threshold assignment and
dynamic threshold algorithm. A value is manually allocated
IoT Perception
Node(IPN)
to the threshold in the former. As for the dynamic method,
the threshold would vary according to current network status.
Specifically, if network storage space is relatively sufficient,
the threshold will be small and the frequency of distributed
storage is high. Otherwise the threshold will be large and the
frequency is low. We assume the number of SDNLSNs is N.
Every SDNLSN stores up to S FlowTables. The 𝑛𝑡ℎ SDNLSN
stores 𝑆𝑛 (1 ≤ 𝑛 ≤ 𝑁 ) number of FlowTables at a time. And
L is the threshold of distributed storage. It is calculated using
Equation (1):
𝑀 𝑎𝑥(𝑆 − 𝑆𝑛 ) − 𝑀 𝑖𝑛(𝑆 − 𝑆𝑛 )
𝐿= (1)
𝑀 𝑖𝑛(𝑆 − 𝑆𝑛 )
It is obvious that the first method is simple and easy to
implement, while the second dynamic way is agiler to adjust
execution time. And we adopt the static method for simplicity
in the experiment simulation.
A. Distributed storage of FlowTables in a region
When the number of FlowTables in a SDNLSN exceeds
the threshold, the SDNLCN will discovery this issue and adopt
distributed storage strategy for the overflowNode. This process
contains three steps: (1) select the storage location; (2) deploy
FlowTables; (3) forward data flow. The following is a detailed
introduction,
1) Selection of storage location: For distributed storage of
FlowTables in a region, we mainly take the following factors
into consideration:
∙ The distance from the SDNLCN (hops). In the IoT, some
SDNLSNs must communicate with the SDNLCN via
multiple other SDNLSNs. If we move partial FlowTables
of the overflowNode to these nodes that are closer to
a SDNLCN, it can not only achieve distributed storage,
but also decrease the resource consumption caused by
distributed storage. This situation is depicted as the
overflowNode1 and overflowNode2 in Fig. 1. If there
are not appropriate storage nodes which are closer to the
SDNLCN, SDNLSNs with the identical distance can be
chosen to store.
∙ The remaining storage space (Sto). As the SDNLCN has
the ability to know remaining storage space of SDNLSNs,
SDNLSNs with bigger remaining storage space would be
candidate nodes to store the FlowTables. This can mini-
mize the impact of performance for involved SDNLSNs.
 ∙ The communication distance (ComDis). For lessening Hash value space
the network consumption in FlowTables lookup and data ...
SDNLSN1 SDNLSN2 SDNLSNn
forwarding, the candidates should be able to directly
communicate with the present storage node. In brief,
the ComDis is better equal to 1. This can avoid more
FlowTable1
communication cost produced by multi-hop transmission.
FlowTable2
∙ The repeatability (Rep). Rep is the quantity of nodes
FlowTable3
which store their own FlowTables in the candidate SDNL- FlowTable4
SNs. It is better to select the SDNLSN that does not store ... hash
FlowTable5
FlowTables of other nodes, namely Rep is 0. It contributes FlowTable6
to the management of FlowTables. If the SDNLSN where ...
Rep is not 0 must be selected, it would move some
FlowTables to make room for storing FlowTables of the Fig. 3: FlowTables deployment
overflowNode.
Based on the above factors, the SDNLCN select the stor- overflowNode on those selected SDNLSNs. Moreover, these
age location of FlowTables through the following steps: (1) distributed storage FlowTables need to be easy to identify and
According to the current network state, the SDNLCN select locate. If the deployment is random, the overflowNode have
a chain path from the overflowNode to a SDNLCN. In this to traverse all involved nodes to search desired FlowTables,
selected path, the distance from the SDNLCN is closer or which can deteriorate network latency and controller load.
equivalent, but note that the shortest path has the highest Unfortunately, typical deployment methods, e.g., round robin,
priority. However, there may exist a problem that the final weight factor, are not suitable in this scene considering the
selected chain is too long. To mitigate it, we claim the final efficient management of FlowTables [10]. Hence we design a
length of a chain path is not more than the 𝑚𝑎𝑥(3/2𝑘, (𝑘+5)), hash-based distributed deployment method. It can effectively
where k denotes the smallest length from the overflowNode to balance the storage load of nodes and accelerate data flow
a SDNLCN. (2) The distributed storage capacity (DisStoCa- forwarding. Furthermore, the hash value space is built on the
pacity) of each SDNLSN in this path is calculated and ranked basis of match fields in FlowTables.
in the descending order. Moreover, the ComDis between the The basic principle of our solution is that a SDNLCN
next storage location and current location should be equal to 1. divides a hash space into N subspaces, each SDNLSN takes
Finally, depending on the number of FlowTables (numFlows) charge of a subspace. When the hash value of a FlowTable is
to be stored, storage locations that fulfil these requirements located in one subspace, this FlowTable will be deployed in
are selected. It is like the overflowNode1, 2, 4, 5 in Fig. 1. the corresponding SDNLSN, as shown in Fig. 3. Since each
(3) If the nodes in the selected chain path does not meet the match field in a FlowTable is different and independent, it is
requirements specified in the second step, it then returns to reasonable that we use them to build this hash value space.
the first step, continues to search the next chain path, and
executes the second step. (4) If there are not eligible chain TABLE I: Hash value of FlowTables
paths, for instance, the storage space of nodes is inadequate
to store FlowTables, the restrictions in the first step will be Match fields Hash value
Location Time Data Type
relaxed. And then multiple chain paths from the overflowNode FlowTable1 A001 h12L6 1011 A001h12L61011
to a SDNLCN would be selected to construct a mesh path. FlowTable2 A002 m4d1L5 2022 B002m4d1L52022
Subsequently it returns to the second step. The situation is ... ... ... ... ...
depicted as the overflowNode3 in Fig. 1.
The DisStoCapacity of SDNLSNx can be obtained by In the traditional OF protocol, the match fields include
Equation (2). Ingress Port, Ethernet Source, Ethernet Destination and so on.
However, IoT is usually a data-centric network, that is to say,
𝑆𝑡𝑜(𝑥) the importance of obtaining interested data is much higher than
𝐷𝑖𝑠𝑆𝑡𝑜𝐶𝑎𝑝𝑎𝑐𝑖𝑡𝑦(𝑥) = (2)
𝐶𝑜𝑚𝐷𝑖𝑠(𝑥) ∗ 𝑒𝑅𝑒𝑝(𝑥) knowing the location of the data. Besides, these fields are not
The StoSpace is the total required storage space of num- adequate for the needs of data flows in the IoT. Therefore, we
Flows, and 𝑆𝑡𝑜𝑥 is remaining storage space of SDNLSNx. We draw lessons from the NDN network and employ the ”location-
assume that NumStr is ultimate number of storage location. It time-data type” as the match fields values of FlowTables to
is equal to the N that meets the Equation (3) for the first time. form a hash value space. Table 1 shows the hash vale of
FlowTable1 and FlowTable2. Their match fields are in the
𝑁
∑ form of ”data sensed location-data sensed time-data type”. In
𝑆𝑡𝑜𝑥 > 𝑆𝑡𝑜𝑆𝑝𝑎𝑐𝑒 (3) this way, each FlowTable will get a unique hash value and be
𝑥=1
mapped to a subspace using a hash algorithm.
2) FlowTables deployment: After the storage location is On the one hand, storage load balance is important for
determined, the SDNLCN would deploy FlowTables of the FlowTables deployment. Thus the hash subspace is allocated
 overflowNode SDNLSN1 SDNLSN2
FlowTable1 FlowTable2 ... FlowTable m FlowTable1 FlowTable2 ... FlowTable n FlowTable1 FlowTable2 ... FlowTable k

90 2
20 40

Distributed
Storage

overflowNode SDNLSN2
50 50
FlowTable1 FlowTable2 FlowTable3 FlowTable44 ...
... F
FlowTable ϹĆ FlowTable1 FlowTable2 ...
. F FlowTable k FlowTable a ... FlowTable b
coarse-grained fine-grained
fi
fine -grained
i d

SDNLSN1
50
FlowTable1 FlowTable2 ... FlowTable n FlowTable p ... FlowTable q

...

Fig. 4: FlowTables partition information for distributed storage

********
according to the storage capacity of SDNLSNs. Namely, a
SDNLSN with greater storage space possesses larger hash 0******* 1*******
overflowNode
subspace. On the other hand, FlowTables deployment should
00****** 10****** 11******
facilitate the data forwarding to reduce the communication SNode SDNLSN2

consumption caused by distributed storage. Thus we need to
distinguish different FlowTables and store them at suitable
locations. In OpenFlow protocol, FlowTables can be classified
into the coarse-grained and the fine-grained according to
their granularity of match fields. In addition, coarse-grained
FlowTables can be replaced by a set of fine-grained ones and
control more data flow forwarding. These characteristics are
utilized to guide data flow forwarding. It will be thoroughly
explained in next section.
3) Data flow forwarding: After distributed storage of
FlowTables, the overflowNode only store partial FlowTables.
Thus when a data flow arrives at the node, it may not know
how to deal with it. If the overflowNode directly sends a
Packet-In message to the controller, the SDNLCN work load is
heavier. Moreover, the SDNLSNs in the IoT may not directly
communicate with a SDNLCN. That means more commu-
nication consumption and network latency. Furthermore, this
approach loses the meaning of distributed storage to a degree.
If all distributed storage nodes are traversed to forward data,
pointless network traffic will be increased. Consequently,
we provide a mechanism that enables the overflowNode to
know storage partition information, thus it easily forward the
unmatched data flows to the SDNLSN which stores relevant
FlowTables without the help of the controller. If there are not
enjoyable FlowTables at last, the Packet-In message would be
sent to the SDNLCN.
Specifically, we give different solutions for the two FlowTa-
bles deployment results. If the deployment location of dis-
tributed storage is a chain path which gradually approaches
the SDNLCN, unmatched data flows are merely forwarded to
the next node in this path, and repeat this process until the
data flow reaches last storage node. When there are still not
FlowTables handling this data flow, the last storage node will
send Packet-In message to the controller.
000**** 001***** 101***** 111*****
1110**** 1111****
0010****
SDNLSN1
11100***
Fig. 5: The built binary tree for FlowTables
If the deployment location of distributed storage is a mesh
path like overflowNode3 and overflowNode4 in Fig. 1, it is
unreasonable to randomly forward the unmatched data flows
to a storage node. This may lead to the problems of not
finding the target FlowTable or more pointless network traffic.
Therefore, we adopt a hash-based mapping and classification
method to tackle this issue. In our solution, every SDNLSN
is aware of their Flowtables partition information, thus data
forwarding is no more blind. To be specific, the overflowNode
stores all coarse-grained FlowTables which point at the fine-
grained FlowTables. If the storage space is enough, a number
of fine-grained FlowTables are also stored in the overflowN-
ode, as shown in Fig. 4. When a data flow reaches the
overflowNode and dose not match corresponding FlowTables,
it will be forwarded to the SDNLSNs where their FlowTables
can match this kind of data flows according to the coarse-
grained FlowTables. Fig. 4 depicts the partition information
of FlowTables in distributed storage.
The concrete implementation method is to build a binary
tree by using the hash value obtained by FlowTables deploy-
ment. As shown in Fig. 5, FlowTables matching granularity
is becoming finer from top to bottom. For example, the hash
value of a coarse-grained FlowTable is 1 ******* (32 bits),
whose first bit represents the sensed position of the data. It
can cover the value range [-2147483648, 2147483647]. When
we add a bit that represents the sensed time, it can be divided

SDNLCN
SDNLSN2
30 FL 20%
SDNLSN1 SDNLSN3
50% 60%
20 FL
overflowNode
SDNLSN5
30% SDNLSN6
40%
overflowNode
SDNLSN9
10 FL
SDNLSN8 60% SDNLSN10
40% 30%
SDNLSN11
20%
FL: FlowTables
overflowNode
Fig. 6: The topology graph of test network
into two FlowTables, they are respectively 10 ****** (32 bits)
and 11 ****** (32 bits). In this way each FlowTable can be
represented by a node in the binary tree.
B. Distributed storage of FlowTables inter regions
As each SDNLCN can directly communicate with the IoT-
Controller, the overflowNode should select a mesh path to
store FlowTables. Therefore, its FlowTables deployment and
data forwarding methods are similar to ones in a region which
are explained above.
V. E XPERIMENTS AND A NALYSIS
We build a simulation test platform to evaluate the per-
formance of our proposed distributed storage strategy which
is implemented by Python 2.7 programming language. More
specifically, experiments are realized on a computer with Intel
(R) Core (TM) i5-3317U 1.70G CPU, 4GB memory and
64-bit Ubuntu 14.04 system. What’s more, Mininet 2.3.0d1
simulation tool is used to create a virtual IoT network that
mimics the properties and functionalities of a real network,
the controller is implemented based on RYU 4.13, and the
SDNLSNs are implemented based on OpenvSwitch 2.02, a
virtual switch supporting OpenFlow among other features.
Fig. 6 displays the topology graph of our test Agg Regions,
there are 1 SDNLCN and 11 SDNLSNs, the bandwidth of
links is 1MB and delay is about 100ms. To simplify test, we
assume there are at most 100 FlowTables in each SDNLSN,
the percentage of storage spaces that SDNLSNs have used is
depicted as Fig. 6. Moreover, the storage space threshold is
assigned a default value, which is 80% for testing.
We choose the number and location of distributed storage
nodes as performance metrics to examine the feasibility of our
proposed method. In addition, new added links number and
data forwarding time are measured to evaluate the network cost
caused by distributed storage. FlowTables distributed storage
certainly brings more network consumption, we utilize these
parameters to confirm that the performance losses are small
and acceptable. In order to evaluate its effectiveness and
efficiency, we compare our method with another well-known
Nodes number for distributed storage of FlowTables
8
the method in this paper
7 the method of DFSoF[10]
6
Number of nodes
5
SDNLSN4
40%
4
SDNLSN7
20% 3
2
1
0
1 2 3 4 5 6 7 8 9 10 11
SDNLSN
Fig. 7: Distributed storage nodes number
solution [10] which distributes FlowTables on all network
switches.
A. Evaluation of storage nodes number
In this experiment, we evaluate the distributed storage
results in the case that every SDNLSN is the overflowNode
individually at a time for simplicity. The following exper-
iments, namely added links number and forwarding time,
are also based on this hypothesis mentioned. The location
results are partially revealed in Fig. 6. For example, when
SDNLSN11 is the overflowNode, our scheme distributes 20
and 10 FlowTables to SDNLSN10 and SDNLSN6 respectively.
Besides, the number results are shown in Fig. 7 when each
SDNLSN is the overflowNode.
From this picture, we can conclude that average storage
number of our strategy is 2.27, while the other in [10] is
3.55. In other words, our method reduces approximately 22%
storage nodes number, which leads to less network cost.
That is because that all interconnected nodes are used in the
distributed storage for paper [10], while our solution adopt
a multi-dimension selection method to decide the storage
nodes. And more distributed storage nodes mean more network
communication for data forwarding, thus our strategy is more
appropriate in the resource-constrained IoT.
B. Evaluation of added links number and data forwarding
time
To further evaluate the performance loss resulted from
distribution storage, we measure different performance metrics
such as added links number and data forwarding time. For the
former, we calculate the links number from the overflowNode
to the SDNLCN with traversing all involved nodes of dis-
tributed storage. Fig. 8 (a) describes the link number results
of each SDNLSN as the overflowNode. It shows that the links
number of most overflowNodes increase to some extent, while
the others are the same as the previous. Overall, the links
number has increased by 0.91 on average. Because our strategy
selects fewer nodes to distribute FlowTables, there are fewer
links to traverse all storage nodes accordingly. Considering
that IoT forwarding nodes usually communicate with others in
a wireless way, the more new links add, the greater network
consumption is. Consequently, the performance loss of our
strategy is relatively smaller compared with other solutions.
Links number for distributed storage of FlowTables
5
after distributed storage
reasonable extent. The simulation results demonstrate the
feasibility and effectiveness of our proposed strategy.
In future work, we will extend our work to evaluate the
distributed storage method in actual IoT scenario. In addition,
the simplification of OpenFlow protocol for IoT will be taken
into account.

4
before distributed storage ACKNOWLEDGMENT
This work is supported by the National Natural Science
Number of links

3
Foundation of China under Grant No.61672109, No.61370196,
2
No.61772085, and No.616532012.
R EFERENCES
1
[1] Asadi A, Wang Q, Mancuso V. A survey on device-to-device communi-
0 cation in cellular networks, IEEE Communications Surveys & Tutorials,
1 2 3 4 5 6 7 8 9 10 11
SDNLSN
vol. 16, no. 4, pp. 1801-1819, 2014.
[2] Whitmore A, Agarwal A, Da Xu L. The Internet of Things: A survey
(a) links number of topics and trends, Information Systems Frontiers, vol. 17, no. 4, pp.
261-274, 2015.
Data forwarding performance for distributed storage [3] Qin Z, Denker G, Giannelli C, et al. A software defined networking
1800
architecture for the internet-of-things in 2014 IEEE Network Operations
1600 after distributed storage
and Management Symposium (NOMS), 2014, pp. 1-9.
Time of forwarding (ms)

before distributed storage

1400
1200
1000
800
600
400
200
1 2 3 4 5 6
Data flows
(b) data flow forwarding time
Fig. 8: Added links number and data forwarding time
Furthermore, data forwarding time is analysed to verify the
reliability of the conclusion obtained from the last experiment.
We measure the forwarding time of 8 different data flows
derived from SDNLSN1(as the overflowNode). Fig.8 (b) gives
the time when the data flows reach the the destination. It
reveals that proposed FlowTables distributed storage only
slightly affects data forwarding performance. Specifically, the
forwarding time increases by about 20%. This is low and
acceptable, in other words, the data forwarding proves to
be efficient. The reason is that the multi-dimension location
selection, hash-based FlowTables deployment and forwarding
method is utilized to lessen invloved nodes and links. There-
fore, our distributed storage strategy is feasible and efficient
in IoT.
VI. C ONCLUSION
In this paper, we present a new hash-based distributed
storage strategy of FlowTables to resolve the problem of in-
sufficient storage space in IoT forwarding nodes. By selecting
the optimal storage location and deploying FlowTables via
hash value space, it can increase FlowTables storage number
while ensuring efficient data flow forwarding. Moreover, the
performance loss caused by distributed storage is within a
[4] Nunes B A A, Mendonca M, Nguyen X N, et al. A survey of software-
defined networking: Past, present, and future of programmable networks,
IEEE Communications Surveys & Tutorials, vol. 16, no. 3, pp. 1617-1634,
2014.
[5] McKeown N, Anderson T, Balakrishnan H, et al. OpenFlow: enabling
innovation in campus networks, ACM SIGCOMM Computer Communi-
cation Review, vol. 38, no. 2, pp. 69-74, 2008.
[6] Canini M, Venzano D, Peresini P, et al. A NICE Way to Test OpenFlow
Applications in 2014 NSDI, 2014, pp. 127-140.
7 8 [7] Olivier F, Carlos G, Florent N. Sdn based architecture for clustered wsn in
2015 Innovative Mobile and Internet Services in Ubiquitous Computing
(IMIS), 2015, pp. 342-347.
[8] Galluccio L, Milardo S, Morabito G, et al. SDN-WISE: Design, prototyp-
ing and experimentation of a stateful SDN solution for Wireless Sensor
networks in Computer Communications (INFOCOM), 2015, pp. 513-521.
[9] Gonzalez C, Flauzac O, Nolot F, et al. A Novel Distributed SDN-Secured
Architecture for the IoT in 2016 Distributed Computing in Sensor Systems
(DCOSS), 2016, pp. 244-249.
[10] Ma H, Yang Y, Mi Z. A distributed storage framework of FlowTable in
software defined network, Computers & Electrical Engineering, vol. 43,
pp. 155-168, 2015.
[11] Congdon P T, Mohapatra P, Farrens M, et al. Simultaneously reducing
latency and power consumption in openflow switches, IEEE/ACM Trans-
actions on Networking (TON), vol. 22, no. 3, pp. 1007-1020, 2014.
[12] Zhang L, Lin R, Xu S, et al. AHTM: Achieving efficient flow table
utilization in software defined networks in 2014 Global Communications
Conference (GLOBECOM), 2014, pp. 1897-1902.
[13] Veeramani S, Kumar M, Mahammad S N. Minimization of flow table
for TCAM based Openflow switches by virtual compression approach in
2013 Advanced Networks and Telecommuncations Systems (ANTS), 2013,
pp. 1-4.
[14] Ge J, Chen Z, Wu Y.HSOFT a heuristic storage space optimisation
algorithm for flow table of OpenFlow, Concurrency and Computation:
Practice and Experience, vol. 27, no. 13, pp. 3497-3509, 2015.
[15] Chiba Y, Shinohara Y, Shimonishi H. Source flow: Handling millions of
flows on flow-based nodes, ACM SIGCOMM Computer Communication
Review, vol. 40, no. 4, pp. 465-466, 2010