See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/317072038

Internet of Everything: A Large-Scale Autonomic IoT Gateway

Article  in  IEEE Transactions on Multi-Scale Computing Systems · July 2017

DOI: 10.1109/TMSCS.2017.2705683

CITATIONS READS

97 6,607

3 authors, including:

Byungseok Kang
University of Derby
45 PUBLICATIONS   417 CITATIONS   

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Special Issue on: CloudIoT: Convergence services between the Cloud and IoT View project

All content following this page was uploaded by Byungseok Kang on 01 June 2017.

The user has requested enhancement of the downloaded file.

 IEEE TRANSACTIONS ON MULTI-SCALE COMPUTING SYSTEMS, VOL. 3, NO. X, XXXXX 2017 1

1 Internet of Everything: A Large-Scale

2 Autonomic IoT Gateway
3 Byungseok Kang, Daecheon Kim, and Hyunseung Choo, Member, IEEE

4 Abstract—Gateways are emerging as a key element of bringing legacy and next generation devices to the Internet of Things (IoT).
5 They integrate protocols for networking, help manage storage and edge analytics on the data, and facilitate data flow securely between
6 edge devices and the cloud. Current IoT gateways solve the communication gap between field control/sensor nodes and customer
7 cloud, enabling field data to be harnessed for manufacturing process optimization, remote management, and preventive maintenance.
8 However, these gateways do not support fully-automatic configuration of newly added IoT devices. In this paper, we proposed a

of
9 self-configurable gateway featuring real time detection and configuration of smart things over the wireless networks. This novel
10 gateway’s main features are: dynamic discovery of home IoT device(s), automatic updates of hardware changes, connection
11 management of smart things connected over AllJoyn. We use the ‘option’ field for automatic configuration of IoT devices rather than
12 modify standard format of CoAP protocol. Proposed gateway functionality has been validated over the large-scale IoT testbed.

13 Index Terms—IoT, IoT gateway, self-configuration, large scale IoT

Ç
14

15
16
17
18
19
20
1

I
INTRODUCTION
NTERNET of Things (IoT) is a new paradigm built up with
a continuum of uniquely addressable things which is able
to communicate with each other through a Internet, with
the bolster of approaches and protocols such as pervasive
computing, ubiquitous computing, sensing technology,
Constrained Application Protocol (CoAP) [1], IPV6 and
ro can be modeled in terms of two main control loops (local
and global) with sensors (for self-monitoring), effectors (for
self-adjustment), knowledge and planner/adapter for
exploiting policies based on self- and network environment
awareness [9], [10].
The huge number and large-scale devices mandate the use
41
42
43
44
45
46
EP
21 other protocols. It is a system that bridges physical and of gateways for Internet services. There are various kinds of 47
22 cyber world is formed to enable symbiotic communication access technologies for this gateway design. While 4G/LTE 48
23 seamlessly between the two parties [2]. It means that, IoT could be a good solution for many applications, it suffers 49
24 gives a vision where a large network of uniquely identified from increasing packet collisions with increasing amounts of 50
25 smart things with different end devices such as sensors and downlink access. Furthermore, traditional cellular network is 51
26 actuators connected at any-time, any-place, any-thing, also not suitable for quality of service (QoS) support and con- 52
27 working together to provide variety of services on demand sumes a fair amount of power [11], [12]. As the proprietary 53
28 to the customers [3], [4]. and unlicensed solutions are not optimized for spectral effi- 54
29 An autonomic networking [5], [6] is the network that has ciency, with exponential increase in IoT deployment, these 55
30 the capabilities of being self-defining, self-healing, self-con- solutions are very likely to congest the unlicensed bands and 56
31 figuring, self-optimizing, etc. Started by IBM in 2001, this trigger complaints from existing customers. At the same 57
IEE

32 initiative ultimately aims to develop network systems capa- time, a variety of new wide area applications (e.g., smart cit- 58
33 ble of self-management, to overcome the rapidly growing ies, traffic monitoring, and smart grids) for IoT are offering 59
34 complexity of network systems management, and to reduce new markets to wireless operators for enhancing their reve- 60
35 the barrier that complexity poses to further growth [7], [8]. nues. As a result, the Third Generation Partnership Project 61
36 The network makes decisions on its own, using high-level (3GPP) has recently decided to include narrowband IoT (NB- 62
37 policies; it will constantly check and optimize its status and IoT) in Release more than 10 standards [13]. 63
38 automatically adapt itself to changing network conditions. With this context, large-scale IoT gateway is one of the 64
39 An autonomic networking framework is composed of auto- challenges in IoT industry. However, current IoT gateways 65
40 nomic components (AC) interacting with each other. An AC operate with passive or semi-automatic modes. [14], [15] In 66
other words, when the customer buys a new IoT device, 67
they manually install the device based on the setup manual. 68
 The authors are with the Department of Computer Science and Engineering, Moreover, IoT gateway asks the user whether to register a 69
Sungkyunkwan University, Suwon 440-746, Korea.
E-mail: {byungseok, daecheon, choo}@skku.edu. new device or not. To counter these limitations, in this 70

Manuscript received 14 Feb. 2017; revised 19 Apr. 2017; accepted 15 May

paper, we propose a self-configurable IoT gateway for the 71
2017. Date of publication 0 . 0000; date of current version 0 . 0000. large-scale IoT environment. When the users bring a new 72
(Corresponding author: Hyunseung Choo.) IoT device into existing wireless networks, IoT gateway 73
Recommended for acceptance by S. Bhunia. automatically detects and registers it. If the user throws out 74
For information on obtaining reprints of this article, please send e-mail to:
reprints@ieee.org, and reference the Digital Object Identifier below. old devices from their place, proposed IoT gateway auto- 75
Digital Object Identifier no. 10.1109/TMSCS.2017.2705683 matically delete that device form the device list (database). 76

2332-7766 ß 2017 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
 2 IEEE TRANSACTIONS ON MULTI-SCALE COMPUTING SYSTEMS, VOL. 3, NO. X, XXXXX 2017

new devices and remove old devices. In [16], authors intro- 109
duced a generic gateway-based framework for WSN-IP net- 110
work interconnection. Their framework allow access to 111
individual sensor nodes and enables network-based query 112
for data-centric WSN. In addition, they provided transpar- 113
ent access from one network to the other without modifying 114
protocols running in either network. Work [21] proposed 115
IoT Gateway system based on Zigbee and generalized radio 116
packet service (GPRS) protocols according to the typical IoT 117
application scenarios and requirements from telecom opera- 118
tors. It presented the data transmission between wireless 119
sensor networks and mobile communication networks. 120
However, those two gateways are not flexible, because they 121

of
cannot be customized for different applications. 122
In [17], authors proposed a generic sensor network plat- 123
form (SwissQM/SwissGate) that provided a high level 124
Fig. 1. General feature of IoT gateway.
interface for programming sensor networks and also pre- 125
sented a multi-tier architecture for efficiently handling and 126
77 Proposed system saves user from trouble of registering, optimizing the operation of the network. Research [18] 127
78 installing, and managing smart devices. The state-of-the-art introduced a smart gateway for specific kinds of IoT appli- 128
79 AllJoyn framework provides convenient user interface with- cations. The gateway has full knowledge and control both 129
80
81
82
83
84
85
86
87
ro
out modifying standard source codes. To validate proposed
gateway, we developed large-scale IoT experimental testbed
and measured several QoS factors. The remainder of this
paper is organized as follows. Related work on IoT gateway
and platform technologies is presented in Section 2, Section 3
describes our proposed IoT gateway, experimental testbed
development and experimental results are provided in
Section 4, Section 5 describe conclusions and future work.
over the sensor network and the Internet. They can act as
performance-enhancing proxies and intelligent caches to
preserve the limited resources of the sensor network.
Authors [22], proposed sensor network middleware that
can translate sensed data from sensor networks using
extended markup language (XML) and provide translated
data for various Web applications. However, most tasks are
130
131
132
133
134
135
136
EP
completed by different network-dependent sensing servers 137
not a smart gateway. It is observed that the required hard- 138
88 2 RELATED WORK ware cost will become too high. 139
89 IoT gateway supports a variety of communication protocols
90 and data types between various sensors, which can realize
2.2 Semi-Automatic Gateways 140
91 the conversion of data format which communicated
Semi-automatic gateways generate a connection link 141
92 between varieties of sensors to unify the uploaded data for-
between new device and gateway. However, they cannot 142
93 mats (see Fig. 1). At the same time, the acquisition or control
support automatic link for device attributes (or functions). 143
94 command which reach at the perception network are
For instance, when a user buys a new TV in living room, the 144
95 mapped to produce messages that meet specific device com-
gateway creates a connection link to TV itself not its 145
96 munication protocol. In this section, we discuss the most
attribute. It means that the user cannot switch the TV chan- 146
IEE

97 known contributions in the literature.

nel and adjust volume. Authors [19], designed a plug- 147
98 The goal of IoT gateway is to bridge various sensing
configurable-play service-oriented gateway. It was aimed at 148
99 domain networks with public communication networks or
making fast and easy to employ various external sensor net- 149
100 Internet, settle with the heterogeneity between these various
work applications. Work of [23] proposed IoT gateway with 150
101 networks, strengthen the management of both IoT gateway
multi-channel RS485. It ensures the real-time performance 151
102 itself and terminal nodes. We mainly classify the IoT gate-
and reliability for IoT while transferring data between the 152
103 ways into three types: passive, semi-automatic, and fully-
application and the underlying layer. It resulted in improv- 153
104 automatic. The proposed gateway belongs to fully-
ing the organization and management efficiency of 154
105 automatic IoT gateway (see Fig. 2).
endpoint devices and solving the problems of data trans- 155
mission among different protocols. However, the gateway 156
106 2.1 Passive Gateways is running on PC which demands for higher hardware envi- 157
107 Passive gateways require manual settings for the new IoT ronment. Thus, the advantages in the actual IoT applica- 158
108 devices. The users or IoT engineers have to manually add tions are not obvious. 159

Fig. 2. Classification of IoT gateways.

 KANG ET AL.: INTERNET OF EVERYTHING: A LARGE-SCALE AUTONOMIC IOT GATEWAY 3

of
160
Fig. 3. Large-scale IoT testbed. ro
In [15], authors proposed a novel configurable smart IoT real-world setting, where real-world data and feedback can 190
EP
161 gateway. This gateway has plug-able architecture, whose be obtained from users and their environment under realis- 191
162 modules with different communication protocols can be tic experimental conditions. While our final ambition is to 192
163 customized and plugged in according to different networks. cover both indoor and outdoor environments on our 193
164 In addition, the gateway has unified external interfaces in research center (see Fig. 3). 194
165 accordance with flexible software development. Work [20], In addition to this, the testbed infrastructure should sup- 195
166 presented a multi-interface gateway. It can be used in spe- port autonomic registration, installation, and experimenta- 196
167 cific smart spaces to automatically control traditional TV, tion cycles for the different envisioned IoT techniques, 197
168 air conditioner, smart meter, sphygmomanometer, and smart objects and allow for the evaluation of those in an 198
169 smart phone. The work of [24] supported heterogeneous interdisciplinary context. We chose to implement the facility 199
170 sensor networks and access networks. moreover, it can sup- as a living lab in our research center called ICT convergence 200
171 port different types of sensor nodes and access methods, research center, where each employee can become part of 201
IEE

172 and can provide a unified data format for middleware or the experiment during his or her daily activities. In order to 202
173 applications. However, all above work do not support auto- increase the flexibility of the testbed to cater for demands of 203
174 nomic attribute generation of new devices. diverse experiments, we deployed in each room a mix of 204
heterogeneous IoT devices, implementing a wide range of 205

175 3 LARGE-SCALE AUTONOMIC IOT GATEWAY sensing modalities and common communication interfaces. 206
This infrastructure is complemented with mobile experi- 207
176 To support full automatic IoT device registration and solve mentation nodes that can be carried around by end users 208
177 the heterogeneous network transmission problems, this and fixed interaction displays in the infrastructure provid- 209
178 paper aimed at designing and implementing a self-configu- ing additional means for the communications. 210
179 rable gateway. It plays the role of analyzer of different type
180 of IoT devices. The designed gateway is able to communi- 3.2 Architecture 211
181 cate with a variety of smart objects using 3G/4G, LTE, Wi- Fig. 4 provides an overview of the IoT architecture of our 212
182 Fi, ZigBee, Bluetooth and IrDA standards. This section pro- testbed. Three main devices can be identified: i) a Server 213
183 vides an overview of the large-scale IoT testbed, introducing system that hosts all the back-end functionalities of the 214
184 its key design considerations and the real testbed. testbed and provides the entry point for an experimenter to 215
access the testbed, ii) an embedded Gateway (GW) tier 216
185 3.1 Design Considerations which forms the testbed infrastructure and allows the 217
186 The gateway is a key component of every IoT solution. As iii) IoT tier to be connected and reachable to a backbone net- 218
187 we mentioned in Section 1, our main objective was to build work through 4G/LTE or WiFi. Although all the devices 219
188 an experimental facility that allows large-scale experimenta- can be involved in each experimentation phase, the IoT 220
189 tion with heterogeneous IoT technologies deployed in a device represents the user-centric component of our testbed, 221
 4 IEEE TRANSACTIONS ON MULTI-SCALE COMPUTING SYSTEMS, VOL. 3, NO. X, XXXXX 2017

of
Fig. 5. Network architecture of AllJoyn framework.
Fig. 4. Three main devices.
and it typically runs in a background/service pro- 265
222 merging embedded IoT nodes with sensing capabilities cess. This is common on Linux systems where the 266
223 together with higher-end user-centric devices such as AllJoyn Router runs as a daemon process and other 267
224 Smartphones and Smart Displays. Each IoT node provides AllJoyn apps connect to the Standalone Router. By 268
225 two forms of connectivity: i) wireless communication capa- having multiple apps on the same device use the 269
226
227
228
229
230
231
232
233
ro
bilities (IEEE 802.15.4, and through the GW devices WiFi
and Bluetooth) that can be exploited during an experiment
in order to form different kinds of networks, ii) a wired USB
connection to a dedicated GW for management purposes.
Differently, Smartphones and Smart Displays are connected
only through wireless links (Bluetooth or WiFi) either
through GWs or directly to the backbone network.
The typical architecture of IoT gateway is usually far more
3)
common AllJoyn Router, the device consumes less
overall resources.
An App uses a Router on a different device. Embed-
ded devices (which use the Thin variant of the AllJoyn
framework, more on this later) typically fall in this
camp as the embedded device typically does not have
enough CPU and memory to run the AllJoyn router.
270
271
272
273
274
275
276
EP
234 complex than the architecture of most enterprise systems. 3.2.1 Gateway Software 277
235 One of the main factors that increases the complexity of IoT The software application is the heart of the IoT gateway. 278
236 systems is that backend services residing in the data center, The gateway software is responsible for collecting messages 279
237 which is the heart of most enterprise systems, are actually from the sensors and storing them appropriately until they 280
238 just a piece of the bigger IoT picture. With IoT gateway, we can be pre-processed and sent to the data center. The gate- 281
239 have to deal with a myriad of devices working in the field. way software decides if the data at a given stage of process- 282
240 Because the nature of these devices is very different from ing should be temporary, persistent, or kept in-memory. 283
241 web, desktop, or even mobile clients, we need an intermedi- The gateway software should be designed with failure 284

242 ate architectural element that will act as a proxy between the and disaster recovery in mind. Since gateway devices are 285

243 world of field devices and the enterprise data center. often operated in the field, we should prepare for working 286

244 For the IoT network testbed development, we use a All- conditions that are far from ideal. For example, the gateway 287
IEE

245 Joyn framework [25]. The AllJoyn framework runs on the software should be prepared for a power outage or other 288

246 local network. It enables devices and apps to advertise and actions that may result in an interruption of gateway proc- 289

247 discover each other. This section explains the network archi- essing. The gateway software should be bootstrapped and 290

248 tecture and the relationship between various AllJoyn com- started automatically as soon as power returns to the device, 291

249 ponents. In addition, this framework comprises AllJoyn and it should continue to work from the point where it was 292

250 Apps and AllJoyn Routers, or Apps and Routers for short. interrupted. Gateway software should also be autonomic 293

251 Apps communicate with Routers and Routers communicate enough to properly handle system logging. It has to find the 294

252 with Apps. Apps can only communicate with other Apps right balance between the number of log entries stored 295

253 by going through a Router. Fig. 5 shows the basic topology on the various kinds of IoT devices and those sent to the 296

254 of AllJoyn network architecture. Apps and Routers can live data center. 297

255 on the same physical device, or on different devices. From An IoT application comprises app code, service frame- 298

256 an AllJoyn perspective, it doesn’t matter. In reality, three works libraries, and core library. First, app code is the appli- 299

257 common topologies exist: cation logic of the application. It can be programmed to 300
either the service frameworks libraries, which provide 301
258 1) An App uses its own Router. In this case, the Router higher level functionality, or the core library, which pro- 302
259 is called a “Bundled Router” as it is bundled with vides direct access to the core APIs. Second, service frame- 303
260 the App. AllJoyn Apps on mobile OSes like Android works libraries implement a set of common services, like 304
261 and iOS and desktop OSes like Mac OS X and Win- onboarding, notification, or control panel. By using the com- 305
262 dows generally fall in this group. mon service frameworks, apps and devices can properly 306
263 2) Multiple Apps on the same device use one Router. In interoperate with each other to perform a specific function- 307
264 this case, the Router is called a “Standalone Router” ality. Finally, core library provides the lowest level set of 308
 KANG ET AL.: INTERNET OF EVERYTHING: A LARGE-SCALE AUTONOMIC IOT GATEWAY 5

TABLE 1 TABLE 2
Informations of Device Table Informations of Attribute Table

ID Name Num. of attributes Joining date ID Attribute name Attribute function Last update
D070410 TV 4 2014.08.23 D070410 Channel up, down 2016.08.20
D431831 HVAC control 2 2015.02.01 D070410 Volume up, down 2016.08.20
S235342 Smoke sensor 0 2016.04.04 D070410 Power on, off 2016.08.20
S889001 Temp sensor 0 2012.11.20 D070410 Mute on, off 2016.08.20
D121110 Lamp 1 2012.11.20 D121110 Lamp on, off 2016.08.20
S900010 Light sensor 0 2012.11.20

enable dynamic discovery of every devices and sensors. 357

309 APIs to interact with the IoT network. It provides direct The detail information of CoAP fields in the header are 358
310 access to advertisements and discovery, session creation, defined in IETF RFC document [1]. 359

of
311 interface definition, and object creation/handling. We use As shown on Tables 1 and 2, proposed gateway uses 360
312 these APIs to implement large-scale IoT service frameworks device and attribute tables, respectively. Every intercon- 361
313 and private interfaces. nected IoT device information is periodically updated by 362
CoAP packet through ACK packet that is being sent by the 363
314 3.2.2 Gateway Data Transfer IoT device. The device Table contains ID, Name, Number of 364
315 Usually gateways are connected to the Internet using GPS, attributes, and joining date (see Table 1). The attribute table 365
316 WiFi, or Ethernet. Some gateways can also work in both GPS includes ID, attribute name, and attribute function, last 366
317 and WiFi modes (for example, gateways mounted in moving update (see Table 2). 367
318
319
320
321
322
323
324
325
ro
vehicles). In general, non-GPS connectivity is preferred to
send data, as it doesnt require a subscription to a paid mobile
plan. Some gateways will be constantly connected to inex-
pensive local networks, but those using GPS connectivity
should be very conservative in terms of what data they send
to the data center. The gateway should apply service logic
against the data it collects to understand which messages
should be sent over expensive GPS networks, and which data
3.4 Self-Configuration Scheme (SCN)
A Self-Configuration Network [28], [29] is an automation
technology designed to make the planning, configuration,
management, optimization and healing of mobile radio
access networks simpler and faster. SCN functionality and
behavior has been defined and specified in generally
accepted mobile industry recommendations produced by
368
369
370
371
372
373
374
EP
326 can be cached on the device for deferred offline processing. organizations such as 3rd Generation Partnership Project 375
327 The messages collected by the gateway from the IoT and the Next Generation Mobile Networks (NGMN). 376
328 device (sensors and actuators) are usually small in size. For SCN has been codified within 3GPP Release 8 and subse- 377
329 example, the current value of the temperature measured by quent specifications in a series of standards including 378
330 the sensor is just a decimal number. GPS coordinates are 36.902, [30] as well as public white papers outlining use 379
331 two decimal numbers, which represent longitude and lati- cases from the NGMN [31]. The first technology making use 380
332 tude. This is an important thing to remember: the gateway of SCN features will be LTE network, but the technology 381
333 operates on a large number of small messages. has also been retro-fitted to older radio access technologies 382
such as Universal Mobile Telecommunications System 383
334 3.3 Real-Time Device Monitoring (RTDM) (UMTS) [32]. The LTE specification inherently supports 384
335 Real-time data monitoring [26], [27] is a process through SCN features like Automatic Neighbor Relation (ANR) 385
IEE

336 which an administrator can review, evaluate and modify detection, which is the 3GPP LTE Rel. 8 flagship feature [33]. 386
337 the addition, deletion, modification and use of data on soft- In our case, we use a Centralized SCN (C-SCN). In C- 387
338 ware, a database or a system. It enables data administrators SCN, function is more typically concentrated closer to 388
339 to review the overall processes and functions performed on higher-order network nodes or the network Operations sup- 389
340 the data in real time, or as it happens, through graphical port systems (OSS), to allow a broader overview of more 390
341 charts and bars on a central interface/dashboard. edge elements and coordination of, e.g., load across a wide 391
342 It is critical to have a good knowledge about the condi- geographic area. Due to the need to inter-work with cells 392
343 tion of the current IoT devices in order to appropriately supplied by different equipment vendors, C-SCN systems 393
344 maintain the device table. In order to be able to meet are more typically supplied by 3rd parties. 394
345 demands and provide satisfactory QoS, device monitoring Self-configuration strives towards the “plug-and-play” 395
346 mechanisms are required and can lead to collection and paradigm in the way that new base stations shall automati- 396
347 processing of a certain amount of runtime data. To monitor cally be configured and integrated into the IoT network. 397
348 the IoT device and sensors continuously, we used CoAP [1] This means both connectivity establishment, and download 398
349 to collect device and attributes information (e.g., Device ID, of configuration parameters are software. Self-configuration 399
350 Name, Join date). is typically supplied as part of the software delivery with 400
351 CoAP is a specialized web transfer protocol for use with each radio cell by equipment vendors. When a new IoT 401
352 constrained nodes and constrained networks in the Internet device is introduced into the network and powered on, it 402
353 of Things. This protocol is designed for M2M applications gets immediately recognized and registered by the network. 403
354 such as smart home and building automation. A CoAP The IoT gateway then automatically adjust their technical 404
355 packet is sent periodically on each IoT device to discover parameters in order to provide the required function and 405
356 and test connections. CoAP packets are broadcasting to capacity, and, in the same time, avoid the device conflict. 406
 6 IEEE TRANSACTIONS ON MULTI-SCALE COMPUTING SYSTEMS, VOL. 3, NO. X, XXXXX 2017

407 The IoT gateways currently available, allow the user(s) to collect sensing data from a set of areas of offices (e.g., meet- 465
408 configure a few properties such as TV channel, light, refrig- ing area, relaxing area, and work area) and the lecture 466
409 erator, etc. However, large-scale environment, such as those room. The deployed sensors (for measuring indoor climate, 467
410 based on software-defined networks, are likely to have very energy consumption of office utilities, peoples presence in 468
411 rich configuration controls. The goal of this research is to offices, and parking lot status) collect information about the 469
412 develop algorithms that enable IoT gateway autonomic con- physical status of indoor and outdoor building environ- 470
413 figuration. We propose an initial study of auto-configura- ment, and transfer it to the IoT server platform, AllJoyn 471
414 tion that considers two properties: dynamic discovery and standard-compatible server platform, which allows further 472
415 auto-registration. processing and analysis. 473
The testbed is composed of 40 compound sensors, each of 474

416 4 EXPERIMENTAL RESULTS them having four kind of raw sensors (temperature, humid- 475
ity, illumination and presence sensor), 10 CO2 (Carbon 476
417 In the following section, we provide a large-scale testbed dioxide) concentration detection sensors, 10 smart sockets 477
418 development for IoT experimentation and evaluation of

of
for measuring the electrical power consumption, and 20 478
419 proposed IoT gateway. We concentrate measurement on parking lot sensors, with total of 200 sensors (i.e., 160 raw 479
420 three main dimensions: device setup response time, energy sensors + 10 CO2 + 10 sockets + 20 parking sensors). Table x 480
421 consumption and monitoring overhead. We further discuss summarizes IoT devices supported by the University 481
422 the extent to which the available testbeds fulfill the require- testbed that will be available in the scope of the IITP-IoT 482
423 ments for future IoT testbeds introduced in the Section 2. federation. 483

424 4.1 Large-Scale Testbed 4.2 System Measurements 484

425
426
427
428
429
430
431
432
ro
In [34], the authors survey relevant experimental wireless
sensor network testbeds available to the community today.
The most advanced and active testbed in that survey is the
“SmartSantander” project which offers an experimental
research facility at the scale of a city, and which can be used
to test smart city applications and services. The Smart-
Santander project targets a 20,OOO-node network. On the
other hand, loT-LAB [35] is a federation of platforms, and is
One of the key issues of the application based on IoT is
extracting the analysis result from huge amount of collected
sensing data and recognizing users context from recent
incoming sensing data to provide customized real-time ser-
vice [36], [37]. Therefore, the response time is one of the
most important criteria of user satisfaction on IoT service.
We also consider dynamical change of IoT devices which
comes from their resource constraint and mobility, as a
485
486
487
488
489
490
491
492
EP
433 part of OneLab (Internet-overlaid, Broadband access, wire- result some devices leave from the network and some 493
434 less loT). In the SmartSantander project, all nodes—called device join to the network for replacing broken-down or left 494
435 “service nodes” (battery-constrained nodes)—only produce devices. The average response time for device registration 495
436 data and can be configured only by the administrators of while increasing the number of IoT device gives a good 496
437 the testbed. They are, however, not open to be reprog- indication of both the general performance and the dynam- 497
438 rammed by users. Some nodes (with fewer battery con- ics of the system. We show the response time prediction 498
439 straints) can be reprogrammed so users can test their own and its dependence on bottleneck utilization in the system. 499
440 protocols. Both approaches are complimentary: Smart- Fig. 6 shows the service time, delay and response time in 500
441 Santander targets smart city applications and experimenta- IoT server. We see that average device response time is less 501
442 tion at the loT application level, loT-LAB give bare-metal than 25 ms. In addition, average service time + delay is 502
443 access to all nodes in all sites. resulted around 12 ms. 503
IEE

444 Development of prototypes for large-scale IoT systems is The definition for the lifetime of a IoT sensor node usu- 504
445 challenging. A few hundred IoT nodes can be deployed on ally depends upon battery lifetime [38], [39]. If the mote 505
446 the IoT platform, but it is not easy to get repeatable results were to stop transmitting and never transmits again, the 506
447 when using a shared and public infrastructure. To over- definition would be helpful. However, as the experiments 507
448 come this critical hurdle, we offers a large-scale federated show a new definition is required to account for motes that 508
449 experimental platform allowing researchers, loT designers, stop transmitting, but could potentially begin transmitting 509
450 developers and engineers to construct, benchmark and opti- at a later time. Multiple tests were necessary to build an 510
451 mize their protocols, applications and services. As a accurate model. Results measuring the direct and indirect 511
452 state-of-the-art testbed, our goal is to answer the needs and energy consumption in each IoT sensor are presented and 512
453 requirements of current and future loT technology. In par- used to derive an basic energy consumption model and 513
454 ticular, it offers: (i) a heterogeneous and rich environment device [40], [41]. Fig. 7 shows the average energy consump- 514
455 (e.g., IoT hardware, topologies, OS, platform, up-to-date tion of temperature sensors while changing the indoor tem- 515
456 standardized protocol stacks and libraries) applicable to a perature. we can estimate the expected energy consumption 516
457 large spectrum of loT services; (ii) the ability to instrument and system overhead based on this result. 517
458 an experiment through virtualization and reproducibility Data that we send across an IoT (wireless) network is 518
459 tools; (iii) the ability to manage, interact with and monitor housed in a data envelope called a packet. Each transmis- 519
460 running experiments. sion includes additional information, called overhead, that 520
461 The large-scale IoT testbed (originally installed for moni- is required to route the data to the proper location. We can 521
462 toring building energy consumption) has been imple- calculate network overhead by sending a fixed-size data 522
463 mented on the ICT research center, 2nd Engineering Bld., transmission across the network and observing the number 523
464 Sungkyunkwan University in Suwon, Korea. It aims to of extra bytes of data transmitted for the action to be 524
 KANG ET AL.: INTERNET OF EVERYTHING: A LARGE-SCALE AUTONOMIC IOT GATEWAY 7

of
ro
EP
Fig. 6. Average response time and CPU overhead at the IoT Server.

525 completed. Fig. 8 shows the endpoint device monitoring structured around the gateway. The architecture is capable of 547
526 overhead at the IoT server. First, we measured level of wire- interacting with a non-smart device over AllJoyn framework. 548
527 less network congestion while chaning the monitoring fre- Detail testbed development and system measurement are 549
IEE

528 quency (top-left). In other words, IoT gateway sends the

529 monitoring packets to all endpoints with different fre-
530 quency. Second, bottom-left figure depicts a comparison of
531 the measured response time against our predictions during
532 50 minutes. The result shows that monitoring overhead has
533 increased continuously. Finally, left figure shows normal-
534 ized histogram of overhead during 10 minutes. The average
535 value of entire network overhead resulted around 1,000
536 control packets.

537 5 CONCLUSION
538 The internet of things is being termed as the future of internet
539 communication. This work is an attempt to integrate sensors
540 and actuators with large-scale environment and wireless gate-
541 way that could fit in most common small devices today. We
542 have introduced a general IoT architecture to provide novel
543 Machine to Machine (M2M) services [42], [43], [44]. The
544 mobile clients interact with the M2M devices and endpoints
545 through the IoT gateway. The M2M services including device Fig. 7. Average energy consumption of IoT devices (temperature
546 discovery, local storage of resource configurations are sensors).
 8 IEEE TRANSACTIONS ON MULTI-SCALE COMPUTING SYSTEMS, VOL. 3, NO. X, XXXXX 2017

of
ro
Fig. 8. Monitoring overhead: (top-left) level of network congestion; (bottom-left) number of message overhead; (right) normalized histogram of
system overhead over 10 minutes.
EP
550 discussed. Such real testbed can be used in several real life sce- [5] P. C. Vinh, “Toward formalized autonomic networking,” Mobile 580
Netw. Appl., vol. 19, no. 5, pp. 598–607, 2014. 581
551 narios including personal health monitoring, environmental [6] K. Tsagkaris, et al., “A survey of autonomic networking architec- 582
552 monitoring and semantics in IoT. The usefulness of the system tures: Towards a unified management framework,” Int. J. Netw. 583
553 lies in the fact that it provides an incremental methodology to Manage., vol. 23, no. 6, pp. 402–423, 2013. 584
554 add new services and is generic enough to be compliant with [7] B. Kang, P. Nguyen, V. Zalyubouskiy, and H. Choo, “A distrib- 585
uted delay-efficient data aggregation scheduling for duty-cycled 586
555 future standards in M2M communications and IoT. The sys- WSNs,” IEEE Sensors J., vol. 17, no. 11, pp. 3422–3437, Jun. 2017. 587
556 tem is scalable to handle huge amount of traffic as it is based [8] B.-S. Kang and I.-Y. Ko, “Effective route maintenance and restora- 588
557 on RESTful paradigm and SenML is very lightweight which tion schemes in mobile ad hoc networks,” Sensors, vol. 10, no. 1, 589
pp. 808–821, 2010. 590
558 can be processed very fast. As future direction, the gateway [9] B. Kang and H. Choo, “An SDN-enhanced load-balancing tech- 591
559 interfaces and their functionalities are being implemented nique in the cloud system,” J. Supercomputing, vol. 72, pp. 1–24, 592
IEE

560 using updated AllJoyn framework, access rights for multiple 2016. 593
561 users and security/privacy of IoT architecture are being [10] B. Kang and H. Choo, “A cluster-based decentralized job dis- 594
patching for the large-scale cloud,” EURASIP J. Wireless Commun. 595
562 investigated. Netw., vol. 2016, no. 1, pp. 1–8, 2016. 596
[11] B. Kang, N. Kwon, and H. Choo, “Developing route optimization- 597
563 ACKNOWLEDGMENTS based PMIPv6 testbed for reliable packet transmission,” IEEE 598
Access, vol. 4, no. 1, pp. 1039–1049, Mar. 2016. 599
564 This work was supported by the G-ITRC Program under [12] B. Kang, S. Myoung, and H. Choo, “Distributed degree-based link 600
565 Grant IITP-2015R6812-15-0001, the NRF Research Fellow scheduling for collision avoidance in wireless sensor networks,” 601
IEEE Access, vol. 4, no. 3, pp. 7452–7468, Sep. 2016. 602
566 program under Grant NRF-2016R1A6A3A11934080, and [13] M. Lauridsen, I. Z. Kov
acs, P. Mogensen, M. Sørensen, and 603
567 the NRF Korea under Grant 2010-0020210. S. Holst, “Coverage and capacity analysis of LTE-M and NB- 604
IoT in a rural area,” in Proc. IEEE 8th Veh. Technol. Conf., 2016, 605
pp. 1–5. 606
568 REFERENCES [14] S. K. Datta, C. Bonnet, and N. Nikaein, “An IoT gateway centric 607
569 [1] C. Bormann, A. P. Castellani, and Z. Shelby, “CoAP: An applica- architecture to provide novel M2M services,” in Proc. IEEE World 608
570 tion protocol for billions of tiny internet nodes,” IEEE Internet Forum Internet Things, 2014, pp. 514–519. 609
571 Comput., vol. 16, no. 2, pp. 62–67, 2012. [15] S. Guoqiang, C. Yanming, Z. Chao, and Z. Yanxu, “Design and 610
572 [2] N. Jazdi, “Cyber physical systems in the context of industry 4.0,” implementation of a smart IoT gateway,” in Proc. IEEE Int. Conf. 611
573 in Proc. IEEE Int. Conf. Autom. Quality Testing Robot., 2014, pp. 1–4. Green Comput. Commun. IEEE Internet Things IEEE Cyber Phys. 612
574 [3] G. Fortino and P. Trunfio, Internet of Things Based on Smart Objects, Social Comput., 2013, pp. 720–723. 613
575 Fortino & P. Trunfio, Eds. Cham, Switzerland: Springer, 2014. [16] K. A. Emara, M. Abdeen, and M. Hashem, “A gateway-based 614
576 [4] M. Soliman, T. Abiodun, T. Hamouda, J. Zhou, and C.-H. Lung, framework for transparent interconnection between WSN and IP 615
577 “Smart home: Integrating internet of things with Web services network,” in Proc. IEEE Int. Conf. Comput. Tool, 2009, pp. 1775– 616
578 and cloud computing,” in Proc. IEEE 5th Int. Conf. Cloud Comput. 1780. 617
579 Technol. Sci., 2013, vol. 2, pp. 317–320.
 KANG ET AL.: INTERNET OF EVERYTHING: A LARGE-SCALE AUTONOMIC IOT GATEWAY 9

618 [17] R. Mueller, J. S. Rellermeyer, M. Duller, and G. Alonso, “Demo: A [42] J. Holler, V. Tsiatsis, C. Mulligan, S. Avesand, S. Karnouskos, and 694
619 generic platform for sensor network applications,” in Proc. IEEE D. Boyle, From Machine-to-Machine to the Internet of Things: 695
620 Int. Conf. Mobile Adhoc Sensor Syst., 2007, pp. 1–3. Introduction to a New Age of Intelligence. New York, NY, USA: 696
621 [18] D. Bimschas, H. Hellbr€ uck, R. Mietz, D. Pfisterer, K. R€ omer, and Academic, 2014. 697
622 T. Teubler, “Middleware for smart gateways connecting sensor- [43] K.-C. Chen and S.-Y. Lien, “Machine-to-machine communications: 698
623 nets to the Internet,” in Proc. 5th Int. Workshop Middleware Tools Technologies and challenges,” Ad Hoc Netw., vol. 18, pp. 3–23, 699
624 Services Run-Time Support Sensor Netw., 2010, pp. 8–14. 2014. 700
625 [19] L. Wu, Y. Xu, C. Xu, and F. Wang, “Plug-configure-play service- [44] M. Hasan, E. Hossain, and D. Niyato, “Random access for 701
626 oriented gateway-for fast and easy sensor network application machine-to-machine communication in LTE-advanced networks: 702
627 development,” in Proc. 2nd Int. Conf. Sensor Netw., 2013, pp. 53–58. Issues and approaches,” IEEE Commun. Mag., vol. 51, no. 6, 703
628 [20] C.-T. Chang, C.-Y. Chang, R. D. B. Martinez, P.-T. Chen, and pp. 86–93, Jun. 2013. 704
629 Y.-D. Chen, “An IoT multi-interface gateway for building a smart
630 space,” Open J. Social Sci., vol. 3, no. 7, 2015, Art. no. 56. Byungseok Kang received the BS degree in 705
631 [21] Q. Zhu, R. Wang, Q. Chen, Y. Liu, and W. Qin, “IOT gateway: computer engineering from Sejong University, 706
632 Bridging wireless sensor networks into Internet of Things,” in Korea, in 2006, the MS degree in electrical and 707
633 Proc. IEEE/IFIP 8th Int. Conf. Embedded Ubiquitous Comput., 2010, electronics engineering from Korea University, 708
634 pp. 347–352. Korea, in 2008, and the PhD degree in electron- 709

of
635 [22] J.-W. Yoon, Y.-K. Ku, C.-S. Nam, and D.-R. Shin, “Sensor network ics and computer science from the University of 710
636 middleware for distributed and heterogeneous environments,” in Southampton, United Kingdom, in 2015. He is 711
637 Proc. Int. Conf. New Trends Inf. Service Sci., 2009, pp. 979–982. currently a research professor with Sungkyunk- 712
638 [23] D. Min, Z. Xiao, B. Sheng, and G. Shiya, “Design and implementa- wan University, in 2015. His research interests 713
639 tion of the multi-channel RS485 IOT gateway,” in Proc. Int. Conf. include wired/wireless networking, internet of 714
640 Cyber Technol. Autom. Control Intell. Syst., 2012, pp. 366–370. things, sensor networking, mobile computing, 715
641 [24] D. Xu, L. Yang, and L. Jiang, “Research and design of IoT gate- network protocols, and simulations/numerical 716
642 way,” in Proc. Int. Ind. Informat. Comput. Eng. Conf., 2015, pp. 837– analysis. 717
643 840. 718
644 [25] I. AllSeen Alliance, “AllJoyn framework,” 2016. [Online]. Avail-
645
646
647
648
649
650
651
652
653
654
able: https://allseenalliance.org/framework/

ro
[26] D. Giannone, “Now-casting and the real-time data flow,” Hand-
book of Economic Forecasting, vol. 2, pp. 195–237, 2013.
[27] Q. Xin, P. Olofsson, Z. Zhu, B. Tan, and C. E. Woodcock, “Toward
near real-time monitoring of forest disturbance by fusion of modis
and landsat data,” Remote Sens. Environ., vol. 135, pp. 234–247,
2013.
[28] R. Wojciechowski, A. Siersze
n, and º. Sturgulewski, “Self-configu-
ration networks,” in Image Processing and Communications Chal-
lenges 7. Berlin, Germany: Springer, 2016, pp. 301–308.
Daecheon Kim received the BS degree in infor-
mation and communication engineering from
Sungkyul University, Korea, in 2016, He is cur-
rently working toward the master’s degree in
information and communication engineering at
Sungkyunkwan University. His research interests
include internet of things, sensor networking, and
mobile computing.
719
720
721
722
723
724
725
726
EP
655 [29] H. Hu, J. Zhang, X. Zheng, Y. Yang, and P. Wu, “Self-configura-
656 tion and self-optimization for LTE networks,” IEEE Commun.
657 Mag., vol. 48, no. 2, pp. 94–100, Feb. 2010.
658 [30] E. U. T. R. Access and Evolved Universal Terrestrial Radio Access Hyunseung Choo received the BS degree in 727
659 Network (E-UTRAN), Overall description, vol. 126, 2008. mathematics from Sungkyunkwan University, 728
660 [31] N. Alliance, “NGMN 5G white paper,” Next Generation Mobile Korea, in 1988, the MS degree in computer sci- 729
661 Networks, White paper, 2015. ence from the University of Texas at Dallas, in 730
662 [32] A. Samukic, “UMTS universal mobile telecommunications sys- 1990, and the PhD degree in computer science 731
663 tem: Development of standards for the third generation,” in Proc. from the University of Texas at Arlington, in 1996. 732
664 Global Telecommun. Conf., 1998, vol. 4, pp. 1976–1983. From 1997 to 1998, he was a patent examiner 733
665 [33] N. A. Ali, A.-E. M. Taha, and H. S. Hassanein, “Quality of service with Korean Industrial Property Office. Since 734
666 in 3GPP R12 LTE-advanced,” IEEE Commun. Mag., vol. 51, no. 8, 1998, he has joined the College of Information 735
667 pp. 103–109, Aug. 2013. and Communication Engineering, Sungkyunkwan 736
668 [34] L. Sanchez, et al., “SmartSantander: IoT experimentation over a University, and is an associate professor and 737
IEE

669 smart city testbed,” Comput. Netw., vol. 61, pp. 217–238, 2014. director of Convergence Research Institute. Since 2005, he is director of 738
670 [35] C. Adjih, et al., “FIT IoT-LAB: A Large scale open experimental Intelligent HCI Convergence Research Center (eight-year research pro- 739
671 IoT testbed,” in Proc. IEEE 2nd World Forum Internet Things, 2015, gram) supported by the Ministry of Knowledge Economy (Korea) under 740
672 pp. 459–464. the Information Technology Research Center support program super- 741
673 [36] B. Kang and H. Choo, “A deep-learning-based emergency alert vised by the Institute of Information Technology Assessment. His 742
674 system,” ICT Express, vol. 2, no. 2, pp. 67–70, 2016. research interests include wired/wireless/optical embedded networking, 743
675 [37] S.-G. Jung, B. Kang, S. Yeoum, and H. Choo, “Trail-using ant mobile computing, and grid computing. He is vice president of Korean 744
676 behavior based energy-efficient routing protocol in wireless sen- Society for Internet Information (KSII). He has been editor-in-chief of the 745
677 sor networks,” Int. J. Distrib. Sensor Netw., vol. 2016, 2016, Journal of KSII for three years and journal editors of the Journal of Com- 746
678 Art. no. 11. munications and Networks, the ACM Transactions on Internet Technol- 747
679 [38] J. Gubbi, R. Buyya, S. Marusic, and M. Palaniswami, “Internet of ogy, the International Journal of Mobile Communication, Springer- 748
680 Things (IoT): A vision, architectural elements, and future Verlag Transactions on Computational Science Journal, and editor of 749
681 directions,” Future Generation Comput. Syst., vol. 29, no. 7, the KSII Transactions on Internet and Information Systems since 2006. 750
682 pp. 1645–1660, 2013. He has published more than 200 papers in international journals and ref- 751
683 [39] P. Bellavista, G. Cardone, A. Corradi, and L. Foschini, ereed conferences. He is a member of IEEE and the ACM. 752
684 “Convergence of MANET and WSN in IoT urban scenarios,”
685 IEEE Sensors J., vol. 13, no. 10, pp. 3558–3567, Oct. 2013.
686 [40] Q. Wang and W. Yang, “Energy consumption model for power " For more information on this or any other computing topic, 753
687 management in wireless sensor networks,” in Proc. 4th Annu. please visit our Digital Library at www.computer.org/publications/dlib. 754
688 IEEE Commun. Soc. Conf. Sensor Mesh Ad Hoc Commun. Netw.,
689 2007, pp. 142–151.
690 [41] Q. Chi, H. Yan, C. Zhang, Z. Pang, and L. Da Xu, “A reconfigura-
691 ble smart sensor interface for industrial WSN in IoT environ-
692 ment,” IEEE Trans. Ind. Informat., vol. 10, no. 2, pp. 1417–1425,
693 May 2014.

View publication stats