Received January 1, 2018, accepted January 22, 2018, date of publication February 5, 2018, date of current version April

4, 2018.
Digital Object Identifier 10.1109/ACCESS.2018.2802489

IoT-TaaS: Towards a Prospective

IoT Testing Framework
HIUN KIM 1 , ABBAS AHMAD2 , JAEYOUNG HWANG1 , HAMZA BAQA2 , FRANCK LE GALL2 ,
MIGUEL ANGEL REINA ORTEGA3 , AND JAESEUNG SONG1 , (Senior Member, IEEE)
1 SejongUniversity, Seoul 05006, South Korea
2 Easy Global Market, 06560 Valbonne, France
3 European Telecommunications Standards Institute, 06921 Valbonne, France

Corresponding author: JaeSeung Song (jssong@sejong.ac.kr).

This work was supported in part by the European Union’s Horizon 2020 Research and Innovation Programmes under Grant 643943 and
Grant 687884, in part by the Institute for Information and communications Technology Promotion grants from the Korea Government
under Grant B0184-15-1003, and in part by the National Research Foundation of Korea under Grant NRF-2017R1D1A1B03036285.

ABSTRACT The Internet of Things (IoT) aims to change many aspects of people’s daily lives by extending
the scope of computing to the physical world, and thus shift the environment of computing more to a
distributed and decentralized form. The amount of IoT devices and their collaborative behavior causes new
challenges to the scalability of traditional software testing, and the heterogeneity of IoT devices increases
costs and the complexity of coordination of testing due to the number of variables. In this paper, we introduce
IoT Testing as a Service- IoT-TaaS, a novel service-based approach for an automated IoT testing framework
aims to resolve constraints regarding coordination, costs, and scalability issues of traditional software testing
in the context of standards-based development of IoT devices, and explore its design and implementation.
IoT-TaaS is composed of remote distributed interoperability testing, scalable automated conformance testing,
and semantics validation testing components adequate for testing IoT devices. To provide a conceptual
overview, we analyze its technical and systemic advancement and compare it to traditional testing with
concrete examples.

INDEX TERMS Internet of Things, conformance testing, interoperability testing, test as a service, automated
testing, semantic testing.

I. INTRODUCTION For example, the usual interoperability testing methods

Interoperability and conformance testing are widely used
established types of testing in the software industry [1]–[5],
where manufacturers have to visit a testing lab in order to
test the compliance of their developed devices against a spe-
cific standard and interoperability of the target devices with
other devices. With the rise of the Internet of Things(IoT),
where billions of devices are expected to be integrated into
horizontal applications [6], [7] relying on many different
standards [8], [9], such testing operations are daunted by rad-
ical changes. The heterogeneity and collaborative nature of
IoT systems require extensive and scalable testing to ensure
functional correctness.
A. BACKGROUND
In this paper, we discuss general methodology and high-level
implementation of standardized conformance and interoper-
ability testing and discuss coordination, costs, and scalability
issues faced when applying conventional testing methodolo-
gies to the development of IoT products.
2169-3536
2018 IEEE. Translations and content mining are permitted for academic research only.
15480 Personal use is also permitted, but republication/redistribution requires IEEE permission.
See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
used in the telecommunication industry are hardly able to
handle distributed IoT systems and lack the flexibility to man-
age a variety of IoT devices, some of which are smart enough
to communicate over a high-level protocols (e.g. Hypertext
Transfer Protocol (HTTP) [10], Constrained Application Pro-
tocol (CoAP) [11], [12] and Message Queuing Telemetry
Transport (MQTT) [13]) while others are extremely con-
strained (e.g. a temperature sensor communicating over a
Low-Power Wide-Area Network) with limited support for
remote testing. Also, the main interoperability challenges for
the IoT now sit at the semantic level where exchanged data
streams need to be checked for their syntactic and semantic
correctness as well as for their interoperability.
B. MOTIVATION
In this paper, we study ongoing research work related to
IoT testing, such as interoperability testing which support for
various testing configuration [14], and conformance testing
VOLUME 6, 2018
H. Kim et al.: IoT-TaaS: Toward a Prospective IoT Testing Framework
that is adaptable and can handle the various protocols of
IoT devices [15], and semantic validation tools that can be
used for checking syntactic and semantic correctness of mes-
sages [16]. However, these tests are studied as individual solu-
tions and do not provide a comprehensive approach including
facilities for test configuration, management, or reporting.
In addition, there is previous work regarding IoT test-
ing to tackle this problem for real-world IoT devices from
dynamic test case generation and execution to testbed
integration [17]–[20], but they are not discussed in the context
of standard- which is the crucial viewpoint of testing IoT
software for practical standards-based development. Even
research, on the test system architecture and test derivation
of standardized IoT software [21]–[24], however, they are
focused on a specific property of testing (e.g. test case deriva-
tion, distributed testing system). This paper aims to discuss
various IoT Testing in the context of an integrated framework
and standard manner.
C. PROSPECTIVE IoT TESTING FRAMEWORK
In order to tackle standards-based IoT testing for beyond level
of individual, technical aspect, we introduce a novel service-
based approach for a IoT testing automation framework,
IoT-TaaS(IoT Testing as a Service), a prospective IoT testing
framework resolving such constraints of IoT testing, and pro-
pose the cross-cutting concept which deals with the technical
difference of testing various IoT devices by obtaining bird’s-
eye view of individual solutions. To do that, we discuss the
concept and implementation of a traditional conformance and
interoperability testing framework to derive the required fea-
tures of IoT-TaaS. We analyze three ongoing research work
related to IoT testing as concrete examples of prospective IoT
testing frameworks.
By extracting essential testing concepts and various config-
urations from these research work we design a scalable IoT
testing framework that can test a huge number of IoT devices.
Using IoT-TaaS, various IoT devices can be tested remotely
on not only their functional features but also their semantic
correctness.
In summary, we make the following main contributions:
1) an empirical analysis of IoT test systems for interop-
erability, conformance, semantics validation testing by
comparison of traditional telecommunications testing;
2) the architectural design of a novel IoT testing frame-
work called IoT-TaaS; the design of a novel IoT test-
ing framework called IoT-TaaS which covers remote
interoperability testing, scalable conformance testing
and semantic validation testing for an integrated and
standard manner;
3) the proof of concept of IoT-TaaS supporting automated
and distributed testing of IoT devices with remotely
located test systems;
4) design of concept for extensible IoT testing framework-
plug and test that are applicable for general IoT
ecosystems.
VOLUME 6, 2018
FIGURE 1. Generic model of conformance and interoperability testing.
Table 1 and 2 summarize contributions of this paper with
comparison of previous technique and benefits of plug and
test concept. The remainder of this paper is structured as
follows. In Section 2, we present an introduction to tradi-
tional software testing to motivate the need for IoT test-
ing. Section 3 gives three categories of ongoing IoT testing
research and provides an overview of the IoT testing frame-
work. Section 4 discusses required IoT testing techniques (i.e.
remote testing, semantics testing and automated testing) that
comprise the testing framework. Finally, Section 5 concludes
this paper by pointing out some future research directions
with plug and test, which is a generalizable design concept
to implement IoT-TaaS.
II. TRADITIONAL SOFTWARE TESTING
Products and systems embedding software (incl. the telecom-
munications industry) require standardization to ensure func-
tionality, interoperability, and security. The prominent way
of validating a standard compliance and interoperability to a
product is testing. Typically, Standards Developing Organi-
zations (SDOs) such as the European Telecommunications
Standards Institute (ETSI)1 standardize specifications for
conformance and interoperability tests and use them to fulfill
the different range of standards-based product testing. This
section introduces conventional software testing methodolo-
gies standardized and used by many SDOs [25].
A. CONFORMANCE TESTING
Conformance testing determines to what extent an imple-
mentation of a particular standard conforms to the individual
requirements of that standard. In generic models of con-
formance testing, a test system contains two main compo-
nents: the System Under Test (SUT), which contains the
Implementation Under Test (IUT), and the Means of Test-
ing (MOT). The MOT includes at least one tester depending
on the architecture of the IUT and its interfaces. The MOT
additionally handles activities such as coordination, logging,
and reporting.
1 http://www.etsi.org/
15481

TABLE 1. Contributions of IoT-TaaS by testing types and its properties.
TABLE 2. Extensible plug and play architecture for IoT-TaaS.
1) ARCHITECTURE FOR CONFORMANCE TESTING
Fig. 1 shows a testing methodology generally used for pro-
tocol and reactive systems. The left side of Fig. 1 represents
conformance testing, and the line between the small boxes
and surrounding large box represents the communication
medium between the subject and object of testing. In order
to perform conformance testing, the test system uses various
components from the creation, i.e., Implementation Confor-
mance Statement (ICS), Implementation eXtra Information
for Testing (IXIT), Testing Description Language (TDL),
Executable Test Suite (ETS), Abstract Test Suite (ATS), and
Equipment under Test (EUT) from [25].
ICS provides an overview of the features, capabilities,
functionality, and options that are implemented in the prod-
uct to check capabilities and provide indications for basic
interoperability testing. IXIT contains additional metadata
necessary for testing such as internet addresses. TDL is a
recent formal language allowing the description of test cases.
Tree and Tabular Combined Notation version 3 (TTCN-3) is
a standardized formal language for testing which is widely
used for the specification of test suites by ETSI and other
SDOs such as 3GPP and oneM2M. ATS is the entire collec-
tion of test cases, which specifies how to complete tests is
usually described in a formal test description language such
as TTCN-3. ETS is an executable program from ATS using a
TTCN compiler.
Not just normal behavior but also various exceptional
behaviors are tested by conformance testing. Therefore, con-
formance testing gives privileges to a tester to allow them
to perform extensive functional testing. However, because
a standard may leave room for options and configura-
tion, conformance testing alone does not guarantee that
15482
H. Kim et al.: IoT-TaaS: Toward a Prospective IoT Testing Framework
the system will interoperate with other systems in the real
world.
2) CHALLENGES FOR IoT CONFORMANCE TESTING
Previousely we introduced general conformance testing
method for the telecommunication industry, however the
requirement of human intervention and manual operation
of testing procedure by MOT in order to test EUT to RE
will limits scalability of IoT testing in terms of hindering
additional effort on managing test tasks, thus to promotes
rapid standards-based IoT product development from variety
of vendor, following property of conformance testing should
required to be advanced:
• Automatic Test Operation. The central issue on tradi-
tional IoT testing is human managed operation and test-
ing procedure, in order to support testing for rapidly
developed, larger amounts of device vendor requires to
automate such manual human interventions. To resolve
these issues automatically tests the IoT device using
given protocol and semantic information is required.
• Extensible Protocol Support. In order to remove hurdles
of standards-based IoT system, supporting scalable IoT
protocol is essential, it is not going to a problem pre-
viously, however in order to automate the testing proce-
dure to wide ranges of IoT devices that perform different
feature, flexibly supporting various of IoT-related proto-
col is required.
B. INTEROPERABILITY TESTING
Interoperability testing determines whether or not a target
product implementing a standard specification is interopera-
ble with other products implementing the same specification.
VOLUME 6, 2018
H. Kim et al.: IoT-TaaS: Toward a Prospective IoT Testing Framework
Interoperability testing generally refers to product debugging
and technology development by the interconnection of pro-
totype equipment [25]. Traditionally, interoperability testing
requires manufacturers to test their products in a same phys-
ical location with a network medium they can agree on and
perform tests on how two or more products communicate to
access particular interoperability functionality.
Interoperability testing does not verify if a product is com-
pliant with its referencing protocol standard specifications.
This testing defines a set of testing scenarios checking a
target software component which can interoperate with other
software components developed based on the same specifica-
tions but which is implemented by different manufacturers.
As a result, to perform proper testing of a target software
component, standards-based interoperability testing and con-
formance testing should be performed as complementary
mechanisms.
1) ARCHITECTURE FOR INTEROPERABILITY TESTING
The right side of Fig. 1 shows an example architecture for
interoperability testing [25] Like conformance testing, Test
Suite Structure (TSS) & Testing Purposes (TP) are generated
from the base standard specifications. TSS&TP is structured
to provide an informal description which focuses on what to
test. ATS is used for test case generation. IXIT and ETS play
the same role as conformance testing. The Interoperability
Functions Statement (IFS) is a checklist of the capabilities
and functionalities supported by the EUT; however, since IFS
concentrates on interoperability, it is not as detailed as EUT.
The IFS simply states a procedural scenario describing how
a target device should exchange messages with a test system.
The IFS can be used to select and parametrize test cases and
operate as an indicator for basic interoperability between dif-
ferent products. Test Descriptions (TD) is a detailed descrip-
tion of test processes and it addresses configurations, for the
performance of interoperable functionalities between IUT.
2) CHALLENGES FOR IoT INTEROPERABILITY TESTING
Previously, we show common interoperability testing used for
telecommunication industry, however, it is difficult to apply
the testing directly for the IoT ecosystem since the divergent
of vendor and feature of IoT devices exceeds feasible human
managed interoperable test, specifically, issues on various
test model involving differently configured, multiple target
device and human, location dependency should be resolved in
order to make feasible interoperability testing for IoT devices:
• Testing between Remote IUT. The major issues
regarding traditional interoperability testing is location
dependency; unlike conformance testing, interoperabil-
ity testing requires custom reference equipment which
results in significant delay of time by human effort,
moreover just like conformance testing, it requires to
manage and operate the test procedure by test operator;
In order to resolve these issues, remotely performed
testing is essential in order to save cost like time and
effort for manual coordination.
VOLUME 6, 2018
• Testing between Distributed IUT. Another important
aspect of interoperability is the configuration of inter-
operable devices which can be varied upon the number
of participant devices of the collaborative behavior to
be tested. At this perspective, interoperability for the
IoT devices require to support testing for interoperation
of multiple, distributedly placed devices, in traditional
model, this work requires human effort to coordinate
and configure testing scheme for multiple devices, thus,
automatic support distributed testing is essential for fea-
sible interoperability testing for the IoT device.
C. DISCUSSIONS AND CHALLENGES
Although the traditional testing methodology is aimed at
practical conformance and interoperability testing, it has
revealed several problems for testing IoT devices in the fol-
lowing aspects.
• Testing Coordination. Testing IoT devices involves
larger and more heterogeneous stakeholder groups than
in other typical software domains. This delays the testing
execution and negatively impacts the development cycle
of standards-based IoT products, which is a serious issue
at the time to market is critical.
• Testing Costs. In the same way as testing coordi-
nation, traditional testing requires costly face-to-face
testing, which creates an entry barrier to new com-
panies or start-ups willing to enter a market with a
standards-based product. High-costs in testing also dis-
courages standards-based product development in large
companies.
• Testing Scalability. The heterogeneity of IoT devices
introduces many testing stakeholders primarily for a
vendor of RE (Reference Equipment) and IUT as well
as underneath devices of RE to simulate complex coop-
eration. To support scalability and variability of testing,
dynamic test-beds are essential, as traditional testing
does not provide the necessary protocols or systems to
achieve such scalability.
As a result of such issues the solutions are proposed for
scalable automatic test with extensible protocol, automatic
coordination support and testing on remote and distributed
IUT. In the following section, we will discuss various IoT
testing related research activities and techniques to design a
novel IoT testing framework resolving the issues above and
provides details of our proposed solutions.
III. TECHNIQUES FOR IoT TESTING
There is a high demand on IoT testing framework for
satisfying coordination, cost, heterogeneity, and scalability
requirements as discussed in the previous section. The role
and design of IoT-TaaS fulfills complementary testing fea-
tures used in traditional testing but also helps to resolve
the IoT-specific testing issues regarding coordination and
scalability. IoT-TaaS can help to reduce testing costs by
providing a converged testing spot. To do that, this section
15483

FIGURE 2. High-level architecture of F-Interop.
presents three ongoing research work: (a) F-Interop, which
introduces remote and distributed conformance and inter-
operability testing for the IoT, (b) oneM2M IoT testing
work, which standardizes specifications for conformance
testing oneM2MTester, which performs conformance testing
to check that the behavior of an IoT device is compliant
with the referenced standard specifications and, (c) FIESTA
semantic testing, which verifies that content of IoT messages
are properly annotated with semantic information based on
the referenced standards.
A. REMOTE INTEROPERABILITY TESTING
A test system for an IoT device should support web-based
remote testing features. An important aim of such a test
system is to provide a platform for remote testing to accelerate
the development speed of a standards-based IoT product [26].
A test system called F-Interop [14] provides the feature of
testing IoT devices located in remote places.
1) SYSTEM ARCHITECTURE OF THE IoT INTEROP. SYSTEM
Figure 2 shows the architecture of F-Interop [27]; the novelty
of F-Interop compared to traditional software testing is the
provision of remote testing in a pragmatic manner by having
a centralized testing area on the web. To implement this
idea, the test system provides an agent which is a secured
communication wrapper for the IUT. When the IUT connec-
tion is established through the agent, the test system creates
an isolated room for testing. When the configuration of the
environment is finished, test execution script, generated from
the test description, is executed. This execution consists of
three phases: STIMULI to trigger IUT to initiate a specific
behavior, CHECK to validate communication, for example,
to capture contents of a packet, and VERIFY so that IUT
would reflect the desired result. The Testing Analysis Tool
verifies the output data and behavior after communication.
The tool then generates a verdict for the test from PASS,
FAIL, and INCONCLUSIVE. The test system has its own
packet generator for input generation and provides a web
interface to allow status checking and testers to manually
input decisions or commands when needed as part of a test.
The architecture of F-Interop is designed to handle the
entire process of IoT interoperability testing, with a focus
15484
H. Kim et al.: IoT-TaaS: Toward a Prospective IoT Testing Framework
on the testing administration and related tasks including
capturing a trace and reporting the verdicts. All of these
operations are controlled and coordinated by Orchestrator
which administrates the cooperation of different testing com-
ponents: test session, message broker, and access control.
Communications are encrypted and managed by Event Bus.
They include control messages, data, and log packets being
transmitted to the central Event Bus which is implemented
in RabbitMQ.2 This centralized communication architec-
ture ensures the independence of each component, with the
messages containing routing keys and topics to indicate
how to route messages to the relevant input queues of the
components.
2) TESTING MODELS OF IOT INTEROP.
To reflect the scale and heterogeneity of IoT deployments,
the IoT test system has to support different configurations
which can be handled by the testbeds federation [28]. Five
test configurations have been identified as follows:
• Simple Conformance Testing. Tests the conformance
of a single IUT. Only functional behavior of an IoT
device or platform interfaces are checked. This model
is suitable for individual testing of an IUT.
• Simple Conformance and Interoperability Testing. Tests
conformance and interoperability of a single IUT with a
unique testbed in the test system. This is the basic model
for testing a new IUT within a standardized testbed
providing a reference implementation.
• Simple Conformance and Compound Interoperability
Testing. Tests conformance and interoperability of a sin-
gle IUT with multiple testbeds in the test system. This is
a suitable model for testing a new IUT when no reference
implementation has been identified.
• Compound Conformance Testing. Tests conformance
and interoperability of multiple IUTs with no testbed.
This is a model for testing cooperative and collaborative
behavior of multiple IUTs.
• Compound Conformance and Compound Interoperabil-
ity Testing. Tests conformance and interoperability of
multiple IUTs with multiple testbeds in a test system.
This is a model for highly interconnected environments
between IUTs and testbeds.
3) CASE STUDY - CoAP PROTOCOL TESTING
F-Interop is useable for testing constrained IoT protocols
including IETF 6LoWPAN [29], [30], CoAP and oneM2M.
In this section, we explain how F-Interop is used for testing
Constrained Application Protocol (CoAP), which is a well-
known protocol for constrained IoT devices.
As shown in Fig. 4, the CoAP interoperability test-
ing in F-Interop consists in analyzing the data exchanged
between CoAP IUTs via CoAP testing tool located at the
F-Interop Central Server. This testing tool can support three
2 RabbitMQ. Opensource message broker for advanced message queueing
protocol. https://www.rabbitmq.com/
VOLUME 6, 2018
H. Kim et al.: IoT-TaaS: Toward a Prospective IoT Testing Framework

FIGURE 3. GUI screen image of CoAP protocol testing at F-Interop.

interoperability scenarios to test a CoAP IUT located at a

remote location; (1) the IUT is connected to the F-Interop test
server, (2) the IUT is connected to other CoAP devices run-
ning on a testbed, (3) the IUT interacts, through the F-Interop
testing tool, with the IUT from other vendors. Using these
various testing scenarios, a User A can rigorously test the
interoperability of its IUT.
For example, there is a case where the sequence of events
for a remote CoAP device is to synchronize to already running
CoAP devices from other vendors. To test this case, two
CoAP IUTs from connected IoT testbeds can be introduced
into the F-Interop test fixture, the remote CoAP IUT is
declared as a root of the network, while the other two IUTs
from the testbed have to be synchronized to the remote IUT, FIGURE 4. Architecture of CoAP protocol testing at F-Interop.
and periodically re-synchronize to it. F-Interop CoAP Testing
Tool analyzes all the exchanged wireless packets to assess the
interoperability of the remote CoAP IUT. right log shows configuration and test case execution needed
The Fig. 3 shows actual F-Interop Web GUI (Graphical for stimulating automated IUT from User A’s IUT including
User Interface) for the User A. The GUI shows information method types and request parameter.
of test cases executed and queued, controls of the test session,
test logs and results. The top-left side of screen displays B. ADVANCED CONFORMANCE TESTING
queued and passed test cases with result verdicts, and top- Conformance testing on an IoT device is challenged by the
right side shows test session controls such as start, stop variability of communication protocols in use, and automated
and terminate for User A. On the bottom side, it displays testing can ease the handling of such variabilities. oneM2M 3
logs about resource and action needed for the test, including is a global standards initiative for developing IoT Service
information about which type of resources are to be tested Layer specifications to resolve such problems. The oneM2M
using which options and request types. In this case, the logs test working group is standardizing not only specifications for
of bottom-left show the URI of requested for resources, size typical testing methodologies (i.e., conformance and interop-
and request type constraint that need to be fulfilled by User erability testing) but also advanced testing features adequate
A’s CoAP IUT to make correct interoperability tests with 3 oneM2M - Standards for M2M and the Internet of Things
other CoAP IUTs from F-Interop testbeds, where the bottom http://www.onem2m.org/

VOLUME 6, 2018 15485


FIGURE 5. Conceptual model of TTCN-3 test system.
for the IoT environment (i.e., testing for binding protocols
and automated device testing). This section introduces how
IoT devices are being tested by the oneM2M initiative [15].
1) TESTING IoT BINDING PROTOCOLS
Various communication protocols are used by IoT devices to
deliver messages. For example, the HTTP is used for general-
purpose message transmission while the CoAP and MQTT
are used for light-weight message exchange [31]. To test
these various protocols, support of a conformance testing
tool is required. oneM2M conformance testing follows this
motivation; it aims to verify that IoT devices are implemented
correctly in respect with the protocol message format, and its
exchange procedures are defined by oneM2M specifications.
As shown in Fig 5., the conceptual TTCN-3 test system
consists of a collection of different components that interact
with each other in order to execute test cases. In this model,
the SUT Adapter (SA) is in charge of communication with the
SUT. The CODEC has functionalities to encode and decode
messages. Basically, because the TTCN-3 tool only sup-
ports standardized encoding schemes such as Abstract Syntax
Notation One (ASN.1), Basic Encoding Rules (BERs) and
Packed Encoding Rules (PERs), testers willing to test other
protocols such as HTTP, CoAP, and MQTT need to define
them [32]. Furthermore, oneM2M primitives must be bound
to each protocol encoding scheme according to the oneM2M
protocol binding specification4 so an additional step is needed
to bind the oneM2M primitive to the encoding schema. In this
regard, the test system supports the oneM2M conformance
testing by developing the oneM2M dedicated SA including
communication procedures and protocol binding procedures;
4 oneM2M TS-0008. CoAP Protocol Binding, v.1.3.2
oneM2M TS-0009. HTTP Protocol Binding, v.2.6.1
oneM2M TS-0010. MQTT Protocol Binding, v.2.4.1
http://www.onem2m.org/technical/published-documents
15486
H. Kim et al.: IoT-TaaS: Toward a Prospective IoT Testing Framework
FIGURE 6. Cloud-based automated IoT testing.
decoding procedures are executed in the same way. In other
words, any protocol can be supported with the extension of
the adapter functionalities.
2) AUTOMATED CONFORMANCE TESTING
Similar to other conformance testing framework, oneM2M
testing framework is designed to perform a traditional testing
lab based conformance testing, which means that developers
and vendors have to physically visit a testing lab to get confor-
mance testing. This typical conformance testing method is not
well adequate for the IoT domain as there will be huge SMEs
(Small to medium sized enterprises) and individual develop-
ers producing small amount of IoT devices. Therefore, it is
natural to investigate low cost IoT testing mechanisms. In
addition, IoT testing framework has to support various com-
munication protocols and standards [31], [33]. It is not easy
for SMEs and developers to construct a testing environment
to cover all these different protocols as well as testing features
such as monitoring, logging, and reporting.
A web-based remote conformance testing framework using
an automated testing feature is considered as a promising
solution to resolve these problems. Main ideas of web-based
IoT testing is to locate a core logic of conformance testing
system to the cloud and provide common test APIs for the
tester to select various configuration options based on their
needs. In addition, by allowing third parties to add their own
protocol system adaptors to the core testing system at the
cloud, web-based testing can easily adopt new IoT protocols.
The key to automated testing procedures for oneM2M
devices is to trigger the device to initiate communication of a
test case. This role is realized by using an UpperTester with
a network protocol used by the oneM2M device to ensure
flexibility to support various network protocols without mod-
ifying the test system. The UpperTester is a logical software
module converting a test indication from the testing system
to a message understandable by IUT. This UpperTester can
be implemented either within an IUT or outside of the IUT
depending on the capability of the IUT.
For a more practical example, oneM2M automated testing
procedures are described in Fig. 6 as follows: A tester who
wants to test a IoT device provides information about testing
VOLUME 6, 2018
H. Kim et al.: IoT-TaaS: Toward a Prospective IoT Testing Framework
FIGURE 7. GUI screen image of oneM2MTester showing tests verdicts over time, IUT configuration and statistics of testing.
configuration (e.g. protocols and device types) and selects test
cases to be performed using a web interface (Steps 1 and
2). Based on the inputs from the tester, the test system
sends a triggering message including specific test cases and
configuration information to the UpperTester of the target
IoT device. The Upper Tester parses the triggering mes-
sage from the test system and performs an operation of the
oneM2M application based on the given instructions in the
message (Step 3). One of oneM2M operations (i.e., CREATE,
RETRIEVE, UPDATE, DELETE and NOTIFICATION) is
being instructed to the target device by the UpperTester
(Steps 4 and 5). In order to perform the above testing
procedures, a minimum consensus is required between the
UpperTester and the target IoT device. After receiving the
test case information from the UpperTester, the IoT device
has to performs the specific operation based on the test cases
described in the triggering message. Finally, the test sys-
tem creates verdicts of conformance testing for IoT devices
by verifying that response messages conform to the stan-
dard [34].
3) CASE STUDY - oneM2M WEB-BASED TESTER
oneM2MTester is a web-based oneM2M IoT device testing
framework. Developers and SMEs who want to test their IoT
devices implemented based on oneM2M standards specifica-
tions can configure the oneM2MTester and select required
test cases for their IoT devices. In this way, they can test
their devices without visiting IoT testing labs. Thanks to
oneM2MTester’s remote IoT device testing feature, develop-
ers can debug their products and reduce development cost.
VOLUME 6, 2018
Let us show a specific example how oneM2MTester can be
used to test a oneM2M feature. We can consider a case that
a developer wants to test the Access Control Policy (ACP)
of their IoT device. ACP test cases are constructed to check
that the IUT properly creates ACP resource to the server. First,
the tester configures IUT as an IoT product that supports ACP
and selects the appropriate ACP test cases. oneM2MTester
then performs the selected test cases with the IUT and gen-
erates verdicts for all test cases. In order to get ‘‘PASS’’
verdict, the IUT has to create the correct ACP resource to
the server system. If the created ACP resource does not con-
tain proper value addressed in the test cases, oneM2MTester
shows the reason of the failure with a ‘‘FAIL’’ or ‘‘INCON-
CLUSIVE’’ verdict. Thus, at the end of the test campaign,
the developer will get immediate feedback from test results
and detailed logs including the verdict of the selected test
cases as well as recommendations to correct failures. Fig. 7
shows the captured screenshot of oneM2MTester. The GUI
of oneM2MTester shows an overview of the testing results
in history over an IUT implemented based on oneM2M.
A chart represents the number of test cases and their ver-
dicts in their different states and dates. The verdicts of each
test case are highlighted with different colors, i.e. green
(‘‘PASS’’), yellow (‘‘INCONCLUSIVE’’) and red (‘‘FAIL’’).
In addition, Fig. 7 shows the presence for a testing verdict
history and an overview of the IUT configuration such as
software version, protocols, address.
C. SEMANTIC TESTING FOR IoT DEVICES
Testing of IoT devices has to be made at different levels.
While the previous section described interoperability and
15487

conformance testing focusing mainly on protocol-level, this
section discusses semantic testing. Semantic Testing aims to
test the correctness of semantic description from the data
stream of IoT devices against standards. The work that has
been done on these challenges in relation to semantic-web
concludes that one of the core aspects to achieve seman-
tic interoperability is the ontology and the annotated data
using these ontologies. Standardization bodies have already
defined reference ontologies such as W3C-SSN ontology,5
ETSI-SAREF ontology,6 and oneM2M base ontology.7 Once
reference ontologies have been defined, one important step to
ensure semantic interoperability is to test conformity against
such reference ontologies.
Semantic testing is performed to target different level of
validation in a semantic description. From a lexical and syn-
tactic validation to a logical and semantical one. Semantic
testing for the IoT environment is especially challenging
due to the large diversity of semantic modeling methods.
For example, discrete event models have a global notion of
time, while the finite state machine is an untimed model.
Among this heterogeneity, semantic models could be also
complex due to a large number of concepts and relations
between these concepts, for example, semantic model like
IoT-Lite [35] optimizes performance to handling large and
diverse semantic model of IoT by partially containing a light-
weight semantic model in order to improve query responsive-
ness for IoT ontologies.
Several research projects are working to propose and
apply novel semantic evaluation methods and tools to resolve
these challenges. The H2020 Fiesta-IoT 8 project is one of
these projects aiming to establish a unique cloud platform
of semantically federated IoT testbeds for novel experi-
ments based on semantic technologies. The platform pro-
vides a unique access point to access semantically enriched
data coming from different types of testbeds, for example,
crowdsensing applications, smart cities, smart campus and
smart buildings. For a proper semantic database that provides
correct data regarding the agreed ontology to be used by
experimenters relying on standard semantic technology, our
semantic validation work is applied on the incoming data
stream. If the data does not meet all the requirements regard-
ing the syntactic and semantic correctness, it will be rejected
to keep the database clean and correct. Thus, it will provide
a full report with a description of all pitfalls that could lead
to modeling errors; this will assist the ontology developers
during the correction process.
Semantic testing is a must for achieving semantic inter-
operability. The presented case demonstrates the neces-
5 Semantic Sensor Network Ontology
https://www.w3.org/2005/Incubator/ssn/ssnx/ssn
6 ETSI Smart Appliance Reference (SAREF) ontology
http://www.etsi.org/technologies-clusters/technologies/smart-appliances
7 oneM2M TS-0012. oneM2M Base Ontology, v2.0.0
http://www.onem2m.org/technical/published-documents
8 Fiesta-IoT: Federated Interoperable Semantic IoT Testbeds and Applica-
tions http://fiesta-iot.eu
15488
H. Kim et al.: IoT-TaaS: Toward a Prospective IoT Testing Framework
sity of such testing in the IoT industry to make sure
that an IoT system is properly working as expected.
As we argued, semantic testing needs to cover multi-
ple levels of semantic interpretation, such as lexical, syn-
tactical, semantical, and cardinal correctness, which are
critical for semantic interoperability. More optional test fea-
tures, such as following the best practice for the linked
data [36], should also be considered for inclusion in semantic
testing.
1) CASE STUDY - SEMANTIC VALIDATOR
One of the main reasons for having semantic validator is
to check the semantic annotation (annotated data against an
ontology). For example, there is an IoT platform introducing
a meta-testbed IoT / Cloud infrastructure to federate various
IoT testbeds distributed around the world. Such platforms
aim to break the barriers between technology and data silos
by applying the semantic technology. Any users who want
to have an access to their data regardless of their location
and connected IoT platforms can send his data that have
been annotated against a common ontology to a semantic
repository. In this perspective, the user needs to check the
consistency of its annotation, otherwise the platform will
decline its data. In this manner, the semantic validation tool is
a key element of IoT platforms to assist the user to generate
a correct annotation. In fact, the tool provides a detailed
validation report that enumerates the different errors.
Typically a semantic validator contains three main compo-
nents that perform a specific aspect of the semantic valida-
tions for a target IoT resource as follows:
• Namespace and URI validation. URI (Uniform
Resource Identifier) and namespace validation are
important for keeping the validity of ontology, and it is
a basic step for semantic validation. A ‘‘bad namespace
URI’’ means that the validator does not accept the URI
for a namespace in the model. When an ontology is
given as an input to a semantic validator, it shows prefix
of validated namespace and shows a result with error
messages. For example, as Fig. 8 shows, in Namespace
and URI validation section of a semantic validator that
we have implemented at the FIESTA project, an error
on multiple use of different prefixes for namespace was
reported when a target resource uses wrong prefixes in
its semantic annotation. The list in Fig. 9 shows that
there are multiple prefix declarations for NameSpace,
namely, ns0 and xmlns which violate the consistency
of namespaces.
• Predicate and Class validation. The validation tool
treats the imported ontology models, and (indepen-
dently) the specified schemas, as a single Ontology
model via an integration process. Then the tool extracts
the properties from the derived single Ontology model.
It includes the Resource Description Framework (RDF)
and RDF Schema (RDFS). Anything used as a predicate
that is not defined as one of those properties is reported
as a definition error.
VOLUME 6, 2018
H. Kim et al.: IoT-TaaS: Toward a Prospective IoT Testing Framework
FIGURE 8. GUI image of semantic validator showing namespace validation results.
FIGURE 9. Example of invalid resource prefixes.
•Semantic validation. The validation tool can determine
whether or not the ontology is consistent, identify sub-
sumption relationships between classes, etc. In order
to perform semantic validation, a developer can upload
ontology to the tool, then it returns a report after check-
ing the syntactic for ontology format, namespaces, and
URI validation. In addition, the tool shows the semantic
correctness of predicate. For example, let us consider the
annotation list in Fig. 10. The list contains two classes
that are defined as a common class, i.e. adult Person
and young Person, which are disjoint each other. As the
list define an instance (Bob) which is a person with the
young and adult classes, it contains inconsistency.
If a developer enters such annotated resource to the
validator, a semantic or logical error will occur by the
inconsistency of the annotation.
The tool is available online in the certification portal 9 of
FIESTA.
9 http://certificate.fiesta-iot.eu
VOLUME 6, 2018
FIGURE 10. Example of invalid semantic resource.
IV. TOWARDS IoT-TaaS
In the previous section, we discussed several key technologies
for IoT testing. Traditional conformance and interoperability
testing approaches should be evolved to handle the diver-
sity of IoT systems and devices. In addition, we realized
that semantic validation is another important step for an
IoT testing framework. In this section, we present IoT-TaaS,
a prospective framework for IoT testing. As shown in Fig. 11,
the framework proposes three major steps of IoT testing.
A. REMOTE DISTRIBUTED INTEROPERABILITY TESTING
The first step of IoT-TaaS is to perform interoperability test-
ing involving remote devices and thus supports the place-
ment of test subjects which can be physically distributed.
For example, a target IoT device can be tested for its inter-
operability with another similar device placed in a testing
lab as well as randomly selected field devices running in
the real world. To support this, the Distributed Interop-
erability Testing component should provide the following
features:
• Agent Wrapper. Such a test system involves two main
components, the client and server (the test server). The
client is distributed with a unique test server. Clients
need to be wrapped to connect to the common server
without code modification, and they need to support the
15489

FIGURE 11. Architecture of cloud-based IoT testing as a service.
standard way of getting the IUT communicating to the
testing server with monitoring and security.
• Configurable Testing Schema. The test server needs
to provide a configurable testing scheme adapted to a
variety of testing combinations. As a consequence of
this, the corresponding architecture of the server has to
implement the separation of concern such as decoupling
of testing coordination, execution, and interaction with
the IUT and user, because one to one mapping of each
resource can not support a configurable testing scheme.
B. SCALABLE AUTOMATED CONFORMANCE TESTING
As a second step, IoT-TaaS needs to provide scalable sup-
port, handling variables in communications as well as auto-
mated support to create verdicts from the conformance tests.
Because many protocols are used by IoT devices in the mar-
ket, easy integration of various protocol adapters is essential
for an IoT testing framework. In addition, for usability and
distributed testing, we propose adding an automated IoT
device testing feature to the framework as follows:
• Protocol Adapter. IoT devices need scalable testing
using various protocols for different domains of applica-
tion. For example, the request-response protocol based
on TCP (Transmission Control Protocol) connection is
used for normal integrity centric data communication
whereas a protocol such as publish-subscribe is used if
the environment needs real-time communication. In IoT-
TaaS, both types of protocol are currently supported in
scalable by the External Codec.
• Automated device testing. In order to test IoT devices,
a test system has to trigger various transactions imple-
mented in the target device. Typically stimulus prepared
by developers are used to trigger a required action.
A standardized triggering message format and transac-
tion types are defined in IoT-TaaS so that IoT device
developers simply need to embed a testing code to per-
form automated device testing.
15490
H. Kim et al.: IoT-TaaS: Toward a Prospective IoT Testing Framework
C. SEMANTICS VALIDATION TESTING
One of the core aspects to achieve semantic interoperability
is ontology and the ontology-based data annotation. There-
fore, an evaluation of such ontologies and annotated data is
essential to determine if they are interoperable with other
ontologies, especially the reference ontologies recognized by
standardization bodies such as SSN ontology. In this perspec-
tive, Semantic Validation Testing is introduced as the final
step of IoT-TaaS to validate the meaning of messages on top
of the communication protocol.
This step uses a tool confirming the correctness of semantic
data. The semantic data validation tool can be implemented
as a web service integrated within a web-based client-server
architecture. Through the Graphical User Interface (GUI),
a user is able to upload his ontology and semantically anno-
tated data to be validated against one or several reference
ontologies such as the SSN ontology. The validator can then
detect syntactic and semantic issues, if there are any, and pro-
duce a detailed test report at the end of the process which will
help the user to correct issues. As described in Fig. 12, there
are three main functionalities in the core of the validation tool,
namely XML Parser or JSON Parser according to the format
of the input file, RDF Parser, and Validation:
• XML and JSON Parser. If a document containing data is
indicated as XML10 or JSON 11 along with the extension
of the two formats (e.g. RDF/XML, OWL/XML and
JSON-LD), the XML and JSON parser check if the syn-
tax of XML or JSON is respected. If it is not validated,
the validation process will not proceed further.
• RDF Parser. This module takes as input document either
a validated XML or JSON file or a file in another
supported format and verifies if the data represents a
valid RDF model. If it respects the specification of the
10 Extensible Markup Language (XML) 1.0 (Fifth Edition)
https://www.w3.org/TR/REC-xml/
11 ECMA-404 The JSON Data Interchange Standard
https://www.ecma-international.org/publications/standards/Ecma-404.htm
VOLUME 6, 2018
H. Kim et al.: IoT-TaaS: Toward a Prospective IoT Testing Framework
FIGURE 12. An Architecture of the Ontology Validator Web Application.
RDF model, triples in this model are extracted to serve
as the input for the next validation step in the validation
module.
• Validation. This module takes the reference ontology
constructed from the different checked ontologies as an
input of the validation and the triples extracted from
the previous step. Then, according to the predefined
reference ontology, it checks for syntactic errors in the
testing document which is based on the functionalities
implemented in Eyeball, an Apache Jena ontology val-
idator.12 A reasoner is also used to enable logical level
verification of the RDF document. The validation results
are sent to a reporting server shows a list of errors and
explanations regarding the ontology affected elements.
D. CONCEPT OF THE IoT-TaaS ARCHITECTURE - PLUG
AND TEST
IoT platforms offer services to applications developers. The
question of conformance testing, interoperability testing, and
semantic validation of IoT platforms can be tackled with the
same ‘‘as a service’’ approach to the testing. The vendors of
IoT products are diverse, and thus a service-oriented approach
to testing is a good option to offer an easy and reconfigurable
approach for the development of various IoT products.
The three phases of IoT testing can be defined as
conformance testing, interoperability testing and semantics
validation testing. For IoT-TaaS, we presented scalable auto-
mated conformance testing. It is scalable and supports various
protocols in a modular manner, and it contains automated
embedded triggering mechanisms. Semantic validation test-
ing is present to verify messages with semantic informa-
tion generated by IoT devices. This semantic testing phase
confirms that the message is syntactically and semantically
correct. Finally, remote distributed interoperability testing is
included to overcome the physical constraint of IoT device
testing. Such a remote feature also benefits testers by not only
connecting IUTs to standard IoT devices but by also offers
connecting various IoT field devices in the IoT testbeds. This
interoperability testing also enables various testing config-
urations without physically wiring IoT devices to the test
system. Establishing a network connection from anywhere
in the world to IoT-TaaS is enough to enable testing IoT in
standards.
12 https://jena.apache.org
VOLUME 6, 2018
In this paper we introduce a set of advancements to
traditional conformance and interoperability testing con-
cepts together with semantic validation for testing vari-
ous IoT devices and platforms. As these concepts are now
implemented and adopted as a tool for IoT certifications,
we strongly believe that these concepts will play key roles in
a modern IoT testing framework. These three key IoT testing
concepts can even be evolved to a ‘‘plug and test’’ concept of
IoT-TaaS. This plug and test concept offers an easy extension
of functionalities of the test systems without modifying the
test system itself. For example, conformance testing can plug
testing specs (incl. test cases and profiles) and perform testing
on IUTs. Semantics validation testing can also enable various
ontologies depending on which ontology is used in a target
document and validate the correctness of used semantics
in the document. Finally, interoperability testing offered by
IoT-TaaS can enable other field IoT devices to check a target
IoT device. If there is a need to add another type of testing
to IoT-TaaS, a new set of test cases, ontology information,
and new messages can easily be added to the system. Such
a design concept could offer a practical architectural concept
to practitioners who design and implement IoT-TaaS for their
IoT ecosystems. We believe IoT-TaaS provides significant
benefits to accelerate the development of standards-based IoT
products.
V. CONCLUSION
This paper presents IoT-TaaS, a new testing approach trans-
forming standards-based IoT testing into a service-oriented
testing framework that enables scalable, distributed, and auto-
mated IoT testing. An analysis of traditional software test-
ing methodologies, i.e., conformance and interoperability
testing, is conducted in order to retrieve key challenges of
IoT testing. Therefore, an insight on how traditional test-
ing methodologies have to be evolved to provide a testing
framework adequate for IoT testing is given. We introduced
three ongoing research projects currently inventing new IoT
testing techniques as they provide fundamental concepts and
techniques to IoT-TaaS. IoT-TaaS provides a generalizable
concept and design method for testing modern IoT products
by showing differences in automated conformance testing,
semantics validation testing, and remote distributed interop-
erability testing.
ACKNOWLEDGEMENT
We thank Federico Sismondi for sharing a case study for the
F-Interop project. All authors made equal contributions to the
study and the publication.
REFERENCES
[1] M. Schmidt, A. Wilde, A. Schulke, and H. Costa, ‘‘IMS interoperabil-
ity and conformance aspects [IP multimedia systems (IMS) infrastruc-
ture and services],’’ IEEE Commun. Mag., vol. 45, no. 3, pp. 138–142,
Mar. 2007.
[2] A. Vallejo, J. Ruiz, J. Abella, A. Zaballos, and J. M. Selga, ‘‘State of the art
of ipv6 conformance and interoperability testing,’’ IEEE Commun. Mag.,
vol. 45, no. 10, pp. 140–146, Oct. 2007.
[3] R. Hao, D. Lee, R. K. Sinha, and N. Griffeth, ‘‘Integrated system interoper-
ability testing with applications to VoIP,’’ IEEE/ACM Trans. Netw., vol. 12,
no. 5, pp. 823–836, Oct. 2004.
15491

[4] S. Maag and C. Grepet, ‘‘Interoperability testing of a manet routing pro-
tocol using a node self-similarity approach,’’ in Proc. ACM Symp. Appl.
Comput., 2008, pp. 1908–1912.
[5] S. Seol, M. Kim, S. Kang, and J. Ryu, ‘‘Fully automated interoperability
test suite derivation for communication protocols,’’ Comput. Netw., vol. 43,
no. 6, pp. 735–759, 2003.
[6] ‘‘More than 50 billion connected devices,’’ Ericsson, White Paper
284 23-3149 Uen, 2011.
[7] S. Ziegler et al., ‘‘IoT6—Moving to an IPv6-based future IoT,’’ in
The Future Internet Assembly (Lecture Notes in Computer Science),
vol. 7858, A. Galis and A. Gavras, Eds. Berlin, Germany: Springer, 2013,
pp. 161–172.
[8] D. Bandyopadhyay and J. Sen, ‘‘Internet of Things: Applications and
challenges in technology and standardization,’’ Wireless Pers. Commun.,
vol. 58, no. 1, pp. 49–69, 2011.
[9] J. Song, A. Kunz, M. Schmidt, and P. Szczytowski, ‘‘Connecting and man-
aging M2M devices in the future Internet,’’ Mobile Netw. Appl., vol. 19,
no. 1, pp. 4–17, 2014.
[10] R. Fielding et al., Hypertext Transfer Protocol—HTTP/1.1, document
RFC 2616, Network Working Group, 1999.
[11] Z. Shelby, K. Hartke, and C. Bormann, The Constrained Application
Protocol (COAP), document RFC 7252, Internet Engineering Task Force
(IETF), 2014.
[12] C. Bormann, ‘‘Test descriptions for ETSI plugtests coap#4,’’ Eur. Telecom-
mun. Standards Inst., London, U.K. Tech. Rep. 7-9, Mar. 2014.
[13] OASIS Message Queuing Telemetry Transport (MQTT) TC, MQTT Ver-
sion 3.1.1, OASIS Standard mqtt-v3.1.1-os, 2014.
[14] S. Ziegler, S. Fdida, C. Viho, and T. Watteyne, ‘‘F-Interop—Online plat-
form of interoperability and performance tests for the Internet of Things,’’
in Proc. Int. Conf. Interoperability IoT, 2016, pp. 49–55.
[15] oneM2M Testing Framework, document oneM2M TS-0015 v2.0.0,
Aug. 2016.
[16] P. Cousin, ‘‘White Paper on flexible approach for semantic validation
in the context of Internet of Things,’’ Easy Global Market, Valbonne,
France, White Paper, 2016. [Online]. Available: http://www.eglobalmark.
com/white-paper-flexible-approach-semantic-validation-context-internet-
things/
[17] E. S. Reetz, D. Kuemper, K. Moessner, and R. Tönjes, ‘‘How to test iot-
based services before deploying them into real world,’’ in Proc. 19th Eur.
Wireless Conf. (EW), 2013, pp. 1–6.
[18] S. De, F. Carrez, E. Reetz, R. Tönjes, and W. Wang, ‘‘Test-enabled archi-
tecture for IoT service creation and provisioning,’’ in The Future Internet
Assembly (Lecture Notes in Computer Science), vol. 7858, A. Galis and
A. Gavras, Eds. Berlin, Germany: Springer, 2013, pp. 233–245.
[19] A. Gluhak, S. Krco, M. Nati, D. Pfisterer, N. Mitton, and
T. Razafindralambo, ‘‘A survey on facilities for experimental Internet
of Things research,’’ IEEE Commun. Mag., vol. 49, no. 11, pp. 58–67,
Nov. 2011.
[20] B. Shala, P. Wacht, U. Trick, A. Lehmann, B. Ghita, and S. Shiaeles,
‘‘Framework for automated functional testing of P2P-based M2M applica-
tions,’’ in Proc. 9th Int. Conf. Ubiquitous Future Netw. (ICUFN), Jul. 2017,
pp. 916–921.
[21] P. Rosenkranz, M. Wählisch, E. Baccelli, and L. Ortmann, ‘‘A distributed
test system architecture for open-source IoT software,’’ in Proc. Workshop
IoT Challenges Mobile Ind. Syst., May 2015, pp. 43–48.
[22] D. Kuemper, E. Reetz, and R. Tönjes, ‘‘Test derivation for semantically
described IoT services,’’ in Proc. Future Netw. Mobile Summit (FutureNet-
workSummit), 2013, pp. 1–10.
[23] M. Vögler, J. Schleicher, C. Inzinger, S. Nastic, S. Sehic, and
S. Dustdar, ‘‘LEONORE—Large-scale provisioning of resource-
constrained iot deployments,’’ in Proc. IEEE Symp. Service-Oriented Syst.
Eng. (SOSE), Mar. 2015, pp. 78–87.
[24] M. Leotta et al., ‘‘Towards an acceptance testing approach for Internet of
Things systems,’’ in Proc. 1st Int. Workshop Eng. Web Things, 2017.
[25] S. Moseley, S. Randall, and A. Wiles, ‘‘Experience within ETSI of
the combined roles of conformance testing and interoperability test-
ing,’’ in Proc. 3rd Conf. Standardization Innov. Inf. Technol., Oct. 2003,
pp. 177–189.
[26] O. Vermesan and P. Friess, Internet of Things—From Research and Innova-
tion to Market Deployment, vol. 29. Aalborg, Denmark: River Publishers,
2014.
[27] R. Leone, F. Sismondi, T. Watteyne, and C. Viho, ‘‘Technical overview
of F-Interop,’’ in Proc. Int. Conf. Interoperability IoT (InterIoT), Paris,
France, 2016, pp. 11–17.
15492
H. Kim et al.: IoT-TaaS: Toward a Prospective IoT Testing Framework
[28] S. Ziegler, M. Hazan, H. Xiaohong, and L. Ladid, ‘‘IPV6-based test beds
integration across Europe and China,’’ in Proc. Int. Conf. Testbeds Res.
Infrastruct., 2014, pp. 87–96.
[29] G. Montenegro, N. Kushalnagar, J. Hui, and D. Culler, Transmission of
IPV6 Packets Over IEEE 802.15.4 Networks, document RFC 4944, 2007.
[30] ETSI 6LoWPAN Plugtests Test Descriptions, document V1.0.0 2013-07,
ETSI, Berlin, Germany, Jul. 2013.
[31] B. Ahlgren, M. Hidell, and E. C.-H. Ngai, ‘‘Internet of Things for smart
cities: Interoperability and open data,’’ IEEE Internet Comput., vol. 20,
no. 6, pp. 52–56, Nov. 2016.
[32] C. Willcock, An Introduction to TTCN-3. Hoboken, NJ, USA: Wiley, 2011.
[33] I. Schieferdecker, S. Kretzschmann, A. Rennoch, and M. Wagner, ‘‘IoT-
testware—An eclipse project,’’ in Proc. IEEE Int. Conf. Softw. Quality, Rel.
Secur. (QRS), Jul. 2017, pp. 1–8.
[34] Functional Architecture, document oneM2M TS-0001-V2.10.0,
Aug. 2016.
[35] M. Bermudez-Edo, T. Elsaleh, I. P. Barnaghi, and K. Taylor, ‘‘A lightweight
semantic model for the Internet of Things,’’ in Proc. Int. IEEE Conf.
Ubiquitous Intell. Comput., Adv., Trusted Comput., Scalable Comput.
Commun., Cloud Big Data Comput., Internet People, Smart World Congr.
(UIC/ATC/ScalCom/CBDCom/IoP/SmartWorld), 2016, pp. 90–97, doi:
10.1109/UIC-ATC-ScalCom-CBDCom-IoP-SmartWorld.2016.0035.
[36] (2017). Linked Data—Connect Distributed Data Across the Web. [Online].
Available: http://linkeddata.org/
HIUN KIM is currently pursuing the B.S. degree
in computer science and engineering from Sejong
University. He is a Researcher with the Software
Engineering and Security Laboratory and cur-
rently with Naver Corporation as a Software Engi-
neer. His research aims to extend boundaries on
principles and applications of computing to solve
more advanced problems, interested in broad areas
of computer science, including programming lan-
guages, software engineering and computer sys-
tems for advance easy, safe, and efficient design and implementation of
software.
ABBAS AHMAD received the master’s degree in
computer sciences and the Ph.D. degree from Easy
Global Market from the University of Franche-
Comté. He specialized his master’s degree in
design and development of secure software. He
has accomplished a six months internship with the
Research and Development Department, Airbus
Operations SAS, where he developed Isami Ana-
lyst Testing Tool, a non-regression test tool. He
also have an International Software Testing Qual-
ifications Board (ISTQBő) foundation level certification, a ISTQBő Model-
Based Tester foundation, a FIWARE developer’s week Brussels diploma,
and an TTCN-3 certificate in test automation. With a strong knowledge of
software and application testing, he also have been a FIWARE friendly tester
during four months, where he tested the FIWARE cloud portal and some
Generic Enablers. One of his achievement’s is winning first prize of the Inter-
net Of Things Hackaton during the IoT Week, Lisbon, 2015. He is currently
running a Ph.D. thesis in the field of model-based testing for integration of
distributed and heterogeneous future Internet Web components.
JAEYOUNG HWANG received the bachelor’s
degree in computer engineering from Sejong Uni-
versity, Seoul, South Korea, in 2015, where he
is currently pursuing the Ph.D. degree. He is a
member of the Software Engineering and Security
Laboratory and also an Assistant Researcher with
the IoT Platform Research Center, Korea Electron-
ics Technology Institute. He contributed a lot in
the oneM2MTester tool, which is the conformance
testing tool to support the oneM2M standard.
VOLUME 6, 2018
H. Kim et al.: IoT-TaaS: Toward a Prospective IoT Testing Framework
HAMZA BAQA received the master’s degree in
computer sciences from INPT and Mines-Telecom
saint Etienne. He is currently pursuing the Ph.D.
degree with Easy Global Market in collaboration
with Mines-Telecom, France. He specialized the
master’s degree in Security and information sys-
tem management. He also certified web master in
Bridge Value and IBM SPPS Modler tool (Data
Mining). With a strong knowledge of software and
semantics and has been active in Fiesta H2020 EU
project where his research interests include Internet of things (IoT), semantic
web, and big data, and he is currently involved in the WISE-IoT project,
contributing to Quality of Information evaluation.
FRANCK LE GALL is currently a CEO with
Easy Global Market, an innovative SME focused
on integration and validation of emerging tech-
nologies. He has participated in large research
and development projects within the big industry
(Orange, Alcatel, and Thomson) and spent nine
years as the Director of innovation management
company. He directed over ten large scale projects
and studies related to the evaluation and monitor-
ing of innovation and technical programs and also
research projects. He is driving a company for the development of advanced
testing technologies and also integration of IoT and data platforms. He
involves himself in standardization area including oneM2M and FIWARE.
He is currently participating in several EU funded projects including tech-
nical management in PROTEUS (advanced sensors for water monitoring),
FIESTA (IoT semantic interoperability), FESTIVAL (EU-JP cooperation
on IoT), U-Test (including uncertainties in cyber physical system testing),
PHANTOM (model-based testing of heterogeneous distributed systems),
and Wise-IoT (EU-KR project on IoT interoperability). He was also a SME
Coach within the FIWARE program.
VOLUME 6, 2018
MIGUEL ANGEL REINA ORTEGA received the
degree in telecommunications engineering from
the University of Malaga, Spain. He is currently a
Technical Officer with the Centre for Testing and
Interoperability, European Telecommunications
Standards Institute (ETSI). His research has been
based on development of interoperability and con-
formance test specifications for communication
technologies following ETSI’s testing methodol-
ogy recommendations and using TTCN-3 lan-
guage. He was with At4Wireless as a Testing Engineer in interoperability
and conformance testing for Radius Protocol Tester, Wimax Radio Confor-
mance Tester, and Wimax Protocol Conformance Tester, and at the same
time, he was participating as ETSI Specialist Task Force expert during two
years in test suite development for WiMAX/HiperMAN. He joined ETSI
in 2007 and continued in developing and validating network and protocol
WiMAX/HiperMAN test specifications. After this, he has contributed to
develop other conformance and interoperability test specifications for several
technologies such as SIP, ITS protocols (CAM, DENM, and GeoNetwork-
ing), vehicle-to-grid protocol, and oneM2M, where he is leading the confor-
mance test specifications development. Besides the technical skills, he has
managed and led several FP7 projects, such as TESTCASE, MOSQUITO,
and PowerUp, where testing activities have been performed and successful
results have been achieved.
JAESEUNG SONG (SM’17) received the B.S. and
M.S. degrees in computer science from Sogang
University, and the Ph.D. degree from the Depart-
ment of Computing, Imperial College London,
U.K. He holds the position of the oneM2M Test
Working Group Chair and the TTA IoT/M2M
Convergence Special Project Group Chair. From
2002 to 2008, he was with LG Electronics as a
Senior Researcher leading the 3GPP SA Standard
Team. He was with NEC Europe Ltd., as a leading
standard Senior Researcher, from 2012 to 2013. He is currently an Associate
Professor, leading the Software Engineering and Security Laboratory, Com-
puter and Information Security Department, Sejong University. His research
interests include software engineering, software testing, networked systems
and security, with focus on the design and engineering of reliable IoT/M2M
platforms, particularly in the context of semantic IoT data interoperability,
secure software patch techniques, blockchain IoT, and edge computing.
15493