2018 8th International Conference on Computer Science and Information Technology (CSIT) ISBN: 978-1-5386-4152-1

Software Testing Techniques in IoT

Ghadeer Murad1, Aalaa Badarneh2, Abdallah Qusef3, Fadi Almasalha1

1
Computer Science Department, Applied Science University
2
Jordanian Engineers Association
3
Software Engineering Department, Princess Sumaya University for Technology
Amman, Jordan
ghader.murad@asu.edu.jo, eng.aalaabadarneh@gmail.com, a.qusef@psut.edu.jo, f_masalha@asu.edu.jo

Abstract- As well as the important industry 4.0 revolution, based
on of the serves that IoT provides in our society in different fields
such as smart building, factory, mobility, health care. Previous
studies explore various technology solution to find the best
technique to test IoT applications in order to ensure quality for
IoT devices.
The paper surveys diverse of aspects of multiples software
testing and tools for IoT devices, and provides details in use case
testing for IoT environment and test different accepts such as
usability, security, connectivity.
The paper considers various security requirement and
challenges an unveils different research problems in testing IoT
applications and proposes multiple tools and software techniques
that help to enhance IoT applications quality
Keywords:- Software Testing, IoT, Testing Methodologies;
Testing Tools;
I. INTRODUCTION
The internet of things (IoT) is a system of interrelated
computing object device, the digital machine which connect to
each other and able to exchange and analyze data over a
network.
A common component of IoT system is a sensor, network,
data center, and application Benefits typically in IoT in cloud
connectivity of this device that provides a appropriates
solutions for wide range of application in different files
searches such as emergency sensors, traffic .smart city health
care …etc.
Nowadays we need to deliver faster and better serves .as
well as there are a huge access and transaction in IoT devices
in real time. Moreover.The goal of IoT testing is to provide
better insight and control interconnected various IoT devices
[17].
As well as testing IoT has multi-type that help to inshore
that every component of IoT is the test and acceptable to use
[17]. The main components of IoT are sensors, application,
network, and data center.
Software testing is a process to identify the correctness,
competency, also the quality of developing computer software
it includes a set of activity convicted with the internal of
finding an error in software so that it could be corrected before
the product release to the client
This paper examines the trends in software testing
approaches that use to test different types of IoT environments
and uncover varicose issues that must address to enhance test
tool quality in this regard.
The rest of paper is organized as follows: Section II describes
the related work. Section III presents the software testing
types in IoT applications, while Section IV presents the
challenges of IoT testing. Section V describes the main tools
and technologies, Section VI presents a case study as an
example of testing in IoT. Finally, Section VII concludes the
paper.
II. RELATED WORK
In [1] a design and IoT solution testing using the
crowdsourcing solution is proposed, the crowd involved in
experiments with sensory data and knowledge through a
smartphone application. The challenges of crowdsourcing for
IoT include many technical challenges. For example, the
heterogeneity of IoT environment and when the sources are
dependents on crowds. There is a difficult to assure availability
and reliability until complete the experiments. So they
developed a crowdsourcing infrastructure that can be used to
experiment the internet of things (IoT) including social and
economic fields. Involving the crowd gave early feedbacks
from the crowd on the potential of the use case, the most
relevant use cases can be selected by the researchers based on
their ranking and feasibility.
In [2] the way that the internet of things process works as a
system when using commercially available components is
investigated by purchasing a component and created a set of
small use cases, for security testing they create use cases to
learn the way of exploiting the vulnerability of commercial
components. Their test results in 1) some of IoT
implementation not always classified user identity as sensitive
data so this can make tracking easy. 2) The cryptographic
libraries that used to encrypt data may have security defect that
exposes the data.
Philipp et al. [3] proposed a testing framework that applied
continuous integration techniques, this framework helps in
integration of project. In addition, they discussed challenges of
testing of open IoT software systems.
Gubbi et al. [4] presented a cloud user-centric vision for
worldwide implementation in the internet of things based on
the interaction of private and public clouds. They defined some
challenges such as privacy, security, and data management
issues that face internet of things environment. They mention

978-1-5386-4152-1/18/$31.00 ©2018 IEEE 17

2018 8th International Conference on Computer Science and Information Technology (CSIT)
some of the smart environment application with a testbed
example for each one, for example: to test smart retail field
there is an SAP future retail center testbed [5].
Karlesky et al. [6] a test based platform for embedded
software development consist of independent tools and new
design pattern Model Conductor Hardware (MCH) is presented
their platform perform automation tests which are logically
decoupled with hardware. Test-driven development reduces
software flaws, the proposed was applied on real-world
production systems estimating an adjusting project by using a
measurable metrics, decreasing debugging efforts and cost.
In Parasoft [9], the researchers implemented a tool called
Parasoft to capture code coverage, collect metrics and use them
with test results to make suitable decisions about ready
software and the impact of any change at the requirement. For
testing the healthcare system, the service testing solution
simulates the virtualized glucose sensor and send the obtained
value results. After that, the service testing solution will be able
to verify the fault alarm. Parasoft tool help in assuring the
quality of test cases, it may appear that additional testing is
needed after code execution due to some part of the code that
may not have been executed.in this research the system is
divided into layers for more easy and effective testing. In
addition, components were isolated to improve automation
test.
Leotta, Maurizio, et al. [10] proposed an approach for
acceptance testing of IoT systems is proposed, by using a
smartphone as a principal way of interaction between the user
and system. Acceptance test done based on black box testing to
test the functionality of a mobile health (m- health) IoT system
for diabetic patients their approach is not limited to them-health
system but also it can be applied to all IoT systems.
Assuring that the system satisfies a security design
requirements, Siboni et al. [11] proposed testbed for a security
testing to test wearable devices against external attackers. His
testbed can simulate the different environment in use the input
from simulated sensors, and it can give the output result as an
analysis report. Finally, their testbed is evaluated by a proof-of-
concept.
III. TESTING TYPES IN IOT
There are many kinds of testing can be used in IoT
applications can be summarized as follows [12]:
• Usability Testing: There are so many devices of
different shape and form factors are used by the
users. Moreover, the perception also varies from one
user to other. That's why checking usability of the
system is very important in IoT testing.
• Reliability and Scalability Testing: Reliability and
Scalability is important for building an IoT test
environment which involves simulation of sensors by
utilizing virtualization tools and technologies.
• Compatibility Testing: There are lots of devices
which can be connected through IoT system. These
devices have varied software and hardware
configuration. Therefore, the possible combination is
978-1-5386-4152-1/18/$31.00 ©2018 IEEE
ISBN: 978-1-5386-4152-1
huge. As a result, checking the compatibility in IoT
system is important.
• Security testing: In the IoT environment, there are
many users are accessing a massive amount of data.
Thus, it is important to validate user via
authentication, have data privacy controls as part of
security testing.
• Data Integrity Testing: It's important to check the
Data integrity in IoT testing as it involves a large
amount of data and its application.
• Performance Testing: Performance testing is
important to create a strategic approach for
developing and implementing an IoT testing plan.
IV. CHALLENGES OF IOT TESTING
The difficulty of testing the software of internet of things
come from the regular computer software take their input from
user through mouse and by typing with keyboard and some
touchable devices, but in internet of things many smart devices
are receiving their input from other smart devices like sensors
that used to pick up the user's body metrics and send the values
to software for analyzing these data. In addition, there is a need
to wear some devices such as some health devices to take the
input values from it, all these new changes in technology must
be taken in consideration while testing software of internet of
things [15].
A. Closely coupled among software and hardware
For all systems to start work there is a need for hardware and
software, testing IoT system includes more test because of
data transfer between components and the dependency
between them [7].
B. Automation
Automate some important role in testing computer
applications, needs to simulate some actions such as the
beating of a heart, and the rolling of a tire, which is more
difficult than testing the interaction with a web browser in a
regular computer software [15].
C. Security
Based on HP estimated study in 2014 there is 70% of IoT
device were vulnerable to attack, the HP report found that
transport encryption and insecure Web interfaces are the most
important issues in IoT security. A very large and
heterogeneous environment of the IoT causes the difficulty of
testing in everything in the IoT. Some practice in avoiding
data vulnerability in IoT is to make an end-to-end data
encryption between devices and to close the internet
communication when it is not in use [16].
Security capabilities are limited within IoT systems. While
encryption algorithms need higher processing power, IoT
system has devices with constrained memory and low CPU
cycles which may not support complex security algorithms.
Security services in the data center of IoT are critical in
18
2018 8th International Conference on Computer Science and Information Technology (CSIT)
ensuring that the IoT system is protected against threats [21].
The wireless medium of IoT can exploit by an attacker to make
a resource unavailable through Denial of Service (DoS). Other
recent attacks such as Stuxnet[19] and Duqu[20] can lead to
compromise within the whole system.
D. User interface
Many devices in the internet of things environment have a
user interface with a different platform, iOS, and Android for
example. The difficulty comes from that user interface may be
accessed from a device that has not been simulated [7].
E. Network availability
The best scenario for communication through the internet of
things is to communicate in fast speed all time without a
disconnect, in IoT, all kinds of the network need to be tested
with their speed, virtual network simulator is used for this
purpose [7]. Smart device tester needs to test their devices in
the real world and to test outside the lab where the internet
connectivity may be intermittent [15].
F. Communication protocol
Due to a large number of devices and the heterogeneous
environment in IoT, there is a need to use a lot of protocol to
interact with these devices such as Constrained Application
Protocol (CoAP), IoT tester needs to test all these different
communications protocols [16].
G. Data centers
Data Collection in IoT can't be done in a centralized manner
because there is a need to provide the scalability that required
by the Internet of things environment. In IoT a, several million
sensors are efficiently managed through Fog computing,
which is an expansion of the cloud computing but it is closer
to the source [21].
billions of devices in IoT will generate very large quantities
of data which will have increased the workloads on data
centers, due to the need of processing and responding in a real
time. If the data is not strongly secured, then a small breach
could lead to vulnerabilities exploiting [22].
V. IOT TESTING TOOL AND TECHNOLOGIES
The software tools used for IoT software testing can be
summarized as follows:
• Wireshark: it is an Open source application used for
displaying the TCP/IP and other packet transfer of a
network [7].
• Tcpdump: it is a command line based utility without
GUI, used for displaying source and destination host
978-1-5386-4152-1/18/$31.00 ©2018 IEEE
ISBN: 978-1-5386-4152-1
addresses, TCP/IP and other packets that are
transmitted or received over a network [7].
• VectorCAST: is a platform for automated software
testing. VectorCAST help tester to do the functional
and unit testing so they are tested according to the
system requirements, in a fast and continuous way.
Through VectorCAST tester can do automated
software test which is used for special exceptions and
hidden code [8].
• Shodan: it is an IoT testing tool, this tool can
investigate all devices that are connected to the
internet. The Shodan search engine can be used to find
IoT vulnerabilities, for example, exposed backdoors
unsecured webcams, and the internet devices that are
still using default passwords. Shodan can be a useful
tool for companies looking to control their own
environment, to provide these companies with
information about their product location and time of
being used [12][13].
• Thingful: is a search engine for the internet of things.
This tool can discover the open IoT data that's most
valuable to companies from around the world through
proprietary search algorithms. Thingful control how
this data is used, and it uses these data to take an
operational decision to improve the company's work
[12][14].
There are many technologies used on an internet of
thing such as Wi-Fi: is the most commonly used
technology for IoT. Bluetooth: used for short-range
communications, it is suitable for wearable devices.
Radio Frequency Code (RFID) and Electronic Product
Code (EPC). To enable two-way interactions between
the electronic devices Near Field Communication
(NFC) technology is used [7].
Table I: Some drawbacks of IoT testing tools.
Tool Drawback
It does not protect the network, it only
Wireshark captures packets coming in the network.in
Wireshark the captured packet information
is saved in a non-editable file, it is difficult
to add comments to the information [23].
Tcpdump It is text-based tool there is no user
interface, its biggest disadvantage is a lack
of analysis, and it can only analyze TCP
network traffic [24].
VectorCAST It is closed commercial product there is no
existence of adaptation layer software for
it, it has a limitation when the test of
aviation electronic software running on a
non-instrumented target system. [25].
Shodan There is the lack of collection of results
you may need for device analysis [26].
Thingful Limited only to IoT resources that are
subscribed to their centralized servers [27].
19
2018 8th International Conference on Computer Science and Information Technology (CSIT) ISBN: 978-1-5386-4152-1

VI. IOT TESTING: CASE STUDY
As an example of software testing in IoT's
environment. A company X provides their solutions for
any health service provider either with "on premises"
approach or cloud-based solution as Pay-Per-Patient
model in order to enable all providers to provide their
services fully digital, smart, and paperless with the
most optimized money investment. The architecture of
their solution is shown in Figure 1.
Now to test such an architecture, we need to go
through multiple testing approaches such as:
1. Usability Testing:
• The equipment should be smart enough to push
not only the notifications but also the error
messages, warnings etc.
• The system should have an option to log all the
events to provide clarity to the end users. If it is
not capable of doing that, the system should push
those as well to a database to store it.
• Usability in terms of displaying data, processing
data, pushing job tasks from the devices should be
tested thoroughly.
• Patient has to be identified at the device side and
data will be pushed along with the patient
identification so that the system will make sure to
store the data for that specific patient to avoid any
mixing of medical data between different patients.
• Different data formats extracted from different
devices must be easily handled by the system and
correctly stored in the patient profile in the
database.
2. Security Testing:
• IoT Security challenges: IoT is data-centric where
all the devices/system connected operate based on
the data that is available.
• Role-Based Authentication is important to give the
access only to the right user(s) prevent the access
of other users who have no rights to view patient
information.
• Sharing of patient data with other users must be
authenticated by the patient approval only.
which can prompt the physicians so that they can
start to monitor the health conditions manually not
depending on the system till it is up. On the other
hand, there has to be a mechanism in the system
which can store all the data in it during the offline
period. Once the system comes online, all that data
should get propagated. Data loss should not be
there in any condition. Also, an order of the data
streamed to the system is important and
information must always be flagged with the
capturing date/time.
• The system has to handle two kinds of dis-
connectivity: System failure: Needs to be handled
via proper failover scenario for system servers and
storage. Internet slowness or disconnection:
offline workflow as mentioned above.
4. Regulatory Testing:
• The solution must be in compliance with
international standards of sharing and sending
healthcare data such as IHE and HIPAA standards.
Using HL7 or FHIR to send the data from patient
side to the devices is important and then from the
devices to the central system.
5. Others:
• Patient identification for safe storing the data on
the specific patient profile.
• Time and date recording for several captures from
the device for a specific patient.
• Trending graphs to compare the continuous
timeline representation of the captured data for
certain parameters.
• Defining the parameters that the system is able to
capture and continuous monitoring such as Vital
signs, Pressure, Cardiac parameters (heart rate,
ECG, …), and Triage level.
• Add continuous monitoring for certain parameters
for each patient and the required device to capture
this parameter.

3. Connectivity Testing:
• Connectivity, transfer of data, receiving job tasks
from the devices should be seamless when the
connection is UP and running.
• The other condition is the connection down
scenario. Doesn't matter how robust is the system
and the network, there are chances that the system
will go offline. Being a tester, we should test the
offline conditions as well. Once the system is not
available on the network, there has to be an alert

978-1-5386-4152-1/18/$31.00 ©2018 IEEE 20

2018 8th International Conference on Computer Science and Information Technology (CSIT) ISBN: 978-1-5386-4152-1

[5] S.R.L. Labs, Future Retail Centeer, SAP Research Living Labs.
http://www.sap.com/corporate-en/our
company/innovation/research/livinglabs/futureretail/index.epx (n.d.).
[6] Karlesky, M., Williams, G., Bereza, W., and Fletcher, M. Mocking the
embedded world: Test-driven development, continuous integration, and
design patterns. In Proc. Emb. Systems Conf, CA, USA (2007).
[7] Internet of Things (IoT) Testing: Challenges, Tools and Testing
Approach Posted In | Testing Concepts, Types of Testing | Last Updated:
"February 22, 2018"
[8] Vehicle Electronics The monthly magazine for automotive electronics
engineers, Issue 52, April 2018.
[9] Parasoft. End-to-end testing for IoT integrity. Technical report.
https://alm.parasoft.com/endto-end-testing-for-IoT-integrity.2017
[10] Leotta, Maurizio, et al. "Towards an Acceptance Testing Approach for
the Internet of Things Systems." Proceedings of 1st International
Workshop on Engineering the Web of Things. 2017.
[11] S. Siboni, A. Shabtai, N. O. Tippenhauer, J. Lee, and Y. Elovici.
Advanced security testbed framework for wearable IoT devices. ACM
Transactions on Internet Technology (TOIT), 16(4):26, 2016.
[12] Internet of Things (IoT) Testing: What is, Challenges & Tools available
at https://www.guru99.com/IoT-testing-challenges-tools.html
[13] Shodan exposes IoT vulnerabilities: By Maria KorolovContributing
Writer, CSO | Jan 13, 2015, available at
https://www.csoonline.com/article/2867407/network-security/shodan-
exposes-IoT-vulnerabilities.html
[14] A Search Engine for the Internet of Things Find & use open IoT data
from around the world: available at
https://www.thingful.net/?overlay=true&lat=20.05593126519445&lng=
47.8125&z=3
[15] Bob Crews and Shailesh Mangal, "Internet of things and its impact on
Figure 1: The architecture of company X’s solution testing": whitepaper available at
https://www.getzephyr.com/resources/whitepapers/IoT-and-its-impact-
testing
VII. CONCLUSION
[16] The Future Of Iot Testing, By Sanjay Zalavadia • February 28, 2017
The paper surveys different types of software testing can be [17] Höller, V. Tsiatsis, C. Mulligan, S. Karnouskos, S. Avesand, D. Boyle,
used in IoT applications such as usability, security, and From Machine-to-Machine to the Internet of Things: Introduction to a
connectivity. The paper also highlights the diversity of the New Age of Intelligence, Amsterdam, The Netherlands:Elsevier, 2014
challenges that could be encountered during the testing of IoT [18] G. Kortuem, F. Kawsar, D. Fitton, V. Sundramoorthy, "Smart objects as
building blocks for the Internet of Things", IEEE Internet Comput., vol.
devices such as user interface, network availability, and 14, pp. 44-51, Jan./Feb. 2010.
automation. Moreover, the paper offers a broad view of tools [19] Computerworld, "Siemens: Stuxnet worm hit industrial systems",
and platforms that can be used to test IoT devices such as September 16, 2010.
Wireshark, Shodan, and Vector platform. [20] "Duqu: A Stuxnet-like malware found in the wild, technical report",
In the end, a healthcare industrial example from company X October 14, 2011, Laboratory of Cryptography of Systems Security
was chosen as a case study to show how testing for IoT [21] Securing the Internet of Things: A Proposed Framework, Cisco Security
environment works. The preliminary results show the Research & Operations. Available at:
https://www.cisco.com/c/en/us/about/security-center/secure-IoT-
importance of this hot research topic to be discussed proposed-framework.html
furthermore. [22] Why the Internet of Things will disrupt data centers, Available at :
https://www1.raritan.com/HOTAISLE-DASHBOARD-12-2016.html
[23] S.Pavithirakini,D.D.M.M.Bandara, C.N.Gunawardhana, K.K.S.Perera,
References B.G.M.M.Abeyrathne, Dhishan Dhammearatchi , “Improve the
Capabilities of Wireshark as a tool for Intrusion Detection in DOS
Attacks”, International Journal of Scientific and Research Publications,
[1] Fernandes, João, et al. "IoT Lab: Towards co-design and IoT solution Volume 6, Issue 4, April 2016.
testing using the crowd." Recent Advances in Internet of Things (RIoT), [24] Awodele, Oludele, et al. "Packet Sniffer–A comparative characteristic
2015 International Conference on. IEEE, 2015. evaluation study." Proceedings of Informing Science & IT Education
[2] Kolias, Constantinos, et al. "Learning Internet-of-Things Security" Conference (InSITE). 2015.
Hands-On"." IEEE Security & Privacy 14.1 (2016): 37-46. [25] Zamojski, Wojciech, et al., eds. Complex Systems and Dependability.
[3] Rosenkranz, Philipp, et al. "A distributed test system architecture for Vol. 170. Springer, 2012.
open-source IoT software." Proceedings of the 2015 Workshop on IoT [26] Eric Gross, CRITICAL INFRASTRUCTURE SECURITY Locating and
challenges in Mobile and Industrial Systems. ACM, 2015. Securing SCADA Devices on the Internet of Things, 2015.
[4] Gubbi, Jayavardhana, et al. "Internet of Things (IoT): A vision, [27] Pattar, Santosh, et al. "Searching for the IoT Resources: Fundamentals,
architectural elements, and future directions." Future generation Requirements, Comprehensive Review and Future Directions." IEEE
computer systems 29.7 (2013): 1645-1660. Communications Surveys & Tutorials (2018).

978-1-5386-4152-1/18/$31.00 ©2018 IEEE 21