IoT Testing and Quality

Assurance
Girish Kolapkar
SAS Research and Development (India)
Agenda
• Introduction to Internet of Things (IoT)
• Industries approaching the opportunity of IoT
• Emerging IoT Applications
• Typical IoT Technology Stack
• Challenges testing software’s supporting IoT
• Software QA’s Role Redefined
• IoT Testing Areas
• IoT Testing Types
• Strategy for effective IoT software testing
• Q&A
Introduction
• The Internet of Things (IoT) is the network of physical
objects or “things” embedded with electronics, software,
sensors, and network connectivity, which enables these
objects to collect and exchange data.
• Software industry is blending the physical and logical cyber
worlds with the Internet of Things.
• IoT isn’t merely a buzzword. It’s where we’re headed.
• Offers potential for businesses worldwide to unlock value
and benefits from devices, systems and services
communicating with each other, using the internet as a
backbone.
Industries approaching the opportunity
of IoT

Smart Cities and Homes

Connected Customer

Communications

Connected Car/
Transportation
Internet
Of Things
Energy

Manufacturing
Healthcare Retail
Emerging IoT Applications

• Information & Analysis

• Tracking Behaviour
• Examples- SCM, Trucks with sensors for predictive
maintenance
• Enhanced Situational Awareness
• Examples- Weather Warning Systems
• Sensor-driven Decision Analytics
• Examples- Oil and gas company optimizing oilfield
production
Emerging IoT Applications (Contd.)

• Automation & Control

• Process Optimization
• Examples-Manufacturing Assembly Line
• Optimized Resource Consumption
• Examples-Energy Distribution Networks
• Complex Autonomous Systems
• Examples- Collision Avoidance Systems
Typical IoT Technology Stack
Application

Websites\Mobile Apps\Industry Specific

appliances & consoles,
Alerts\Reports\ Decision Dashboards

Application Platform
API Design\Build

Event Management, Intelligent Filter\ Transform,

Steaming Model Execution

Infrastructure
Network, Compute ,Data Storage

Device Hub\ Gateway, Device Management

Data Streams at the Edge

Short Haul\Long haul Communication
Controller, Agent,
Thing\Device\Sensor\Actuator
Challenges testing software’s supporting
IoT
Big Data-Edges generates astronomical volume of data, speed & variety
Challenges testing software’s supporting
IoT (Contd.)
• Complex mix of modern IT & Web services with embedded
systems
• Security
• Expensive test environment setup
• Too Many- networks, protocols, gateways, device hubs
Challenges testing software’s supporting
IoT (Contd.)
• Multi-user Applications & distributed environment
deployment
• Applications require decision support in real-time, batch,
hybrid modes
• Debugging requires understanding of multiple technologies
• Difficult to establish point of failures
• Limited User Interface\console
Software QA’s Role Redefined 

• More focus on user centric testing

• Just not find bugs,prevent them
• Quality advocates, influencing both development and
operational processes
• Software testers who do not have experience testing
hardware or embedded systems may need to develop
some new skills
• Testers with domain knowledge
IoT Testing Areas

• Connectivity
• Security
• Performance
• Functionality
• Compatibility
• Localization
• User Experience
IoT Testing Areas-Connectivity

• Seamless connectivity
• Data restore when no connectivity
• Sample Test cases
• Verify that all the gadgets involved in the Internet of
things testing are able to register to the network.
• Verify that application ensure data security and storage
so that whenever the connection is restored, data is
back to shape as it was.
IoT Testing Areas-Security

• More possibilities and avenues for cyber attacks

• More threats with financial data. E.g. RFID based highway
toll collection systems
• IoT ecosystem provides multiple entry point for attacks
• Lost or stolen devices
• Sample Test cases
• Verify no unauthorized access to device or information.
• Verify data on compromised IoT devices can be remote
wiped out.
IoT Testing Areas-Performance

• Device- to-Device communication

• Device to Server communication
• Server To Server communication
• Network bandwidth, latency, packet loss
• Sample Test cases
• Verify response time against benchmarked time with
defined connectivity conditions.
IoT Testing Areas-Functionality

• Web\Mobile applications
• Functional\business requirements
• Access control
• Data Storage
• Identity\Roles management
• Sample Test cases
• Verify medical monitoring system returning correct
outputs.
IoT Testing Areas-Compatibility
• Device-to-device communication protocols are not
compatible, forcing the user to select a family of devices
that employ a common protocol
• Lack of standardization leads to incompatibility issues
• Compatibility needs to be validated for combinations of
devices, protocols, operating systems
• Sample Test cases
• Verify IoT software supports defined set of decives.
IoT Testing Areas-User Experience
• Refers to a person's total experience using a particular
product, system or service
• Familiar
• Clear & Concise
• Feedback
• Consistent
• Attractive
• Efficient
• Forgiving
IoT Testing Types
 •
Compatibility Testing
• Functional Testing
• Performance Testing
• Security Testing
• Exploratory Testing
• Load Testing
Strategy for effective IoT software
testing
• Automation rules-Develop automation systems, through
code, that can ensure quality standards at each level and
which assures -
• Response Time
• Data Validation
• Real Time Data Accuracy
• Secure Access control
• Early detection of issues at design time
• Needs more detailed Acceptance Criteria
Strategy for effective IoT software testing (Contd.)

• Robust Backend – NoSQL ?

• Minimal manual end to end testing
• Virtualization
• Simulators for various industries
• Remote Testing
• Mock up the API
Strategy for effective IoT software testing (Contd.)

• Elevate a test approach rather than simply executing it

• Analyse Test Results and Test effectiveness
• Measure quality of Test
• Timely review testing tool set as per user feedback,
remote debugging and quick turnaround
Strategy for effective IoT software testing (Contd.)

• Generate Realistic Test data

• Data Generator Tool
• Test Analytics
• Understand signals in data to derive new patterns
based test cases
Strategy for effective IoT software testing (Contd.)

• Effective Test Data Management Tool

• Data on Demand
• Data Integration
• Data Security
• Data validation
Example- Test Event Stream Processing Engine (Contd.)

• Test suite of source adapters

• Predefined adapters e.g. Hadoop (+YARN),
RabbitMQ
• Test for publish(read) only- e.g. Sniffer,Twitter,
SYSLOG, HTTP RESTful Interfaces
• Test for subscribe(write) only- e.g. SOAP,SMTP
• Customizable publish\subscribe API’s
Example- Test Event Stream Processing Engine (Contd.)

• Test visual interface which define\refine

event stream models
Example- Test Event Stream Processing Engine (Contd.)

• Deploy & Test On Workstation

• E.g. Test for iPhone sensor logs

• Deploy & Test on Edge Gateway

• E.g. Test for physical raspberry-pi
Example- Test Event Stream Processing Engine (Contd.)

• Connect iPhone and Laptop to wireless network

Example- Test Event Stream Processing Engine (Contd.)

• Step 1: Install and check SensorLog app on iPhone, get ip

address.
• Step2: Start ESP server and put ESP model in test mode
• Step3:Run SensorLog and start streaming data through
socket
• Step4: Run adapter script to read iPhone sensor data
• Step5: Open visualization to see streaming data
• Choose line graph to show sensor data and moving
average
• Shake your iPhone to see graph changes
QUESTIONS?
Thank You!

girish.kolapkar@gmail.com
https://www.linkedin.com/in/kolapkar
https://www.facebook.com/girish.kolapkar