Bahga Madisetti IoT Book Slides Justin Joseph

Chapter 1
Introduction to IoT
Book website: http://www.internet-of-things-book.com Bahga & Madisetti, © 2015

Outline
• IoT definition
• Characteristics of IoT
• Physical Design of IoT
• Logical Design of IoT
• IoT Protocols
• IoT Levels & Deployment Templates

Definition of IoT
A dynamic global network infrastructure with self-configuring

capabilities based on standard and interoperable communication
protocols where physical and virtual "things" have identities, physical
attributes, and virtual personalities and use intelligent interfaces, and
are seamlessly integrated into the information network, often
communicate data associated with users and their environments.

Characteristics of IoT
• Dynamic & Self-Adapting

• Self-Configuring
• Interoperable Communication Protocols
• Unique Identity
• Integrated into Information Network

Physical Design of IoT
• The "Things" in IoT usually refers to IoT devices which have unique
identities and can perform remote sensing, actuating and monitoring
capabilities.
• IoT devices can:
• Exchange data with other connected devices and applications (directly or
indirectly), or
• Collect data from other devices and process the data locally or
• Send the data to centralized servers or cloud-based application back-ends for
processing the data, or
• Perform some tasks locally and other tasks within the IoT infrastructure,
based on temporal and space constraints

Generic block diagram of an IoT Device
• An IoT device may consist of

several interfaces for
connections to other devices,
both wired and wireless.
• I/O interfaces for sensors
• Interfaces for Internet
connectivity
• Memory and storage interfaces
• Audio/video interfaces.

IoT Protocols
• Link Layer
• 802.3 – Ethernet
• 802.11 – WiFi
• 802.16 – WiMax
• 802.15.4 – LR-WPAN
• 2G/3G/4G
• Network/Internet Layer
• IPv4
• IPv6
• 6LoWPAN
• Transport Layer
• TCP
• UDP
• Application Layer
• HTTP
• CoAP
• WebSocket
• MQTT
• XMPP
• DDS
• AMQP
Logical Design of IoT
• Logical design of an IoT system

refers to an abstract
representation of the entities
and processes without going
into the low-level specifics of
the implementation.
• An IoT system comprises of a

number of functional blocks
that provide the system the
capabilities for identification,
sensing, actuation,
communication, and
management.

Request-Response communication model
• Request-Response is a
communication model in which
the client sends requests to the
server and the server responds
to the requests.
• When the server receives a

request, it decides how to
respond, fetches the data,
retrieves resource
representations, prepares the
response, and then sends the
response to the client.

Publish-Subscribe communication model
• Publish-Subscribe is a
communication model that
involves publishers, brokers and
consumers.
• Publishers are the source of data.
Publishers send the data to the
topics which are managed by the
broker. Publishers are not aware
of the consumers.
• Consumers subscribe to the topics
which are managed by the broker.
• When the broker receives data for
a topic from the publisher, it
sends the data to all the
subscribed consumers.

Push-Pull communication model
• Push-Pull is a communication
model in which the data
producers push the data to
queues and the consumers pull
the data from the queues.
Producers do not need to be
aware of the consumers.
• Queues help in decoupling the
messaging between the producers
and consumers.
• Queues also act as a buffer which
helps in situations when there is a
mismatch between the rate at
which the producers push data
and the rate rate at which the
consumers pull data.

Exclusive Pair communication model
• Exclusive Pair is a
bidirectional, fully duplex
uses a persistent connection
between the client and
server.
• Once the connection is setup
it remains open until the
client sends a request to
close the connection.
• Client and server can send
messages to each other after
connection setup.

REST-based Communication APIs
• Representational State Transfer

(REST) is a set of architectural
principles by which you can design
web services and web APIs that
focus on a system’s resources and
how resource states are
addressed and transferred.
• REST APIs follow the request-
response communication model.
• The REST architectural constraints
apply to the components,
connectors, and data elements,
within a distributed hypermedia
system.

WebSocket-based Communication APIs
• WebSocket APIs allow bi-

directional, full duplex
communication between
clients and servers.
• WebSocket APIs follow the
exclusive pair
communication model

Exclusive Pair communication model
• Exclusive Pair is a
bidirectional, fully duplex
uses a persistent connection
between the client and
server.
• Once the connection is setup
it remains open until the
client sends a request to
close the connection.
• Client and server can send
messages to each other after
connection setup.

IoT Levels & Deployment Templates
An IoT system comprises of the following components:

• Device: An IoT device allows identification, remote sensing, actuating and
remote monitoring capabilities. You learned about various examples of IoT
devices in section
• Resource: Resources are software components on the IoT device for
accessing, processing, and storing sensor information, or controlling
actuators connected to the device. Resources also include the software
components that enable network access for the device.
• Controller Service: Controller service is a native service that runs on the
device and interacts with the web services. Controller service sends data
from the device to the web service and receives commands from the
application (via web services) for controlling the device.

IoT Levels & Deployment Templates
• Database: Database can be either local or in the cloud and stores the data
generated by the IoT device.
• Web Service: Web services serve as a link between the IoT device,
application, database and analysis components. Web service can be either
implemented using HTTP and REST principles (REST service) or using
WebSocket protocol (WebSocket service).
• Analysis Component: The Analysis Component is responsible for analyzing
the IoT data and generate results in a form which are easy for the user to
understand.
• Application: IoT applications provide an interface that the users can use to
control and monitor various aspects of the IoT system. Applications also
allow users to view the system status and view the processed data.

IoT Level-1
• A level-1 IoT system has a

single node/device that
performs sensing and/or
actuation, stores data,
performs analysis and hosts
the application
• Level-1 IoT systems are
suitable for modeling low-
cost and low-complexity
solutions where the data
involved is not big and the
analysis requirements are
not computationally
intensive.

IoT Level-2

single node that performs
sensing and/or actuation and
local analysis.
• Data is stored in the cloud and
application is usually cloud-
based.
suitable for solutions where
the data involved is big,
however, the primary analysis
requirement is not
computationally intensive and
can be done locally itself.

IoT Level-3

single node. Data is stored
and analyzed in the cloud
and application is cloud-
based.
suitable for solutions
where the data involved is
big and the analysis
requirements are
computationally intensive.

IoT Level-4
• A level-4 IoT system has multiple

nodes that perform local analysis.
Data is stored in the cloud and
application is cloud-based.
• Level-4 contains local and cloud-
based observer nodes which can
subscribe to and receive
information collected in the cloud
from IoT devices.
• Level-4 IoT systems are suitable
for solutions where multiple
nodes are required, the data
involved is big and the analysis
requirements are computationally
intensive.

IoT Level-5
• A level-5 IoT system has multiple end

nodes and one coordinator node.
• The end nodes that perform sensing
and/or actuation.
• Coordinator node collects data from
the end nodes and sends to the cloud.
• Data is stored and analyzed in the
cloud and application is cloud-based.
• Level-5 IoT systems are suitable for
solutions based on wireless sensor
networks, in which the data involved
is big and the analysis requirements
are computationally intensive.

IoT Level-6
• A level-6 IoT system has multiple

independent end nodes that
perform sensing and/or actuation
and send data to the cloud.
• Data is stored in the cloud and
application is cloud-based.
• The analytics component analyzes
the data and stores the results in
the cloud database.
• The results are visualized with the
cloud-based application.
• The centralized controller is aware
of the status of all the end nodes
and sends control commands to
the nodes.

Chapter 3
IoT & M2M

Outline
• M2M
• Differences and Similarities between M2M and IoT
• SDN and NFV for IoT

Machine-to-Machine (M2M)
• Machine-to-Machine (M2M) refers to networking of machines (or

devices) for the purpose of remote monitoring and control and data
exchange.

Machine-to-Machine (M2M)
• An M2M area network comprises of machines (or M2M nodes) which have
embedded hardware modules for sensing, actuation and communication.
• Various communication protocols can be used for M2M local area
networks such as ZigBee, Bluetooh, ModBus, M-Bus, Wirless M-Bus, Power
Line Communication (PLC), 6LoWPAN, IEEE 802.15.4, etc.
• The communication network provides connectivity to remote M2M area
networks.
• The communication network can use either wired or wireless networks (IP-
based).
• While the M2M area networks use either proprietary or non-IP based
communication protocols, the communication network uses IP-based
networks.

M2M gateway
• Since non-IP based protocols are used within M2M area networks, the
M2M nodes within one network cannot communicate with nodes in an
external network.
• To enable the communication between remote M2M area networks,
M2M gateways are used.

Difference between IoT and M2M
• Communication Protocols
• M2M and IoT can differ in how the communication between the machines or
devices happens.
• M2M uses either proprietary or non-IP based communication protocols for
communication within the M2M area networks.
• Machines in M2M vs Things in IoT
• The "Things" in IoT refers to physical objects that have unique identifiers and
can sense and communicate with their external environment (and user
applications) or their internal physical states.
• M2M systems, in contrast to IoT, typically have homogeneous machine types
within an M2M area network.

Difference between IoT and M2M
• Hardware vs Software Emphasis

• While the emphasis of M2M is more on hardware with embedded modules, the
emphasis of IoT is more on software.
• Data Collection & Analysis
• M2M data is collected in point solutions and often in on-premises storage
infrastructure.
• In contrast to M2M, the data in IoT is collected in the cloud (can be public, private or
hybrid cloud).
• Applications
• M2M data is collected in point solutions and can be accessed by on-premises
applications such as diagnosis applications, service management applications, and on-
premisis enterprise applications.
• IoT data is collected in the cloud and can be accessed by cloud applications such as
analytics applications, enterprise applications, remote diagnosis and management
applications, etc.

Communication in IoT vs M2M

SDN
• Software-Defined Networking
(SDN) is a networking
architecture that separates the
control plane from the data
plane and centralizes the
network controller.
• Software-based SDN controllers
maintain a unified view of the
network and make confi
guration, management and
provisioning simpler.
• The underlying infrastructure in
SDN uses simple packet
forwarding hardware as
opposed to specialized
hardware in conventional
networks.

Key elements of SDN
• Centralized Network Controller

• With decoupled control and data planes and centralized network controller,
the network administrators can rapidly configure the network.
• Programmable Open APIs
• SDN architecture supports programmable open APIs for interface between
the SDN application and control layers (Northbound interface).
• Standard Communication Interface (OpenFlow)
• SDN architecture uses a standard communication interface between the
control and infrastructure layers (Southbound interface).
• OpenFlow, which is defined by the Open Networking Foundation (ONF) is
the broadly accepted SDN protocol for the Southbound interface.

NFV
• Network Function
Virtualization (NFV) is a
technology that leverages
virtualization to
consolidate the
heterogeneous network
devices onto industry
standard high volume
servers, switches and
storage.
• NFV is complementary to
SDN as NFV can provide
the infrastructure on
which SDN can run.

Key elements of NFV
• Virtualized Network Function (VNF):

• VNF is a software implementation of a network function which is capable of
running over the NFV Infrastructure (NFVI).
• NFV Infrastructure (NFVI):
• NFVI includes compute, network and storage resources that are virtualized.
• NFV Management and Orchestration:
• NFV Management and Orchestration focuses on all virtualization-specific
management tasks and covers the orchestration and life-cycle
management of physical and/or software resources that support the
infrastructure virtualization, and the life-cycle management of VNFs.

NFV Use Case
• NFV can be used to virtualize the Home Gateway. The NFV infrastructure in the cloud hosts a
virtualized Home Gateway. The virtualized gateway provides private IP addresses to the
devices in the home. The virtualized gateway also connects to network services such as VoIP
and IPTV.

Chapter 4
IoT System Management with

NETCONF-YANG

Outline
• Need for IoT Systems Management

• SNMP
• Network Operator Requirements
• NETCONF
• YANG
• IoT Systems Management with NETCONF-YANG

Need for IoT Systems Management
• Automating Configuration
• Monitoring Operational & Statistical Data
• Improved Reliability
• System Wide Configurations
• Multiple System Configurations
• Retrieving & Reusing Configurations

Simple Network Management Protocol (SNMP)
• SNMP is a well-known and widely used

network management protocol that allows
monitoring and configuring network devices
such as routers, switches, servers, printers, etc.
• SNMP component include
• Network Management Station (NMS)
• Managed Device
• Management Information Base (MIB)
• SNMP Agent that runs on the device

Limitations of SNMP
• SNMP is stateless in nature and each SNMP request contains all the
information to process the request. The application needs to be intelligent
to manage the device.
• SNMP is a connectionless protocol which uses UDP as the transport protocol,
making it unreliable as there was no support for acknowledgement of
requests.
• MIBs often lack writable objects without which device configuration is not
possible using SNMP.
• It is difficult to differentiate between configuration and state data in MIBs.
• Retrieving the current configuration from a device can be difficult with
SNMP.
• Earlier versions of SNMP did not have strong security features.

Network Operator Requirements
• Ease of use • Configuration validation

• Distinction between configuration and state data • Configuration database schemas
• Fetch configuration and state data separately • Comparing configurations
• Configuration of the network as a whole • Role-based access control
• Configuration transactions across devices • Consistency of access control lists:
• Configuration deltas • Multiple configuration sets
• Dump and restore configurations • Support for both data-oriented and task-
oriented access control

NETCONF
• Network Configuration Protocol (NETCONF) is a session-based network management protocol.

NETCONF allows retrieving state or configuration data and manipulating configuration data on
network devices

NETCONF
• NETCONF works on SSH transport protocol.

• Transport layer provides end-to-end connectivity and ensure reliable delivery of messages.
• NETCONF uses XML-encoded Remote Procedure Calls (RPCs) for framing request and
response messages.
• The RPC layer provides mechanism for encoding of RPC calls and notifications.
• NETCONF provides various operations to retrieve and edit configuration data from
network devices.
• The Content Layer consists of configuration and state data which is XML-encoded.
• The schema of the configuration and state data is defined in a data modeling language
called YANG.
• NETCONF provides a clear separation of the configuration and state data.
• The configuration data resides within a NETCONF configuration datastore on the server.

YANG
• YANG is a data modeling language used to model configuration and state data
manipulated by the NETCONF protocol
• YANG modules contain the definitions of the configuration data, state data, RPC calls that
can be issued and the format of the notifications.
• YANG modules defines the data exchanged between the NETCONF client and server.
• A module comprises of a number of 'leaf' nodes which are organized into a hierarchical
tree structure.
• The 'leaf' nodes are specified using the 'leaf' or 'leaf-list' constructs.
• Leaf nodes are organized using 'container' or 'list' constructs.
• A YANG module can import definitions from other modules.
• Constraints can be defined on the data nodes, e.g. allowed values.
• YANG can model both configuration data and state data using the 'config' statement.

YANG Module Example
• This YANG module is a YANG version of the toaster

MIB
• The toaster YANG module begins with the header
information followed by identity declarations
which define various bread types.
• The leaf nodes (‘toasterManufacturer’,
‘toasterModelNumber’ and oasterStatus’) are
defined in the ‘toaster’ container.
• Each leaf node definition has a type and optionally
a description and default value.
• The module has two RPC definitions (‘make-toast’
and ‘cancel-toast’).

IoT Systems Management with NETCONF-YANG
• Management System
• Management API
• Transaction Manager
• Rollback Manager
• Data Model Manager
• Configuration Validator
• Configuration Database
• Configuration API
• Data Provider API

Chapter 5
IoT Design Methodology

Outline
• IoT Design Methodology that includes:

• Purpose & Requirements Specification
• Process Specification
• Domain Model Specification
• Information Model Specification
• Service Specifications
• IoT Level Specification
• Functional View Specification
• Operational View Specification
• Device & Component Integration
• Application Development

IoT Design Methodology - Steps
Step 1: Purpose & Requirements Specification
• The first step in IoT system design methodology is to define the
purpose and requirements of the system. In this step, the system
purpose, behavior and requirements (such as data collection
requirements, data analysis requirements, system management
requirements, data privacy and security requirements, user interface
requirements, ...) are captured.
Step 2: Process Specification
• The second step in the IoT design methodology is to define the

process specification. In this step, the use cases of the IoT system are
formally described based on and derived from the purpose and
requirement specifications.
Step 3: Domain Model Specification
• The third step in the IoT design methodology is to define the Domain
Model. The domain model describes the main concepts, entities and
objects in the domain of IoT system to be designed. Domain model
defines the attributes of the objects and relationships between
objects. Domain model provides an abstract representation of the
concepts, objects and entities in the IoT domain, independent of any
specific technology or platform. With the domain model, the IoT
system designers can get an understanding of the IoT domain for
which the system is to be designed.
Step 4: Information Model Specification
• The fourth step in the IoT design methodology is to define the

Information Model. Information Model defines the structure of all
the information in the IoT system, for example, attributes of Virtual
Entities, relations, etc. Information model does not describe the
specifics of how the information is represented or stored. To define
the information model, we first list the Virtual Entities defined in the
Domain Model. Information model adds more details to the Virtual
Entities by defining their attributes and relations.
Step 5: Service Specifications
• The fifth step in the IoT design methodology is to define the service
specifications. Service specifications define the services in the IoT
system, service types, service inputs/output, service endpoints,
service schedules, service preconditions and service effects.
Step 6: IoT Level Specification
• The sixth step in the IoT design methodology is to define the IoT level
for the system. In Chapter-1, we defined five IoT deployment levels.
Step 7: Functional View Specification
• The seventh step in the IoT design methodology is to define the
Functional View. The Functional View (FV) defines the functions of
the IoT systems grouped into various Functional Groups (FGs). Each
Functional Group either provides functionalities for interacting with
instances of concepts defined in the Domain Model or provides
information related to these concepts.
Step 8: Operational View Specification
• The eighth step in the IoT design methodology is to define the
Operational View Specifications. In this step, various options
pertaining to the IoT system deployment and operation are defined,
such as, service hosting options, storage options, device options,
application hosting options, etc
Step 9: Device & Component Integration
• The ninth step in the IoT design methodology is the integration of the
devices and components.
Step 10: Application Development
• The final step in the IoT design methodology is to develop the IoT
application.
Home Automation Case Study
Step:1 - Purpose & Requirements
• Applying this to our example of a smart home automation system, the
purpose and requirements for the system may be described as follows:
• Purpose : A home automation system that allows controlling of the lights in a home
remotely using a web application.
• Behavior : The home automation system should have auto and manual modes. In
auto mode, the system measures the light level in the room and switches on the
light when it gets dark. In manual mode, the system provides the option of manually
and remotely switching on/off the light.
• System Management Requirement : The system should provide remote monitoring
and control functions.
• Data Analysis Requirement : The system should perform local analysis of the data.
• Application Deployment Requirement : The application should be deployed locally
on the device, but should be accessible remotely.
• Security Requirement : The system should have basic user authentication capability.
Step:2 - Process Specification
Step 3: Domain Model Specification
Step 4: Information Model Specification
Step 6: IoT Level Specification
Step 7: Functional View Specification
Step 8: Operational View Specification
Step 9: Device & Component Integration
Step 10: Application Development
• Auto
• Controls the light appliance automatically based on the lighting
conditions in the room
• Light
• When Auto mode is off, it is used for manually controlling the
light appliance.
• When Auto mode is on, it reflects the current state of the light
appliance.
Implementation: RESTful Web Services
REST services implemented with Django REST Framework
# Models – models.py
from django.db import models
class Mode(models.Model):
name = models.CharField(max_length=50)
1. Map services to models. Model class State(models.Model):
fields store the states (on/off, name = models.CharField(max_length=50)
auto/manual)
2. Write Model serializers. Serializers allow

complex data (such as model instances) to be
converted to native Python datatypes that can
then be easily rendered into JSON, XML or
other content types.
# Serializers – serializers.py
from myapp.models import Mode, State
from rest_framework import serializers
class ModeSerializer(serializers.HyperlinkedModelSerializer):
class Meta:
model = Mode
fields = ('url', 'name')
class StateSerializer(serializers.HyperlinkedModelSerializer):
class Meta:
model = State
fields = ('url', 'name')
# Views – views.py
# Models – models.py from myapp.models import Mode, State
from django.db import models
3. Write ViewSets for the Models which from rest_framework import viewsets
combine the logic for a set of related views in from myapp.serializers import ModeSerializer, StateSerializer
class Mode(models.Model): a single class.
name = models.CharField(max_length=50) class ModeViewSet(viewsets.ModelViewSet):
queryset = Mode.objects.all()
class State(models.Model): serializer_class = ModeSerializer
name = models.CharField(max_length=50)
class StateViewSet(viewsets.ModelViewSet):
queryset = State.objects.all()
serializer_class = StateSerializer
# URL Patterns – urls.py

from django.conf.urls import patterns, include, url
from django.contrib import admin
from rest_framework import routers
4. Write URL patterns for the services.
from myapp import views
admin.autodiscover() Since ViewSets are used instead of views, we
router = routers.DefaultRouter() can automatically generate the URL conf by
router.register(r'mode', views.ModeViewSet) simply registering the viewsets with a router
router.register(r'state', views.StateViewSet) class.
urlpatterns = patterns('',
Routers automatically determining how the
url(r'^', include(router.urls)),
url(r'âpi-auth/', include('rest_framework.urls', namespace='rest_framework')), URLs for an application should be mapped to
url(r'âdmin/', include(admin.site.urls)), the logic that deals with handling incoming
url(r'^home/', 'myapp.views.home'), requests.
)
Screenshot of browsable
State REST API
Screenshot of browsable
Mode REST API
Implementation: Controller Native Service
#Controller service def runAutoMode():
import RPi.GPIO as GPIO ldr_reading = readldr(LDR_PIN)
if ldr_reading < threshold:
import time switchOnLight(LIGHT_PIN)
Native service deployed locally import sqlite3 as lite setCurrentState('on')
import sys else:
switchOffLight(LIGHT_PIN)
setCurrentState('off')
con = lite.connect('database.sqlite')
cur = con.cursor() def runManualMode():
state = getCurrentState()
GPIO.setmode(GPIO.BCM) if state=='on':
threshold = 1000 switchOnLight(LIGHT_PIN)
setCurrentState('on')
LDR_PIN = 18 elif state=='off':
LIGHT_PIN = 25 switchOffLight(LIGHT_PIN)
setCurrentState('off')
def readldr(PIN):
def getCurrentMode():
reading=0 cur.execute('SELECT * FROM myapp_mode')
1. Implement the native service in GPIO.setup(PIN, GPIO.OUT) data = cur.fetchone() #(1, u'auto')
Python and run on the device GPIO.output(PIN, GPIO.LOW) return data[1]
time.sleep(0.1)
GPIO.setup(PIN, GPIO.IN) def getCurrentState():
cur.execute('SELECT * FROM myapp_state')
while (GPIO.input(PIN)==GPIO.LOW): data = cur.fetchone() #(1, u'on')
reading=reading+1 return data[1]
return reading
def setCurrentState(val):
query='UPDATE myapp_state set name="'+val+'"'
def switchOnLight(PIN): cur.execute(query)
GPIO.setup(PIN, GPIO.OUT)
GPIO.output(PIN, GPIO.HIGH) while True:
currentMode=getCurrentMode()
def switchOffLight(PIN): if currentMode=='auto':
runAutoMode()
GPIO.setup(PIN, GPIO.OUT) elif currentMode=='manual':
GPIO.output(PIN, GPIO.LOW) runManualMode()
time.sleep(5)
Implementation: Application
# Views – views.py
def home(request):
1. Implement Django Application View out=‘’
if 'on' in request.POST:
values = {"name": "on"}
r=requests.put('http://127.0.0.1:8000/state/1/', data=values, auth=(‘username', ‘password'))
result=r.text
output = json.loads(result)
out=output['name']
if 'off' in request.POST:
values = {"name": "off"}
r=requests.put('http://127.0.0.1:8000/state/1/', data=values, auth=(‘username', ‘password'))
result=r.text
out=output['name']
if 'auto' in request.POST:
values = {"name": "auto"}
r=requests.put('http://127.0.0.1:8000/mode/1/', data=values, auth=(‘username', ‘password'))
result=r.text
out=output['name']
if 'manual' in request.POST:
values = {"name": "manual"}
r=requests.put('http://127.0.0.1:8000/mode/1/', data=values, auth=(‘username', ‘password'))
result=r.text
out=output['name']
r=requests.get('http://127.0.0.1:8000/mode/1/', auth=(‘username', ‘password'))

result=r.text
currentmode=output['name']
r=requests.get('http://127.0.0.1:8000/state/1/', auth=(‘username', ‘password'))
result=r.text
currentstate=output['name']
return render_to_response('lights.html',{'r':out, 'currentmode':currentmode, 'currentstate':currentstate},
context_instance=RequestContext(request))
Implementation: Application
<div class="app-content-inner">
<fieldset>
<div class="field clearfix">
2. Implement Django Application <label class="input-label icon-lamp" for="lamp-state">Auto</label>

<input id="lamp-state" class="input js-lamp-state hidden" type="checkbox">
Template
{% if currentmode == 'auto' %}
<div class="js-lamp-state-toggle ui-toggle " data-toggle=".js-lamp-state">
{% else %}
<div class="js-lamp-state-toggle ui-toggle js-toggle-off" data-toggle=".js-lamp-state">
{% endif %}
<span class="ui-toggle-slide clearfix">
<form id="my_form11" action="" method="post">{% csrf_token %}
<input name="auto" value="auto" type="hidden" />
<a href="#" onclick="$(this).closest('form').submit()"><strong class="ui-toggle-off">OFF</strong></a>
</form>
<strong class="ui-toggle-handle brushed-metal"></strong>
<input name="manual" value="manual" type="hidden" />
<a href="#" onclick="$(this).closest('form').submit()"><strong class="ui-toggle-on">ON</strong></a>
</form></span>
</div></div>
<div class="field clearfix">
<label class="input-label icon-lamp" for="tv-state">Light</label>
<input id="tv-state" class="input js-tv-state hidden" type="checkbox">
{% if currentstate == 'on' %}
<div class="js-tv-state-toggle ui-toggle " data-toggle=".js-tv-state">
{% else %}
<div class="js-tv-state-toggle ui-toggle js-toggle-off" data-toggle=".js-tv-state">
{% endif %}
{% if currentmode == 'manual' %}
<span class="ui-toggle-slide clearfix">
<input name="on" value="on" type="hidden" />
<a href="#" onclick="$(this).closest('form').submit()"><strong class="ui-toggle-off">OFF</strong></a>
</form>
<strong class="ui-toggle-handle brushed-metal"></strong>
<input name="off" value="off" type="hidden" />
<a href="#" onclick="$(this).closest('form').submit()"><strong class="ui-toggle-on">ON</strong></a>
</form>
</span>
{% endif %}
{% if currentmode == 'auto' %}
{% if currentstate == 'on' %}
<strong class="ui-toggle-on"> ON</strong>
{% else %}
<strong class="ui-toggle-on"> OFF</strong>
{% endif %}{% endif %}
</div>
</div>
</fieldset></div></div></div>
Finally - Integrate the System
• Setup the device
• Deploy and run the REST and Native services
• Deploy and run the Application
• Setup the database
Django Application
REST services implemented with Django-REST framework
SQLite Database
Native service implemented in Python
OS running on Raspberry Pi
Raspberry Pi device to which sensors and actuators are connected

Chapter 6
IoT Systems –
Logical Design using Python

Outline
• Introduction to Python
• Installing Python
• Python Data Types & Data Structures
• Control Flow
• Functions
• Modules
• Packages
• File Input/Output
• Date/Time Operations
• Classes

Python
• Python is a general-purpose high level programming language and suitable for providing a solid
foundation to the reader in the area of cloud computing.
• The main characteristics of Python are:

• Multi-paradigm programming language
• Python supports more than one programming paradigms including object-oriented programming and structured
programming
• Interpreted Language
• Python is an interpreted language and does not require an explicit compilation step. The Python interpreter
executes the program source code directly, statement by statement, as a processor or scripting engine does.
• Interactive Language
• Python provides an interactive mode in which the user can submit commands at the Python prompt and interact
with the interpreter directly.

Python - Benefits
• Easy-to-learn, read and maintain

• Python is a minimalistic language with relatively few keywords, uses English keywords and has fewer syntactical constructions
as compared to other languages. Reading Python programs feels like English with pseudo-code like constructs. Python is easy
to learn yet an extremely powerful language for a wide range of applications.
• Object and Procedure Oriented
• Python supports both procedure-oriented programming and object-oriented programming. Procedure oriented paradigm
allows programs to be written around procedures or functions that allow reuse of code. Procedure oriented paradigm allows
programs to be written around objects that include both data and functionality.
• Extendable
• Python is an extendable language and allows integration of low-level modules written in languages such as C/C++. This is
useful when you want to speed up a critical portion of a program.
• Scalable
• Due to the minimalistic nature of Python, it provides a manageable structure for large programs.
• Portable
• Since Python is an interpreted language, programmers do not have to worry about compilation, linking and loading of
programs. Python programs can be directly executed from source
• Broad Library Support
• Python has a broad library support and works on various platforms such as Windows, Linux, Mac, etc.

Python - Setup
• Windows
• Python binaries for Windows can be downloaded from http://www.python.org/getit .
• For the examples and exercise in this book, you would require Python 2.7 which can be directly downloaded from:
http://www.python.org/ftp/python/2.7.5/python-2.7.5.msi
• Once the python binary is installed you can run the python shell at the command prompt using
> python
• Linux
#Install Dependencies
sudo apt-get install build-essential
sudo apt-get install libreadline-gplv2-dev libncursesw5-dev libssl-dev libsqlite3-dev tk-dev libgdbm-dev libc6-dev libbz2-dev
#Download Python
wget http://python.org/ftp/python/2.7.5/Python-2.7.5.tgz
tar -xvf Python-2.7.5.tgz
cd Python-2.7.5
#Install Python
./configure
make
sudo make install

Numbers
• Numbers
• Number data type is used to store numeric values. Numbers are immutable data types, therefore changing the value of a number data
type results in a newly allocated object.
#Integer #Addition #Division
>>>a=5 >>>c=a+b >>>f=b/a
>>>type(a) >>>c >>>f
<type ’int’> 7.5 0.5
>>>type(c) >>>type(f)
#Floating Point <type ’float’> <type float’>
>>>b=2.5
>>>type(b) #Subtraction #Power
<type ’float’> >>>d=a-b >>>g=a**2
>>>d >>>g
#Long 2.5 25
>>>x=9898878787676L >>>type(d)
>>>type(x) <type ’float’>
<type ’long’>
#Multiplication
#Complex >>>e=a*b
>>>y=2+5j >>>e
>>>y 12.5
(2+5j) >>>type(e)
>>>type(y) <type ’float’>
<type ’complex’>
>>>y.real
2
>>>y.imag
5

Strings
• Strings
• A string is simply a list of characters in order. There are no limits to the number of characters you can have in a string.
#Create string #Print string #strip: Returns a copy of the string with the
>>>s="Hello World!" >>>print s #leading and trailing characters removed.
>>>type(s) Hello World!
<type ’str’> >>>s.strip("!")
#Formatting output ’Hello World’
#String concatenation >>>print "The string (The string (Hello World!)
>>>t="This is sample program." has 12 characters
>>>r = s+t
>>>r #Convert to upper/lower case
’Hello World!This is sample program.’ >>>s.upper()
’HELLO WORLD!’
#Get length of string >>>s.lower()
>>>len(s) ’hello world!’
12
#Accessing sub-strings
#Convert string to integer >>>s[0]
>>>x="100" ’H’
>>>type(s) >>>s[6:]
<type ’str’> ’World!’
>>>y=int(x) >>>s[6:-1]
>>>y ’World’
100

Lists
• Lists
• List a compound data type used to group together other values. List items need not all have the same type. A list contains items
separated by commas and enclosed within square brackets.
#Mixed data types in a list
#Create List #Removing an item from a list >>>mixed=[’data’,5,100.1,8287398L]
>>>fruits=[’apple’,’orange’,’banana’,’mango’] >>>fruits.remove(’mango’) >>>type(mixed)
>>>type(fruits) >>>fruits <type ’list’>
<type ’list’> [’apple’, ’orange’, ’banana’, ’pear’] >>>type(mixed[0])
<type ’str’>
#Get Length of List #Inserting an item to a list >>>type(mixed[1])
>>>len(fruits) >>>fruits.insert(1,’mango’) <type ’int’>
4 >>>fruits >>>type(mixed[2])
[’apple’, ’mango’, ’orange’, ’banana’, ’pear’] <type ’float’>
#Access List Elements >>>type(mixed[3])
>>>fruits[1] #Combining lists <type ’long’>
’orange’ >>>vegetables=[’potato’,’carrot’,’onion’,’beans’,’r
>>>fruits[1:3] adish’] #Change individual elements of a list
[’orange’, ’banana’] >>>vegetables >>>mixed[0]=mixed[0]+" items"
>>>fruits[1:] [’potato’, ’carrot’, ’onion’, ’beans’, ’radish’] >>>mixed[1]=mixed[1]+1
[’orange’, ’banana’, ’mango’] >>>mixed[2]=mixed[2]+0.05
>>>eatables=fruits+vegetables >>>mixed
#Appending an item to a list >>>eatables [’data items’, 6, 100.14999999999999, 8287398L]
>>>fruits.append(’pear’) [’appl
>>>fruits e’, #Lists can be nested
[’apple’, ’orange’, ’banana’, ’mango’, ’pear’] ’mang >>>nested=[fruits,vegetables]
o’, >>>nested
’orang [[’apple’, ’mango’, ’orange’, ’banana’, ’pear’],
e’, [’potato’, ’carrot’, ’onion’, ’beans’, ’radish’]]
’banan
a’,
’pear’, ’potato’, ’carrot’, ’onion’, ’beans’, ’radish’]
Tuples
• Tuples
• A tuple is a sequence data type that is similar to the list. A tuple consists of a number of values separated by commas and enclosed
within parentheses. Unlike lists, the elements of tuples cannot be changed, so tuples can be thought of as read-only lists.
#Create a Tuple #Get an element from a tuple

>>>fruits=("apple","mango","banana","pineapple") >>>fruits[0]
>>>fruits ’apple’
(’apple’, ’mango’, ’banana’, ’pineapple’) >>>fruits[:2]
(’apple’, ’mango’)
>>>type(fruits)
<type ’tuple’> #Combining tuples
>>>vegetables=(’potato’,’carrot’,’onion’,’radish’)
#Get length of tuple >>>eatables=fruits+vegetables
>>>len(fruits) >>>eatables
4 (’apple’, ’mango’, ’banana’, ’pineapple’, ’potato’, ’carrot’, ’onion’, ’radish’)

Dictionaries
• Dictionaries
• Dictionary is a mapping data type or a kind of hash table that maps keys to values. Keys in a dictionary can be of any data type, though
numbers and strings are commonly used for keys. Values in a dictionary can be any data type or object.
#Create a dictionary #Get all keys in a dictionary #Check if dictionary has a key
>>>student={’name’:’Mary’,’id’:’8776’,’major’:’CS’} >>>student.keys() >>>student.has_key(’name’)
>>>student [’gender’, ’major’, ’name’, ’id’] True
{’major’: ’CS’, ’name’: ’Mary’, ’id’: ’8776’} >>>student.has_key(’grade’)
>>>type(student) #Get all values in a dictionary False
<type ’dict’> >>>student.values()
[’female’, ’CS’, ’Mary’, ’8776’]
#Get length of a dictionary
>>>len(student) #Add new key-value pair
3 >>>student[’gender’]=’female’
>>>student
#Get the value of a key in dictionary {’gende
>>>student[’name’] r’: ’female’, ’major’: ’CS’, ’name’: ’Mary’, ’id’: ’8776’}
’Mary’
#A value in a dictionary can be another dictionary
#Get all items in a dictionary >>>student1={’name’:’David’,’id’:’9876’,’major’:’ECE’}
>>>student.items() >>>students={’1’: student,’2’:student1}
[(’gender’, ’female’), (’major’, ’CS’), (’name’, ’Mary’), >>>students
(’id’, ’8776’)] {’1’:
{’gende
r’: ’female’, ’major’: ’CS’, ’name’: ’Mary’, ’id’: ’8776’}, ’2’:
{’
major’: ’ECE’, ’name’: ’David’, ’id’: ’9876’}}

Type Conversions
• Type conversion examples
#Convert to string #Convert to long

>>>a=10000 >>>long(b)
>>>str(a) 2013L
’10000’
#Convert to list
#Convert to int >>>s="aeiou"
>>>b="2013" >>>list(s)
>>>int(b) [’a’, ’e’, ’i’, ’o’, ’u’]
2013
#Convert to set
#Convert to float >>>x=[’mango’,’apple’,’banana’,’mango’,’banana’]
>>>float(b) >>>set(x)
2013.0 set([’mango’, ’apple’, ’banana’])

Control Flow – if statement
• The if statement in Python is similar to the if statement in other languages.
>>>a = 25**5 >>>if a>10000: >>>s="Hello World" >>>student={’name’:’Mary’,’id’:’8776’}

>>>if a>10000: if a<1000000: >>>if "World" in s: >>>if not student.has_key(’major’):
print "More" print "Between 10k and 100k" s=s+"!" student[’major’]=’CS’
else: else: print s
print "Less" print "More than 100k" >>>student
elif a==10000: Hello World! {’major’: ’CS’, ’name’: ’Mary’, ’id’: ’8776’}
More print "Equal to 10k"
else:
print "Less than 10k"
More than 100k

Control Flow – for statement
• The for statement in Python iterates over items of any sequence (list, string, etc.) in the order in which they
appear in the sequence.
• This behavior is different from the for statement in other languages such as C in which an initialization,
incrementing and stopping criteria are provided.
#Looping over characters in a string #Looping over items in a list #Looping over keys in a dictionary
helloString = "Hello World" fruits=[’apple’,’orange’,’banana’,’mango’] student

=
for c in helloString: i=0 ’nam
print c for item in fruits: e’:
print "Fruit-%d: %s" % (i,item) ’Mar
i=i+1 y’, ’id’: ’8776’,’gender’: ’female’, ’major’: ’CS’
for key in student:

print "%s: %s" % (key,student[key]

Control Flow – while statement
• The while statement in Python executes the statements within the while loop as long as the while condition is
true.
#Prints even numbers upto 100
>>> i = 0
>>> while i<=100:

if i%2 == 0:
print i
i = i+1

Control Flow – range statement
• The range statement in Python generates a list of numbers in arithmetic progression.
#Generate a list of numbers from 0 – 9 #Generate a list of numbers from 10 - 100 with increments
of 10
>>>range (10)
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9] >>>range(10,110,10)
[10, 20, 30, 40, 50, 60, 70, 80, 90,100]

Control Flow – break/continue statements
• The break and continue statements in Python are similar to the statements in C.
• Break #Break statement example
• Break statement breaks out of the for/while loop >>>y=1
>>>for x in range(4,256,4):
y=y*x
if y > 512:
break
print y
4
32
384
• Continue #Continue statement example

• Continue statement continues with the next iteration. >>>fruits=[’apple’,’orange’,’banana’,’mango’]
>>>for item in fruits:
if item == "banana":
continue
else:
print item
apple
orange
mango

Control Flow – pass statement
• The pass statement in Python is a null operation.

• The pass statement is used when a statement is required syntactically but you do not want any command or
code to execute.
>fruits=[’apple’,’orange’,’banana’,’mango’]
>for item in fruits:
if item == "banana":
pass
else:
print item
apple
orange
mango

Functions
• A function is a block of code that takes information in (in the form of students = { '1': {'name': 'Bob', 'grade': 2.5},
parameters), does some computation, and returns a new piece of '2': {'name': 'Mary', 'grade': 3.5},
'3': {'name': 'David', 'grade': 4.2},
information based on the parameter information. '4': {'name': 'John', 'grade': 4.1},
'5': {'name': 'Alex', 'grade': 3.8}}
• A function in Python is a block of code that begins with the keyword

def averageGrade(students):
def followed by the function name and parentheses. The function “This function computes the average grade”
parameters are enclosed within the parenthesis. sum = 0.0
for key in students:
sum = sum + students[key]['grade']
average = sum/len(students)
• The code block within a function begins after a colon that comes after return average
the parenthesis enclosing the parameters.
avg = averageGrade(students)
print "The average garde is: %0.2f" % (avg)
• The first statement of the function body can optionally be a
documentation string or docstring.

Functions - Default Arguments
• Functions can have default values of the parameters.

• If a function with default values is called with fewer parameters or without any parameter, the default values of the
parameters are used
>>>def displayFruits(fruits=[’apple’,’orange’]):
print "There are %d fruits in the list" % (len(fruits))
for item in fruits:
print item
#Using default arguments

>>>displayFruits()
apple
orange
>>>fruits = [’banana’, ’pear’, ’mango’]

>>>displayFruits(fruits)
banana
pear
mango

Functions - Passing by Reference
• All parameters in the Python functions are passed by reference.

• If a parameter is changed within a function the change also reflected back in the calling function.
>>>def displayFruits(fruits):
print "There are %d fruits in the list" % (len(fruits))
for item in fruits:
print item
print "Adding one more fruit"
fruits.append('mango')
>>>fruits = ['banana', 'pear', 'apple']

>>>displayFruits(fruits)
There are 3 fruits in the list
banana
pear
apple
#Adding one more fruit

>>>print "There are %d fruits in the list" % (len(fruits))
There are 4 fruits in the list

Functions - Keyword Arguments
• Functions can also be called using keyword arguments that identifies the arguments by the parameter name when the
function is called.
>>>def #Correct use #name is a formal argument.

printStudentRecords(name,age=20,major=’CS’): >>>printStudentRecords(name=’Alex’) #**kwargs is a keyword argument that receives all
print "Name: " + name Name: Alex arguments except the formal argument as a
print "Age: " + str(age) Age: 20 dictionary.
print "Major: " + major Major: CS
>>>def student(name, **kwargs):
#This will give error as name is required argument >>>printStudentRecords(name=’Bob’,age=22,major=’EC print "Student Name: " + name
>>>printStudentRecords() E’) for key in kwargs:
Traceback (most recent call last): Name: Bob print key + ’: ’ + kwargs[key]
File "<stdin>", line 1, in <module> Age: 22
TypeError: printStudentRecords() takes at least 1 Major: ECE >>>student(name=’Bob’, age=’20’, major = ’CS’)
argument (0 given) Student Name: Bob
>>>printStudentRecords(name=’Alan’,major=’ECE’) age: 20
Name: Alan major: CS
Age: 20
Major: ECE

Functions - Variable Length Arguments
• Python functions can have variable length arguments. The variable length arguments are passed to as a tuple to the
function with an argument prefixed with asterix (*)
>>>def student(name, *varargs):

print "Student Name: " + name
for item in varargs:
print item
>>>student(’Nav’)
Student Name: Nav
>>>student(’Amy’, ’Age: 24’)

Student Name: Amy
Age: 24
>>>student(’Bob’, ’Age: 20’, ’Major: CS’)

Student Name: Bob
Age: 20
Major: CS

Modules
#student module - saved as student.py #Using student module
• Python allows organizing the program def averageGrade(students): >>>import student
code into different modules which sum = 0.0 >>>students = '1': 'name': 'Bob', 'grade': 2.5,
improves the code readability and for key in students: '2': 'name': 'Mary', 'grade': 3.5,
management. sum = sum + students[key]['grade'] '3': 'name': 'David', 'grade': 4.2,
average = sum/len(students) '4': 'name': 'John', 'grade': 4.1,
'5': 'name': 'Alex', 'grade': 3.8
• A module is a Python file that defines return average
some functionality in the form of functions def printRecords(students): >>>student.printRecords(students)

or classes. print "There are %d students" %(len(students)) There are 5 students
i=1 Student-1:
• Modules can be imported using the import for key in students: Name: Bob
Grade: 2.5
keyword. print "Student-%d: " % (i)
Student-2:
print "Name: " + students[key]['name']
print "Grade: " + str(students[key]['grade']) Name: David
• Modules to be imported must be present i = i+1 Grade: 4.2
in the search path. Student-3:
Name: Mary
# Importing a specific function from a module Grade: 3.5
>>>from student import averageGrade Student-4:
Name: Alex
# Listing all names defines in a module Grade: 3.8
>>>dir(student) Student-5:
Name: John
Grade: 4.1
>>>avg = student. averageGrade(students)

>>>print "The average garde is: %0.2f" % (avg)
3.62
Packages
• Python package is hierarchical file structure that consists of # skimage package listing
modules and subpackages. skimage/ Top level package
__init__.py Treat directory as a package
• Packages allow better organization of modules related to a single
application environment. color/ color color subpackage
__init__.py
colorconv.py
colorlabel.py
rgb_colors.py
draw/ draw draw subpackage

__init__.py
draw.py
setup.py
exposure/ exposure subpackage

__init__.py
_adapthist.py
exposure.py
feature/ feature subpackage

__init__.py
_brief.py
_daisy.py
...

File Handling
• Python allows reading and writing to files using the file # Example of reading an entire file
object. >>>fp = open('file.txt','r')
>>>content = fp.read()
>>>print content
This is a test file.
• The open(filename, mode) function is used to get a file >>>fp.close()
object.
# Example of reading line by line
>>>fp = open('file1.txt','r')
• The mode can be read (r), write (w), append (a), read and >>>print "Line-1: " + fp.readline()
write (r+ or w+), read-binary (rb), write-binary (wb), etc. Line-1: Python supports more than one programming paradigms.
>>>print "Line-2: " + fp.readline()
Line-2: Python is an interpreted language.
>>>fp.close()
• After the file contents have been read the close function is
called which closes the file object. # Example of reading lines in a loop
>>>fp = open(’file1.txt’,’r’)
>>>lines = fp.readlines()
>>>for line in lines:
print line
Python supports more than one programming paradigms.

Python is an interpreted language.

File Handling
# Example of reading a certain number of bytes # Example of seeking to a certain position
>>>fp = open('file.txt','r') >>>fp = open('file.txt','r')

>>>fp.read(10) >>>fp.seek(10,0)
'Python sup' >>>content = fp.read(10)
>>>fp.close() >>>print content
ports more
>>>fp.close()
# Example of getting the current position of read # Example of writing to a file
>>>fp = open('file.txt','r') >>>fo = open('file1.txt','w')

>>>fp.read(10) >>>content='This is an example of writing to a file in
'Python sup' Python.'
>>>currentpos = fp.tell >>>fo.write(content)
>>>print currentpos >>>fo.close()
<built-in method tell of file object at 0x0000000002391390>
>>>fp.close()

Date/Time Operations
• Python provides several functions for date and time access and conversions.
• The datetime module allows manipulating date and time in several ways.
• The time module in Python provides various time-related functions.
# Examples of manipulating with date # Examples of manipulating with time
>>>from datetime import date >>>import time

>>>nowtime = time.time()
>>>now = date.today() >>>time.localtime(nowtime)
>>>print "Date: " + now.strftime("%m-%d-%y") time.struct_time(tm_year=2013, tm_mon=7, tm_mday=24, tm_ec=51, tm_wday=2, tm_yday=205,
Date: 07-24-13 tm_isdst=0)
>>>print "Day of Week: " + now.strftime("%A") >>>time.asctime(time.localtime(nowtime))

Day of Week: Wednesday 'Wed Jul 24 16:14:51 2013'
>>>print "Month: " + now.strftime("%B") >>>time.strftime("The date is %d-%m-%y. Today is a %A. It is %H hours, %M minutes and %S seconds now.")
Month: July 'The date is 24-07-13. Today is a Wednesday. It is 16 hours, 15 minutes and 14 seconds now.'
>>>then = date(2013, 6, 7)
>>>timediff = now - then
>>>timediff.days
47

Classes
• Python is an Object-Oriented Programming (OOP) language. Python provides all the standard features of Object
Oriented Programming such as classes, class variables, class methods, inheritance, function overloading, and
operator overloading.
• Class
• A class is simply a representation of a type of object and user-defined prototype for an object that is composed of three things: a name,
attributes, and operations/methods.
• Instance/Object
• Object is an instance of the data structure defined by a class.
• Inheritance
• Inheritance is the process of forming a new class from an existing class or base class.
• Function overloading
• Function overloading is a form of polymorphism that allows a function to have different meanings, depending on its context.
• Operator overloading
• Operator overloading is a form of polymorphism that allows assignment of more than one function to a particular operator.
• Function overriding
• Function overriding allows a child class to provide a specific implementation of a function that is already provided by the base class. Child class
implementation of the overridden function has the same name, parameters and return type as the function in the base class.

Class Example
• The variable studentCount is a # Examples of a class >>>s = Student(’Steve’,’98928’)

class Student: Constructor called
class variable that is shared by studentCount = 0
all instances of the class >>>s.addGrade(’Math’,’90’)
Student and is accessed by def __init__(self, name, id): >>>s.addGrade(’Physics’,’85’)
print "Constructor called" >>>s.printGrades()
Student.studentCount. self.name = name Physics: 85
self.id = id Math: 90
• The variables name, id and Student.studentCount = Student.studentCount + 1
grades are instance variables self.grades={} >>>mathgrade = s.getGrade(’Math’)
which are specific to each >>>print mathgrade
def __del__(self): 90
instance of the class. print "Destructor called"
>>>count = s.getStudentCount()
• There is a special method by def getStudentCount(self): >>>print count
the name __init__() which is return Student.studentCount 1
the class constructor.
def addGrade(self,key,value): >>>del s
• The class constructor self.grades[key]=value Destructor called
def getGrade(self,key):
initializes a new instance return self.grades[key]
when it is created. The
function __del__() is the class def printGrades(self):
for key in self.grades:
destructor print key + ": " + self.grades[key]

Class Inheritance
• In this example Shape is the base class and Circle is the derived class. The class Circle inherits the attributes of the Shape class.
• The child class Circle overrides the methods and attributes of the base class (eg. draw() function defined in the base class Shape is overridden in child
class Circle).
# Examples of class inheritance class Circle(Shape): class Point: >>>p = Point(2,4)

class Shape: def __init__(self, c,r): def __init__(self, x, y): >>>circ = Circle(p,7)
def __init__(self): print "Child class constructor" self.xCoordinate = x Child class constructor
print "Base class constructor" self.center = c self.yCoordinate = y >>>circ.getColor()
self.color = ’Green’ self.radius = r ’Green’
self.lineWeight = 10.0 self.color = ’Green’ def setXCoordinate(self,x): >>>circ.setColor(’Red’)
self.lineWeight = 10.0 self.xCoordinate = x >>>circ.getColor()
def draw(self): self.__label = ’Hidden circle label’ ’Red’
print "Draw - to be implemented" def getXCoordinate(self): >>>circ.getLineWeight()
def setColor(self, c): def setCenter(self,c): return self.xCoordinate 10.0
self.color = c self.center = c >>>circ.getCenter().getXCoordinate()
def getColor(self): def getCenter(self): def setYCoordinate(self,y): 2
return self.color return self.center self.yCoordinate = y >>>circ.getCenter().getYCoordinate()
4
def setLineWeight(self,lwt): def setRadius(self,r): def getYCoordinate(self): >>>circ.draw()
self.lineWeight = lwt self.radius = r return self.yCoordinate Draw Circle (overridden function)
>>>circ.radius
def getLineWeight(self): def getRadius(self): 7
return self.lineWeight return self.radius
def draw(self):
print "Draw Circle (overridden function)"

Further Reading
• Code Academy Python Tutorial, http://www.codecademy.com/tracks/python

• Google's Python Class, https://developers.google.com/edu/python/
• Python Quick Reference Cheat Sheet, http://www.addedbytes.com/cheat-sheets/python-cheat-sheet/
• PyCharm Python IDE, http://www.jetbrains.com/pycharm/

Chapter 7
IoT Physical Devices & Endpoints

Outline
• Basic building blocks of an IoT Device

• Exemplary Device: Raspberry Pi
• Raspberry Pi interfaces
• Programming Raspberry Pi with Python
• Other IoT devices

What is an IoT Device
• A "Thing" in Internet of Things (IoT) can be any object that has a

unique identifier and which can send/receive data (including user
data) over a network (e.g., smart phone, smart TV, computer,
refrigerator, car, etc. ).
• IoT devices are connected to the Internet and send information

about themselves or about their surroundings (e.g. information
sensed by the connected sensors) over a network (to other devices or
servers/storage) or allow actuation upon the physical
entities/environment around them remotely.

IoT Device Examples
• A home automation device that allows remotely monitoring the

status of appliances and controlling the appliances.
• An industrial machine which sends information abouts its operation
and health monitoring data to a server.
• A car which sends information about its location to a cloud-based
service.
• A wireless-enabled wearable device that measures data about a
person such as the number of steps walked and sends the data to a
cloud-based service.

Basic building blocks of an IoT Device
• Sensing
• Sensors can be either on-board the IoT device or attached to the device.
• Actuation
• IoT devices can have various types of actuators attached that allow taking
• actions upon the physical entities in the vicinity of the device.
• Communication
• Communication modules are responsible for sending collected data to other
devices or cloud-based servers/storage and receiving data from other devices
and commands from remote applications.
• Analysis & Processing
• Analysis and processing modules are responsible for making sense of the
collected data.
Block diagram of an IoT Device

Exemplary Device: Raspberry Pi
• Raspberry Pi is a low-cost mini-computer with the physical size of a

credit card.
• Raspberry Pi runs various flavors of Linux and can perform almost all
tasks that a normal desktop computer can do.
• Raspberry Pi also allows interfacing sensors and actuators through
the general purpose I/O pins.
• Since Raspberry Pi runs Linux operating system, it supports Python
"out of the box".

Exemplary Device: Raspberry Pi
• Raspberry Pi is a low-cost mini-computer with the physical size of a

credit card.
• Raspberry Pi runs various flavors of Linux and can perform almost all
tasks that a normal desktop computer can do.
• Raspberry Pi also allows interfacing sensors and actuators through
the general purpose I/O pins.
• Since Raspberry Pi runs Linux operating system, it supports Python
"out of the box".

Raspberry Pi

Linux on Raspberry Pi
• Raspbian
• Raspbian Linux is a Debian Wheezy port optimized for Raspberry Pi.
• Arch
• Arch is an Arch Linux port for AMD devices.
• Pidora
• Pidora Linux is a Fedora Linux optimized for Raspberry Pi.
• RaspBMC
• RaspBMC is an XBMC media-center distribution for Raspberry Pi.
• OpenELEC
• OpenELEC is a fast and user-friendly XBMC media-center distribution.
• RISC OS
• RISC OS is a very fast and compact operating system.

Raspberry Pi GPIO

Raspberry Pi Interfaces
• Serial
• The serial interface on Raspberry Pi has receive (Rx) and transmit (Tx) pins
for communication with serial peripherals.
• SPI
• Serial Peripheral Interface (SPI) is a synchronous serial data protocol used
for communicating with one or more peripheral devices.
• I2C
• The I2C interface pins on Raspberry Pi allow you to connect hardware
modules. I2C interface allows synchronous data transfer with just two pins -
SDA (data line) and SCL (clock line).

Raspberry Pi Example:
Interfacing LED and switch with Raspberry Pi
from time import sleep
import RPi.GPIO as GPIO
GPIO.setmode(GPIO.BCM)
#Switch Pin
GPIO.setup(25, GPIO.IN)
#LED Pin
GPIO.setup(18, GPIO.OUT)
state=false
def toggleLED(pin):
state = not state
GPIO.output(pin, state)
while True:
try:
if (GPIO.input(25) == True):
toggleLED(pin)
sleep(.01)
except KeyboardInterrupt:
exit()

Other Devices
• pcDuino
• BeagleBone Black
• Cubieboard

Chapter 8
IoT Physical Servers &

Cloud Offerings

Outline
• WAMP - AutoBahn for IoT

• Python for Amazon Web Services
• Python for MapReduce
• Python Packages of Interest
• Python Web Application Framework - Django
• Development with Django

WAMP for IoT
• Web Application Messaging Protocol (WAMP) is a sub-protocol of Websocket

which provides publish-subscribe and remote procedure call (RPC) messaging
patterns.

WAMP - Concepts
• Transport: Transport is channel that connects two peers.
• Session: Session is a conversation between two peers that runs over a transport.
• Client: Clients are peers that can have one or more roles. In publish-subscribe model client can have following roles:
– Publisher: Publisher publishes events (including payload) to the topic maintained by the Broker.
– Subscriber: Subscriber subscribes to the topics and receives the events including the payload.
In RPC model client can have following roles:

– Caller: Caller issues calls to the remote procedures along with call arguments.
– Callee: Callee executes the procedures to which the calls are issued by the caller and returns the results back to the caller.
• Router: Routers are peers that perform generic call and event routing. In publish-subscribe model Router has the role of
a Broker:
– Broker: Broker acts as a router and routes messages published to a topic to all subscribers subscribed to the topic.
In RPC model Router has the role of a Broker:

– Dealer: Dealer acts a router and routes RPC calls from the Caller to the Callee and routes results from Callee to Caller.
• Application Code: Application code runs on the Clients (Publisher, Subscriber, Callee or Caller).

Amazon EC2 – Python Example
• Boto is a Python package that provides interfaces to Amazon Web Services (AWS)
#Python program for launching an EC2 instance
• In this example, a connection to EC2 service is fi import boto.ec2
rst established by calling from time import sleep
boto.ec2.connect_to_region. ACCESS_KEY="<enter access key>"
SECRET_KEY="<enter secret key>"
• The EC2 region, AWS access key and AWS secret REGION="us-east-1"
key are passed to this function. After connecting AMI_ID = "ami-d0f89fb9"
EC2_KEY_HANDLE = "<enter key handle>"
to EC2 , a new instance is launched using the INSTANCE_TYPE="t1.micro"
conn.run_instances function. SECGROUP_HANDLE="default"
conn = boto.ec2.connect_to_region(REGION, aws_access_key_id=ACCESS_KEY,

• The AMI-ID, instance type, EC2 key handle and aws_secret_access_key=SECRET_KEY)
security group are passed to this function.
reservation = conn.run_instances(image_id=AMI_ID, key_name=EC2_KEY_HANDLE,
instance_type=INSTANCE_TYPE,
security_groups = [ SECGROUP_HANDLE, ] )

Amazon AutoScaling – Python Example
#Python program for creating an AutoScaling group (code excerpt)
• AutoScaling Service import boto.ec2.autoscale
:
• A connection to AutoScaling service is first established by print "Connecting to Autoscaling Service"
calling boto.ec2.autoscale.connect_to_region function. conn = boto.ec2.autoscale.connect_to_region(REGION,
aws_access_key_id=ACCESS_KEY,
aws_secret_access_key=SECRET_KEY)
• Launch Configuration
• After connecting to AutoScaling service, a new launch print "Creating launch configuration"
configuration is created by calling
conn.create_launch_con f iguration. Launch configuration lc = LaunchConfiguration(name='My-Launch-Config-2',
contains instructions on how to launch new instances image_id=AMI_ID,
including the AMI-ID, instance type, security groups, etc. key_name=EC2_KEY_HANDLE,
instance_type=INSTANCE_TYPE,
security_groups = [ SECGROUP_HANDLE, ])
• AutoScaling Group conn.create_launch_configuration(lc)
• After creating a launch configuration, it is then associated print "Creating auto-scaling group"
with a new AutoScaling group. AutoScaling group is
created by calling conn.create_auto_scaling_group. The ag = AutoScalingGroup(group_name='My-Group',
settings for AutoScaling group such as the maximum and availability_zones=['us-east-1b'],
minimum number of instances in the group, the launch launch_config=lc, min_size=1, max_size=2,
configuration, availability zones, optional load balancer to connection=conn)
use with the group, etc. conn.create_auto_scaling_group(ag)

• AutoScaling Policies #Creating auto-scaling policies
• After creating an AutoScaling group, the policies for scale_up_policy = ScalingPolicy(name='scale_up',

adjustment_type='ChangeInCapacity',
scaling up and scaling down are defined. as_name='My-Group',
• In this example, a scale up policy with adjustment type scaling_adjustment=1,
cooldown=180)
ChangeInCapacity and scaling_ad justment = 1 is
defined. scale_down_policy = ScalingPolicy(name='scale_down',
adjustment_type='ChangeInCapacity',
• Similarly a scale down policy with adjustment type as_name='My-Group',
ChangeInCapacity and scaling_ad justment = -1 is scaling_adjustment=-1,
defined. cooldown=180)
conn.create_scaling_policy(scale_up_policy)
conn.create_scaling_policy(scale_down_policy)

#Connecting to CloudWatch
cloudwatch = boto.ec2.cloudwatch.connect_to_region(REGION,
• CloudWatch Alarms aws_access_key_id=ACCESS_KEY,
• With the scaling policies defined, the next step is to aws_secret_access_key=SECRET_KEY)
alarm_dimensions = {"AutoScalingGroupName": 'My-Group'}
create Amazon CloudWatch alarms that trigger these
policies. #Creating scale-up alarm
scale_up_alarm = MetricAlarm(
• The scale up alarm is defined using the CPUUtilization name='scale_up_on_cpu', namespace='AWS/EC2',
metric with the Average statistic and threshold greater metric='CPUUtilization', statistic='Average',
70% for a period of 60 sec. The scale up policy created comparison='>', threshold='70',
period='60', evaluation_periods=2,
previously is associated with this alarm. This alarm is alarm_actions=[scale_up_policy.policy_arn],
triggered when the average CPU utilization of the dimensions=alarm_dimensions)
cloudwatch.create_alarm(scale_up_alarm)
instances in the group becomes greater than 70% for
more than 60 seconds. #Creating scale-down alarm
scale_down_alarm = MetricAlarm(
• The scale down alarm is defined in a similar manner name='scale_down_on_cpu', namespace='AWS/EC2',
with a threshold less than 50%. metric='CPUUtilization', statistic='Average',
comparison='<', threshold='40',
period='60', evaluation_periods=2,
alarm_actions=[scale_down_policy.policy_arn],
dimensions=alarm_dimensions)
cloudwatch.create_alarm(scale_down_alarm)

Amazon S3 – Python Example
• In this example, a connection to S3 service is first established by calling boto.connect_s3 function.

• The upload_to_s3_bucket_path function uploads the file to the S3 bucket specified at the specified
path.
# Python program for uploading a file to an S3 bucket
import boto.s3
conn = boto.connect_s3(aws_access_key_id='<enter>',
aws_secret_access_key='<enter>')
def percent_cb(complete, total):

print ('.')
def upload_to_s3_bucket_path(bucketname, path, filename):

mybucket = conn.get_bucket(bucketname)
fullkeyname=os.path.join(path,filename)
key = mybucket.new_key(fullkeyname)
key.set_contents_from_filename(filename, cb=percent_cb, num_cb=10)

Amazon RDS – Python Example
#Python program for launching an RDS instance (excerpt)
import boto.rds
• In this example, a connection to RDS service is first
established by calling boto.rds.connect_to_region ACCESS_KEY="<enter>"
SECRET_KEY="<enter>"
function. REGION="us-east-1"
INSTANCE_TYPE="db.t1.micro"
• The RDS region, AWS access key and AWS secret key ID = "MySQL-db-instance-3"
are passed to this function. USERNAME = 'root'
PASSWORD = 'password'
• After connecting to RDS service, the DB_PORT = 3306
DB_SIZE = 5
conn.create_dbinstance function is called to launch a DB_ENGINE = 'MySQL5.1'
new RDS instance. DB_NAME = 'mytestdb'
SECGROUP_HANDLE="default"
• The input parameters to this function include the #Connecting to RDS

instance ID, database size, instance type, database conn = boto.rds.connect_to_region(REGION,
username, database password, database port, aws_secret_access_key=SECRET_KEY)
database engine (e.g. MySQL5.1), database name,
security groups, etc. #Creating an RDS instance
db = conn.create_dbinstance(ID, DB_SIZE, INSTANCE_TYPE,
USERNAME, PASSWORD, port=DB_PORT, engine=DB_ENGINE,
db_name=DB_NAME, security_groups = [ SECGROUP_HANDLE, ] )

Amazon DynamoDB – Python Example
# Python program for creating a DynamoDB table (excerpt)
import boto.dynamodb
• In this example, a connection to DynamoDB service is
first established by calling ACCESS_KEY="<enter>"
SECRET_KEY="<enter>"
boto.dynamodb.connect_to_region. REGION="us-east-1"
• After connecting to DynamoDB service, a schema for #Connecting to DynamoDB

the new table is created by calling conn = boto.dynamodb.connect_to_region(REGION,
conn.create_schema. aws_secret_access_key=SECRET_KEY)
• The schema includes the hash key and range key table_schema = conn.create_schema(
names and types. hash_key_name='msgid',
hash_key_proto_value=str,
range_key_name='date',
• A DynamoDB table is then created by calling range_key_proto_value=str
conn.create_table function with the table schema, )
read units and write units as input parameters. #Creating table with schema
table = conn.create_table(
name='my-test-table',
schema=table_schema,
read_units=1,
write_units=1
)

Python for MapReduce
• The example shows inverted index mapper #Inverted Index Mapper in Python
program. #!/usr/bin/env python
• The map function reads the data from the import sys
for line in sys.stdin:
standard input (stdin) and splits the tab-limited doc_id, content = line.split(’’)
data into document-ID and contents of the words = content.split()
for word in words:
document. print ’%s%s’ % (word, doc_id)
• The map function emits key-value pairs where

key is each word in the document and value is
the document-ID.

Python for MapReduce
• The example shows inverted index reducer #Inverted Index Reducer in Python
program. #!/usr/bin/env python
• The key-value pairs emitted by the map phase import sys
current_word = None
are shuffled to the reducers and grouped by the current_docids = []
key. word = None
for line in sys.stdin:

• The reducer reads the key-value pairs grouped # remove leading and trailing whitespace
by the same key from the standard input (stdin) line = line.strip()
# parse the input we got from mapper.py
and creates a list of document-IDs in which the word, doc_id = line.split(’’)
word occurs. if current_word == word:
current_docids.append(doc_id)
• The output of reducer contains key value pairs else:
if current_word:
where key is a unique word and value is the list print ’%s%s’ % (current_word, current_docids)
of document-IDs in which the word occurs. current_docids = []
current_docids.append(doc_id)
current_word = word

Python Packages of Interest
• JSON
• JavaScript Object Notation (JSON) is an easy to read and write data-interchange format. JSON is used as an alternative to XML
and is is easy for machines to parse and generate. JSON is built on two structures - a collection of name-value pairs (e.g. a
Python dictionary) and ordered lists of values (e.g.. a Python list).
• XML
• XML (Extensible Markup Language) is a data format for structured document interchange. The Python minidom library
provides a minimal implementation of the Document Object Model interface and has an API similar to that in other languages.
• HTTPLib & URLLib
• HTTPLib2 and URLLib2 are Python libraries used in network/internet programming
• SMTPLib
• Simple Mail Transfer Protocol (SMTP) is a protocol which handles sending email and routing e-mail between mail servers. The
Python smtplib module provides an SMTP client session object that can be used to send email.
• NumPy
• NumPy is a package for scientific computing in Python. NumPy provides support for large multi-dimensional arrays and
matrices
• Scikit-learn
• Scikit-learn is an open source machine learning library for Python that provides implementations of various machine learning
algorithms for classification, clustering, regression and dimension reduction problems.

Python Web Application Framework - Django
• Django is an open source web application framework for developing web applications in Python.
• A web application framework in general is a collection of solutions, packages and best practices
that allows development of web applications and dynamic websites.
• Django is based on the Model-Template-View architecture and provides a separation of the data
model from the business rules and the user interface.
• Django provides a unified API to a database backend.
• Thus web applications built with Django can work with different databases without requiring any
code changes.
• With this fiexibility in web application design combined with the powerful capabilities of the Python
language and the Python ecosystem, Django is best suited for cloud applications.
• Django consists of an object-relational mapper, a web templating system and a regular-expression-
based URL dispatcher.

Django Architecture
• Django is Model-Template-View (MTV) framework.
• Model
• The model acts as a definition of some stored data and handles the interactions with the database. In a web
application, the data can be stored in a relational database, non-relational database, an XML file, etc. A Django model
is a Python class that outlines the variables and methods for a particular type of data.
• Template
• In a typical Django web application, the template is simply an HTML page with a few extra placeholders. Django’s
template language can be used to create various forms of text files (XML, email, CSS, Javascript, CSV, etc.)
• View
• The view ties the model to the template. The view is where you write the code that actually generates the web pages.
View determines what data is to be displayed, retrieves the data from the database and passes the data to the
template.

Further Reading
• boto, http://boto.readthedocs.org/en/latest/
• Python JSON package, http://docs.python.org/library/json.html
• Python socket package, http://docs.python.org/2/library/socket.html
• Python email package, http://docs.python.org/2/library/email
• Python HTTPLib, http://code.google.com/p/httplib2/
• Python URLLib, http://docs.python.org/2/howto/urllib2.html
• Python SMTPLib, http://docs.python.org/2/library/smtplib.html
• NumPy, http://www.numpy.org/
• Scikit-learn, http://scikit-learn.org/stable/
• Django, https://docs.djangoproject.com/en/1.5/
• Google App Engine, https://developers.google.com/appengine/
• Google Cloud Storage, https://developers.google.com/storage/
• Google BigQuery, https://developers.google.com/bigquery/
• Google Cloud Datastore, http://developers.google.com/datastore/
• Google Cloud SQL, https://developers.google.com/cloud-sql/
• AFINN, http://www2.imm.dtu.dk/pubdb/views/publication_details.php?id=6010
• Tweepy Package, https://github.com/tweepy/tweepy

Chapter 10
Data Analytics for IoT

Outline
• Overview of Hadoop ecosystem

• MapReduce architecture
• MapReduce job execution flow
• MapReduce schedulers

Hadoop Ecosystem
• Apache Hadoop is an open source framework for distributed batch processing of big data.
• Hadoop Ecosystem includes:
• Hadoop MapReduce
• HDFS
• YARN
• HBase
• Zookeeper
• Pig
• Hive
• Mahout
• Chukwa
• Cassandra
• Avro
• Oozie
• Flume
• Sqoop

Apache Hadoop
• A Hadoop cluster comprises of a Master node, backup node and a number

of slave nodes.
• The master node runs the NameNode and JobTracker processes and the
slave nodes run the DataNode and TaskTracker components of Hadoop.
• The backup node runs the Secondary NameNode process.
• NameNode
• NameNode keeps the directory tree of all files in the file system, and tracks
where across the cluster the file data is kept. It does not store the data of these
files itself. Client applications talk to the NameNode whenever they wish to
locate a file, or when they want to add/copy/move/delete a file.
• Secondary NameNode
• NameNode is a Single Point of Failure for the HDFS Cluster. An optional
Secondary NameNode which is hosted on a separate machine creates
checkpoints of the namespace.
• JobTracker
• The JobTracker is the service within Hadoop that distributes MapReduce tasks to
specific nodes in the cluster, ideally the nodes that have the data, or at least are
in the same rack.

Apache Hadoop
• TaskTracker
• TaskTracker is a node in a Hadoop cluster that accepts Map, Reduce
and Shuffie tasks from the JobTracker.
• Each TaskTracker has a defined number of slots which indicate the
number of tasks that it can accept.
• DataNode
• A DataNode stores data in an HDFS file system.
• A functional HDFS filesystem has more than one DataNode, with data
replicated across them.
• DataNodes respond to requests from the NameNode for filesystem
operations.
• Client applications can talk directly to a DataNode, once the
NameNode has provided the location of the data.
• Similarly, MapReduce operations assigned to TaskTracker instances
near a DataNode, talk directly to the DataNode to access the files.
• TaskTracker instances can be deployed on the same servers that host
DataNode instances, so that MapReduce operations are performed
close to the data.

MapReduce
• MapReduce job consists of two phases:

• Map: In the Map phase, data is read from a distributed
file system and partitioned among a set of computing
nodes in the cluster. The data is sent to the nodes as a set
of key-value pairs. The Map tasks process the input
records independently of each other and produce
intermediate results as key-value pairs. The intermediate
results are stored on the local disk of the node running
the Map task.
• Reduce: When all the Map tasks are completed, the
Reduce phase begins in which the intermediate data with
the same key is aggregated.
• Optional Combine Task

• An optional Combine task can be used to perform data
aggregation on the intermediate data of the same key for
the output of the mapper before transferring the output
to the Reduce task.

MapReduce Job Execution Workflow
• MapReduce job execution starts when the client applications submit jobs to the Job tracker.
• The JobTracker returns a JobID to the client application. The JobTracker talks to the NameNode to determine
the location of the data.
• The JobTracker locates TaskTracker nodes with available slots at/or near the data.
• The TaskTrackers send out heartbeat messages to the JobTracker, usually every few minutes, to reassure the
JobTracker that they are still alive. These messages also inform the JobTracker of the number of available
slots, so the JobTracker can stay up to date with where in the cluster, new work can be delegated.

MapReduce Job Execution Workflow
• The JobTracker submits the work to the TaskTracker nodes when they poll for tasks. To choose a task for a
TaskTracker, the JobTracker uses various scheduling algorithms (default is FIFO).
• The TaskTracker nodes are monitored using the heartbeat signals that are sent by the TaskTrackers to
JobTracker.
• The TaskTracker spawns a separate JVM process for each task so that any task failure does not bring down
the TaskTracker.
• The TaskTracker monitors these spawned processes while capturing the output and exit codes. When the
process finishes, successfully or not, the TaskTracker notifies the JobTracker. When the job is completed, the
JobTracker updates its status.

MapReduce 2.0 - YARN
• In Hadoop 2.0 the original processing engine of Hadoop

(MapReduce) has been separated from the resource
management (which is now part of YARN).
• This makes YARN effectively an operating system for

Hadoop that supports different processing engines on a
Hadoop cluster such as MapReduce for batch processing,
Apache Tez for interactive queries, Apache Storm for
stream processing, etc.
• YARN architecture divides architecture divides the two

major functions of the JobTracker - resource management
and job life-cycle management - into separate components:
• ResourceManager
• ApplicationMaster.

YARN Components
• Resource Manager (RM): RM manages the global assignment of

compute resources to applications. RM consists of two main
services:
• Scheduler: Scheduler is a pluggable service that manages and enforces the
resource scheduling policy in the cluster.
• Applications Manager (AsM): AsM manages the running Application Masters in
the cluster. AsM is responsible for starting application masters and for monitoring
and restarting them on different nodes in case of failures.
• Application Master (AM): A per-application AM manages the

application’s life cycle. AM is responsible for negotiating resources
from the RM and working with the NMs to execute and monitor
the tasks.
• Node Manager (NM): A per-machine NM manages the user
processes on that machine.
• Containers: Container is a bundle of resources allocated by RM
(memory, CPU, network, etc.). A container is a conceptual entity
that grants an application the privilege to use a certain amount of
resources on a given machine to run a component task.

Hadoop Schedulers
• Hadoop scheduler is a pluggable component that makes it open to support

different scheduling algorithms.
• The default scheduler in Hadoop is FIFO.
• Two advanced schedulers are also available - the Fair Scheduler, developed
at Facebook, and the Capacity Scheduler, developed at Yahoo.
• The pluggable scheduler framework provides the flexibility to support a
variety of workloads with varying priority and performance constraints.
• Efficient job scheduling makes Hadoop a multi-tasking system that can
process multiple data sets for multiple jobs for multiple users
simultaneously.

FIFO Scheduler
• FIFO is the default scheduler in Hadoop that maintains a work queue

in which the jobs are queued.
• The scheduler pulls jobs in first in first out manner (oldest job first)
for scheduling.
• There is no concept of priority or size of job in FIFO scheduler.

Fair Scheduler
• The Fair Scheduler allocates resources evenly between multiple jobs and also provides capacity
guarantees.
• Fair Scheduler assigns resources to jobs such that each job gets an equal share of the available
resources on average over time.
• Tasks slots that are free are assigned to the new jobs, so that each job gets roughly the same
amount of CPU time.
• Job Pools
• The Fair Scheduler maintains a set of pools into which jobs are placed. Each pool has a guaranteed capacity.
• When there is a single job running, all the resources are assigned to that job. When there are multiple jobs in
the pools, each pool gets at least as many task slots as guaranteed.
• Each pool receives at least the minimum share.
• When a pool does not require the guaranteed share the excess capacity is split between other jobs.
• Fairness
• The scheduler computes periodically the difference between the computing time received by each job and
the time it should have received in ideal scheduling.
• The job which has the highest deficit of the compute time received is scheduled next.

Capacity Scheduler
• The Capacity Scheduler has similar functionally as the Fair Scheduler but
adopts a different scheduling philosophy.
• Queues
• In Capacity Scheduler, you define a number of named queues each with a
configurable number of map and reduce slots.
• Each queue is also assigned a guaranteed capacity.
• The Capacity Scheduler gives each queue its capacity when it contains jobs, and
shares any unused capacity between the queues. Within each queue FIFO scheduling
with priority is used.
• Fairness
• For fairness, it is possible to place a limit on the percentage of running tasks per user,
so that users share a cluster equally.
• A wait time for each queue can be configured. When a queue is not scheduled for
more than the wait time, it can preempt tasks of other queues to get its fair share.

Further Reading
• Apache Hadoop, http://hadoop.apache.org

• Apache Hive, http://hive.apache.org
• Apache HBase, http://hbase.apache.org
• Apache Chukwa, http://chukwa.apache.org
• Apache Flume, http://flume.apache.org
• Apache Zookeeper, http://zookeeper.apache.org
• Apache Avro, http://avro.apache.org
• Apache Oozie, http://oozie.apache.org
• Apache Storm, http://storm-project.net
• Apache Tez, http://tez.incubator.apache.org
• Apache Cassandra, http://cassandra.apache.org
• Apache Mahout, http://mahout.apache.org
• Apache Pig, http://pig.apache.org
• Apache Sqoop, http://sqoop.apache.org

Bahga Madisetti IoT Book Slides Justin Joseph

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Bahga Madisetti IoT Book Slides Justin Joseph

Uploaded by

Copyright:

Available Formats

Chapter 1

Book website: http://www.internet-of-things-book.com Bahga & Madisetti, © 2015

Book website: http://www.internet-of-things-book.com Bahga & Madisetti, © 2015

A dynamic global network infrastructure with self-conﬁguring

Book website: http://www.internet-of-things-book.com Bahga & Madisetti, © 2015

• Dynamic & Self-Adapting

Book website: http://www.internet-of-things-book.com Bahga & Madisetti, © 2015

Book website: http://www.internet-of-things-book.com Bahga & Madisetti, © 2015

• An IoT device may consist of

Book website: http://www.internet-of-things-book.com Bahga & Madisetti, © 2015

• Logical design of an IoT system

• An IoT system comprises of a

Book website: http://www.internet-of-things-book.com Bahga & Madisetti, © 2015

• When the server receives a

Book website: http://www.internet-of-things-book.com Bahga & Madisetti, © 2015

Book website: http://www.internet-of-things-book.com Bahga & Madisetti, © 2015

Book website: http://www.internet-of-things-book.com Bahga & Madisetti, © 2015

Book website: http://www.internet-of-things-book.com Bahga & Madisetti, © 2015

• Representational State Transfer

Book website: http://www.internet-of-things-book.com Bahga & Madisetti, © 2015

• WebSocket APIs allow bi-

Book website: http://www.internet-of-things-book.com Bahga & Madisetti, © 2015

Book website: http://www.internet-of-things-book.com Bahga & Madisetti, © 2015

An IoT system comprises of the following components:

Book website: http://www.internet-of-things-book.com Bahga & Madisetti, © 2015

Book website: http://www.internet-of-things-book.com Bahga & Madisetti, © 2015

• A level-1 IoT system has a

Book website: http://www.internet-of-things-book.com Bahga & Madisetti, © 2015

• A level-2 IoT system has a

Book website: http://www.internet-of-things-book.com Bahga & Madisetti, © 2015

• A level-3 IoT system has a

Book website: http://www.internet-of-things-book.com Bahga & Madisetti, © 2015

• A level-4 IoT system has multiple

Book website: http://www.internet-of-things-book.com Bahga & Madisetti, © 2015

• A level-5 IoT system has multiple end

Book website: http://www.internet-of-things-book.com Bahga & Madisetti, © 2015

• A level-6 IoT system has multiple

Book website: http://www.internet-of-things-book.com Bahga & Madisetti, © 2015

IoT & M2M

Book website: http://www.internet-of-things-book.com Bahga & Madisetti, © 2015

Book website: http://www.internet-of-things-book.com Bahga & Madisetti, © 2015

• Machine-to-Machine (M2M) refers to networking of machines (or

Book website: http://www.internet-of-things-book.com Bahga & Madisetti, © 2015

Book website: http://www.internet-of-things-book.com Bahga & Madisetti, © 2015

Book website: http://www.internet-of-things-book.com Bahga & Madisetti, © 2015

Book website: http://www.internet-of-things-book.com Bahga & Madisetti, © 2015

• Hardware vs Software Emphasis

Book website: http://www.internet-of-things-book.com Bahga & Madisetti, © 2015

Book website: http://www.internet-of-things-book.com Bahga & Madisetti, © 2015

Book website: http://www.internet-of-things-book.com Bahga & Madisetti, © 2015

• Centralized Network Controller

Book website: http://www.internet-of-things-book.com Bahga & Madisetti, © 2015

Book website: http://www.internet-of-things-book.com Bahga & Madisetti, © 2015

• Virtualized Network Function (VNF):

Book website: http://www.internet-of-things-book.com Bahga & Madisetti, © 2015

Book website: http://www.internet-of-things-book.com Bahga & Madisetti, © 2015

IoT System Management with

Book website: http://www.internet-of-things-book.com Bahga & Madisetti, © 2015

• Need for IoT Systems Management

Book website: http://www.internet-of-things-book.com Bahga & Madisetti, © 2015

Book website: http://www.internet-of-things-book.com Bahga & Madisetti, © 2015

• SNMP is a well-known and widely used

Book website: http://www.internet-of-things-book.com Bahga & Madisetti, © 2015

Book website: http://www.internet-of-things-book.com Bahga & Madisetti, © 2015