11/5/19

The Internet of Things:

Architecture and Applications
(ELEC423)

Dr Valerio Selis

V.Selis@liverpool.ac.uk

Outline
• M2M in the IoT:
• Reference Architecture
• Abstraction
• MQTT messages
• MQTT subscriber with Paho-MQTT Python3
module
• Messages formatting with XML module
• Messages parsing with defusedxml module
• MQTT solutions for IoT

M2M in the IoT

Abstraction
Server
M2M Network The
domain M2M app
Internet
Communication

Gateway

M2M Device
and Gateway Local
domain Network

“Thing” “Thing”
M2M app M2M app

Communication Communication

Sensor Actuator

1
 11/5/19

M2M in the IoT

Server – MQTT Broker

Mosquitto MQTT broker

Unencrypted port: 1883

Server Encrypted port: 8883
M2M app
Communication

Ethernet

mqtt.eclipse.org
(137.135.83.217)

M2M in the IoT

Things – MQTT Publisher
Python with Paho-MQTT module

“Thing” BCM43438 Wi-Fi chipset

M2M app

Communication

Sensor

DHT11 sensor module

Humidity and
Temperature

M2M in the IoT

Things – MQTT Publisher
import paho.mqtt.client as mqtt
def on_publish(client, userdata, mid):
print("Message ID: " + str(mid))
pass
def on_disconnect(client, userdata, rc):
print("Disconnected with result code: " + str(rc))
publisher = mqtt.Client("Sensor")
publisher.username_pw_set(username="pi", password="pwd")
publisher.on_publish = on_publish
publisher.connect("mqtt.eclipse.org", 8883, 60)
(rc, mid) = publisher.publish("sensor/temp", "10")
print("Published with return code: " + str(rc))
publisher.on_disconnect = on_disconnect
publisher.disconnect()

2
 11/5/19

M2M in the IoT

MQTT messages
MQTT subscriber MQTT broker MQTT publisher
CONNECT command
(MQTT 3.1.1, Client ID, flags)

CONNECT ack
(Return Code)

SUBSCRIBE request
(topic, QoS 1)

SUBSCRIBE ack
(Granted QoS 1)

CONNECT command
(MQTT 3.1.1, Client ID, flags)

CONNECT ack
(Return code)

PUBLISH message
(topic, message)

PUBLISH message
(topic, message)

M2M in the IoT

Things – MQTT Subscriber
Python with Paho-MQTT module

“Thing” BCM43438 Wi-Fi chipset

M2M app

Communication

Actuator

LED

M2M in the IoT

Things – MQTT Subscriber

The MQTT subscriber can also be created as a

Python3 script.
• Import the Paho-MQTT module as:
import paho.mqtt.client as mqtt
• To manage MQTT connections, the “Client”
instance needs to be created:
subscriber = mqtt.Client()
The “Client” constructor supports:
• client_id: unique client ID for the publisher (string)
• protocol: version of the MQTT protocol to be used
(MQTTv31 or MQTTv311)

3
 11/5/19

M2M in the IoT

Things – MQTT Subscriber

• Before connecting to the broker:

• Set the subscriber to use the previous configured
user/password:
subscriber.username_pw_set(
username="<username>",password="<password>")
• Replace “<username>” and “<password>” with
the username and password used when configuring
the broker

10

M2M in the IoT

Things – MQTT Subscriber
• Create a callback function for when the subscriber
receives a CONNACK response from the broker:
def on_connect(client, userdata, flags, rc):
print("Connected with result code: " +
str(rc))
pass
This can be then initialised with:
subscriber.on_connect = on_connect
• Create a callback function for when the subscriber
receives a PUBLISH message from the broker:
def on_message(client, userdata, msg):
print(msg.topic+" "+str(msg.payload))
This can be then initialised with:
subscriber.on_message = on_message

11

M2M in the IoT

Things – MQTT Subscriber

• We can now establish the connection with the broker

by using the “connect” method:
subscriber.connect(<host>, <port>, <keepalive>)
• Replace:
• <host> with the hostname or IP address of the
broker (string)
• <port> with the TCP port to connect to, e.g. 8883
for the encrypted connection (integer)
• <keepalive>: maximum period in seconds
between communications with the broker (integer)

12

4
 11/5/19

M2M in the IoT

Things – MQTT Subscriber

• After the connection is established, it is possible to

subscribe to a topic with:
(rc, mid) = subscriber.subscribe(<topic>,
<qos>)
• Replace:
• <topic> with the topic from where to subscribe
for receiving the message from the broker (string)
• <qos> with the desired Quality of Service level for
the subscription (integer)
• This returns the result code (rc) and the message ID
(mid). If rc is equal to 0 then the message has been
successfully published, otherwise an error happened

13

M2M in the IoT

Things – MQTT Subscriber

• It is possible also to subscribe to multiple topics at

the same time by:
(rc, mid) = subscriber.subscribe(
[(<topic1>, <qos1>), (<topic2>, <qos2>)])
• This requires to provide a list of tuples, where each
tuple contains:
• <topicN> with the N topic from where to
subscribe for receiving the message from the
broker (string)
• <qosN> with the associated desired Quality of
Service level for the subscription (integer)

14

M2M in the IoT

Things – MQTT Subscriber
• System information about the broker can be retrieved
from by using the MQTT $SYS topic:
• Retrieve all available information:
subscriber.subscribe("$SYS/#")
• $SYS/broker/messages/received: retrieve the total
number of messages received
• $SYS/broker/messages/sent: retrieve the total number
of messages sent
• $SYS/broker/clients/connected: retrieve the number of
clients connected
• $SYS/broker/version: version of the broker

https://github.com/mqtt/mqtt.github.io/wiki/SYS-Topics

15

5
 11/5/19

M2M in the IoT

Things – MQTT Subscriber

• The method “loop_forever” can be used to

continuously receive messages published by the
publisher:
subscriber.loop_forever()
• Before closing the connection, a callback can be
used to retrieve the result code:
def on_disconnect(client, userdata, rc):
print("Disconnected with result code: "+
str(rc))
• Finally, the connection can be closed by using the
“on_disconnect” variable:
subscriber.on_disconnect = on_disconnect

16

M2M in the IoT

Things – MQTT Subscriber
import paho.mqtt.client as mqtt
def on_connect(client, userdata, flags, rc):
print("Connected with result code " + str(rc))
pass
def on_message(client, userdata, msg):
print(msg.topic + " " + str(msg.payload))
subscriber = mqtt.Client("Actuator")
subscriber.username_pw_set(username="pi", password="pwd")
subscriber.on_connect = on_connect
subscriber.on_message = on_message
subscriber.connect("mqtt.eclipse.org", 8883, 60)
(ret, mid) = subscriber.subscribe("sensor/temp", 1)
print("Subscribed with return code: " + str(rc))
subscriber.loop_forever()

17

M2M in the IoT

Abstraction – MQTT solution
MQTT broker
M2M Network The
domain Mosquitto
Internet
Ethernet

Gateway

M2M Device
and Gateway Local
domain Network

MQTT publisher MQTT subscriber

Paho-MQTT Paho-MQTT

Wi-Fi Wi-Fi

Sensor (DHT11) Actuator (LED)

18

6
 11/5/19

M2M in the IoT

Things – MQTT Publisher
import paho.mqtt.client as mqtt MQTT publisher

def on_publish(client, userdata, mid): Paho-MQTT

print("Message ID: " + str(mid)) Wi-Fi

pass Sensor (DHT11)
def on_disconnect(client, userdata, rc):
print("Disconnected with result code: " + str(rc))
publisher = mqtt.Client("Sensor")
publisher.username_pw_set(username="pi", password="pwd")
publisher.on_publish = on_publish
publisher.connect("mqtt.eclipse.org", 8883, 60)
(rc, mid) = publisher.publish("sensor/temp", "10")
print("Published with return code: " + str(rc))
publisher.on_disconnect = on_disconnect
publisher.disconnect()

19

M2M in the IoT

Things – MQTT Publisher

The data from the sensor can be obtained by:

import RPi.GPIO as GPIO Import the relevant modules
import dht11
Initialise GPIOs
GPIO.setwarnings(False)
GPIO.setmode(GPIO.BCM) Initialise the
DHT11 sensor
pin=23
module
dht11_sensor_module = dht11.DHT11(pin)
data = dht11_sensor_module.read() Read data
if data.is_valid():
Validate the data
temp = data.temperature # Celsius
humidity = data.humidity # % Retrieve the data
GPIO.cleanup()
Cleanup the GPIO

20

M2M in the IoT

Message formatting – XML

The data contained in the message sent by the

publisher is not formatted!
A better way to format a message is to use
eXtensible Markup Language (XML):
• Language used to specify structured contents for
automated processing
• Simple and flexible text format used for data
representation and annotation
• Use user-defined tags to structure the content

21

7
 11/5/19

M2M in the IoT

Message formatting – XML

Remember that a well formed XML document should

have a correct syntax:
• XML declaration, which begins with <?xml>
• XML root element
• XML elements, which begins with < and end with >
• XML elements must have a closing tag, which begins with
</ and end with >
• XML elements must be properly nested
• XML attribute must be in the form name="value”
• XML attribute values must be quoted with double (" ") or
single (' ') quotes
• XML comments must begins with <!-- and end with -->

22

M2M in the IoT

Message formatting – XML

The information that can be collected from the

system is:
• temp: temperature value in Celsius
data.temperature MQTT publisher
• humidity: humidity value in % Paho-MQTT
data.humidity Wi-Fi
• date: current date Sensor (DHT11)
str(dt.date.today())
• time: current time
str(dt.datetime.time(dt.datetime.now())

23

M2M in the IoT

Message formatting – XML
The information can then be formatted by using XML:
Declaration
<?xml version="1.0" encoding="UTF8"?>
<sensor_data> Root element

<measurement> Child element

<temperature type="double">temp</temperature>
<temperature_unit>"C"</temperature_unit>
<humidity type="double">humidity</humidity>
<humidity_unit>"%"</humidity_unit>
<date>date</date>
<time>time</time>
Subchild elements
<TZ>"BST"</TZ>
</measurement>
</sensor_data>

24

8
 11/5/19

M2M in the IoT

Message formatting – XML

The XML module can be used to format the data by

importing the “ElementTree” module:
import xml.etree.ElementTree as ET
• The first step to create the XML structure is to use the
“Element” object for defining the root element: tag

root_element = ET.Element('sensor_data')
• Other elements can be added to the structure by using the
“SubElement” object:
child_element =
ET.SubElement(root_element, 'measurement')

parent tag

25

M2M in the IoT

Message formatting – XML

• A subchild can also be added by using the

“SubElement” object along with other siblings (children
on the same level):
subchild_element = parent

ET.SubElement(child_element,
tag 'temperature')
• Attributes can be added by using the “set” method:
subchild_element.set('type', 'double')
• The text can be added by using the “text” variable:
subchild_element.text = temp

data.temperature

26

M2M in the IoT

Message formatting – XML

• The XML structure can be converted to a string for being

included in the MQTT message by using the
“tostring” method: root element
xml_string =
ET.tostring(root_element, 'UTF8')
• Finally it can be used as a message: encoding
(rc, mid) =
publisher.publish("sensor/information",
xml_string)

27

9
 11/5/19

M2M in the IoT

Things – MQTT Subscriber
import paho.mqtt.client as mqtt
def on_connect(client, userdata, flags, rc):
print("Connected with result code " + str(rc))
pass
def on_message(client, userdata, msg):
print(msg.topic + " " + str(msg.payload))
subscriber = mqtt.Client("Actuator")
subscriber.username_pw_set(username="pi", password="pwd")
subscriber.on_connect = on_connect
subscriber.on_message = on_message
subscriber.connect("mqtt.eclipse.org", 8883, 60)
(ret, mid) = subscriber.subscribe("sensor/temp", 1)
print("Subscribed with return code: " + str(rc))
subscriber.loop_forever()

28

M2M in the IoT

Message formatting – XML

The XML module cannot be used to parse the data

received by the subscriber
• This does not provide secure mechanisms against
maliciously constructed data
The “defusedxml” module can be used instead by
importing the “ElementTree” module:
import defusedxml.ElementTree as ET
• A function can be defined in the “on_message” callback to
parse the received XML structure for the given topic:
if msg.topic == "sensor/information":
parse_xml(msg)

29

M2M in the IoT

Message formatting – XML
def parse_xml(msg):
# Initialise the “temp” and “humidity” variables
temp = None
humidity = None
# Extract the root element (sensor_data) from the
# payload by decoding it
root = ET.fromstring(msg.payload.decode('utf-8’))
# Extract each child element (measurement) from the
# root element
for child in root:
# Extract each subchild element from the
# child element
for subchild in child:
# Retrieve the sensor values
if "temperature" == subchild.tag:
temp = subchild.text
if "humidity" == subchild.tag:
humidity = subchild.text
# Function that makes the decision
actuator(temp, humidity)

30

10
 11/5/19

M2M in the IoT

Abstraction – MQTT solution
MQTT broker
M2M Network The
domain Mosquitto
Internet
Ethernet

Gateway

M2M Device
and Gateway Local
domain Network Is this the
best
approach?
MQTT publisher MQTT subscriber
Paho-MQTT Paho-MQTT

Wi-Fi Wi-Fi
Sensor (DHT11) Actuator (LED)

31

M2M in the IoT

Big Data – Computing

Global
Th

Cloud
ou
sa

Country/
nd

regional
s

Local/
metro
Fog
Mi
llio
ns

Premise/
site
Bi

Edge
llio
ns

Device

32

M2M in the IoT

Big Data – Computing
Cloud: large Data Centres with global applicability
and reach
Fog: intermediate computational layer that
decentralise processes and make local decisions.
Nodes aim to:
• Collect and process data, and issue control
commands to the actuators
• Filter data that needs to be used locally and send
the rest to the Cloud
Edge: computational capabilities are introduced in
the smart devices with network access to process
data and forward only relevant information

33

11
 11/5/19

M2M in the IoT

Abstraction – MQTT solution
Cloud
M2M Network The
domain M2M app
Internet
Ethernet

MQTT broker
Mosquitto
M2M Device Wi-Fi
and Gateway
domain
Local
Network

MQTT publisher MQTT subscriber

Paho-MQTT Paho-MQTT

Wi-Fi Wi-Fi
Sensor (DHT11) Actuator (LED)

34

M2M in the IoT

Abstraction – MQTT solution
MQTT broker
M2M Network The
domain Mosquitto
Internet
Ethernet

Gateway Gateway

M2M Device
Local Local
and Gateway
Network Network Is this the
domain
best
approach?
MQTT publisher MQTT subscriber
Paho-MQTT Paho-MQTT

Wi-Fi Wi-Fi

Sensor (DHT11) Actuator (LED)

35

Summary
M2M in the IoT

MQTT with Python

Messages formatting and parsing with XML

MQTT solutions for IoT

36

12
 11/5/19

Next class?

Thursday at 10 a.m. in the

Electrical Engineering &
Electronics Building, Room E4
(ELEC-204, Building 235).

37

13