Mobile Networks and Applications

https://doi.org/10.1007/s11036-020-01620-5

The Implementation of a Cloud-Edge Computing Architecture

Using OpenStack and Kubernetes for Air Quality Monitoring
Application
Endah Kristiani 1,2 & Chao-Tung Yang 3 & Chin-Yin Huang 1 & Yuan-Ting Wang 4 & Po-Cheng Ko 3

# Springer Science+Business Media, LLC, part of Springer Nature 2020

Abstract
The combination of edge and cloud computing is going to make the Internet of Things (IoT) rapid, light, and more
reliable. IoT and cloud-edge computing are distinct disciples that have evolved separately over time. However, they
are increasingly becoming interdependent, and are what the future holds. A crucial aspect is how to design a
compound of cloud and edge computing architectures, and implement IoT effectively. In this paper, we proposed
a combination of Cloud and Edge Computing architecture and built a set of an intelligent air-quality monitoring
system in Tunghai University as a case study. In this case, we implemented container-based virtualization which
constructs Kubernetes Minion (Nodes) in the Docker container service independently for each service on the Edge
side. Finally, to monitor the high-performance computing systems, clusters, and networks, we used Ganglia
Monitoring System. Ganglia collects relevant information such as Central Processing Unit (CPU), memory, network
and usage of Protocol Data Unit (PDU) to monitor the power consumption and makes a measurement and evaluation
for Kubernetes Pods.

Keywords Internet of things . Cloud computing . Edge computing . OpenStack . Microservices . Kubernetes . Container-based
virtualization

1 Introduction government, enterprise and personal are getting some

Cloud computing has a massive change in industry de-
velopment after the Internet. Not only the Information
Technology (IT) industry which provides cloud comput-
ing technology but also the general using way of the
change with the born of cloud computing. In the IT
industry, cloud computing is undoubtedly caused a sig-
nificant impact. From the most basic computer compo-
nents †“processors, servers, storage devices, network
equipment, information security equipment, software,

* Chao-Tung Yang 1
Department of Industrial Engineering and Enterprise Information,
ctyang@thu.edu.tw Tunghai University, Taichung 40704, Taiwan, R.O.C.

Endah Kristiani 2
Department of Informatics, Krida Wacana Christian University,
endahkristi@gmail.com Jakarta 11470, Indonesia
Chin-Yin Huang
3
huangcy@thu.edu.tw Department of Computer Science, Tunghai University,
Taichung 40704, Taiwan, R.O.C.
Yuan-Ting Wang
yttom@cht.com.tw 4
Cloud Computing Laboratory, Chunghwa Telecom Laboratories,
Po-Cheng Ko Taoyuan City 326, Taiwan, R.O.C.
pocheng0605@gmail.com

data centers, information services, even though
smartphone, tablet computer, and other emerging mobile
devices are unable to break off relations from cloud
computing. Along with cloud computing development,
Internet of Things (IoT) industry has grown substantial-
ly. So that, it triggers an exponentially increasing
amount of data which need to improve data analysis
process [1]. While cloud computing is essential to the
success of IoT, and pure cloud computing itself not
quite adequate for faster data analysis. The long dis-
tance of the network logic between the cloud and the
end device might easily cause Network delay, thereby
affecting the system can not respond promptly. In this
case, the concept of edge computing [2, 3] meets the
need for faster data analysis. Edge computing is a form
of cloud computing, which pushes data processing out
to the edge device to handle. This mechanism transports
only the results of the data processing over networks.
Therefore, it provides accurate results and consumes
more lightweight network bandwidth than independent
cloud computing. Moreover, Docker and Kubernetes
when combined together provide best solutions by ef-
fectively organizing containerized applications.
In this paper, we integrated open source software to
implement a complete set of edge computing architec-
tures. Edge computing between IoT devices and the
cloud establishes a relay station to store the data col-
lected by the sensors and provide the most immediate
preprocessing and response [4]. The campus air-quality
monitoring system mainly distributes Arduino sensors
on campus [5, 6]. Through LoRa, Low Power Wide
Area Network (LPWAN), LoRa uses star topology to
allow All nodes to send data to LoRa Gateway, and
then upload to the cloud. In the cloud, OpenStack main-
ly built as Infrastructure as a Service (IaaS) platform.
Through virtualization technology, OpenStack can estab-
lish a cluster to provide various virtualized services
such as storage Virtualization, network virtualization,
CPU virtualization and so on [7]. To maximize the uti-
lization of hardware resources, we set up the control
node (Kubernetes Master) on OpenStack as a central
service control, and use Ceph Storage as a complete
system data backup mechanism [8, 9]. On the Edge
side, we set up the Raspberry Pi as the Edge Gateway
and Kubernetes minion on the Raspberry Pi [10] to
provide the service application, which contains MySQL
database [11] for data storage, and the MQTT Broker
for exception notification [12].
The second phase of the platform is a benchmark test
to verify Kubernetes in creating containers, memory con-
sumption, CPU performance, and power consumption
[13]. At this point, edge computing can be the best so-
lution and the efficiency of the Monitoring Application.
Mobile Netw Appl
In this paper, we started from the network virtualization
and added software-defined networking (SDN) Controller
[14] and OpenDaylight to Neutron network architecture
of OpenStack. Differences in network performance im-
provement and comparison with the original Neutron ar-
chitecture will ultimately provide the best open source
architecture for edge computing and IoT applications
[15].
In particular, this work aims to propose an architecture of
cloud-edge computing combined with IoT for air quality mon-
itoring. Specific purposes are the following:
& Deploy a complete set of cloud-edge computing architec-
tures and IoT using container-based virtualization which
constructs Kubernetes Minion (Nodes) in the Docker con-
tainer service independently for each service on the Edge
side.
& Evaluate and monitor the high-performance computing
systems, clusters, and networks using Ganglia
Monitoring System.
The rest of the paper is organized as follows. Section 2
describes the background and related work. Section 3 presents
the experimental environment and system architecture.
Section 4 shows the preliminary results and discussion. The
last one, Section 5 provides a conclusion and the future work
of this paper.
2 Background review and related works
In this section, we provide several components that are ap-
proaching in this paper: Cloud Computing, Edge Computing,
OpenStack, Kubernetes, and so on. The next sections discuss
each component in more detail.
2.1 Cloud computing and edge computing
2.1.1 Cloud computing
Cloud computing is an information technology (IT) paradigm
that enables universal access to shared pools of configurable
system resources and higher-level services that can be rapidly
provisioned with minimal management effort, often over the
Internet. Cloud computing relies on sharing of resources to
achieve coherence and economy of scale, similar to a utility
[16].
Third-party clouds enable organizations to focus on their
core businesses instead of expending resources on computer
infrastructure and maintenance. Advocates note that cloud
computing allows companies to avoid or minimize up-front
IT infrastructure costs. Proponents also claim that cloud com-
puting will enable enterprises to get their applications up and
Mobile Netw Appl
to run faster, with improved manageability and less mainte-
nance, and that it allows IT teams to more rapidly adjust re-
sources to meet fluctuating and unpredictable business de-
mand. Cloud providers typically use a pay-as-you-go model,
which can lead to unexpected operating expenses if adminis-
trators are not familiarized with cloud-pricing models.
Since the launch of Amazon EC2 in 2006, the availability
of high-capacity networks, low-cost computers and storage
devices as well as the widespread adoption of hardware
virtualization, service-oriented architecture, and autonomic
and utility computing has led to growth in cloud computing.
2.1.2 Edge computing
The rapid development of Machine Learning and Big Data has
led to a lot of Model or Pattern finding based on Data Science
saved in Cloud Computing or data centers. Although Cloud
Computing can get the result of data analysis, Cloud
Computing Major data centers or on a fixed geographical
location, such as people in Taiwan, data analysis but to the
United States, the inevitable delay of the network still exists.
Several of Cloud Computing’s leading vendors, including
Microsoft, Amazon, and Google, have come up with the con-
cept of edge computing. Edge Computing, also known as Fog
Computing, refers to moving data processing and computa-
tion closer to data sources on a logical network to shorten the
delay of network transmission and obtain faster data Analysis,
or faster access to feedback.
Edge computing is a method of optimizing cloud comput-
ing systems by performing data processing at the edge of the
network, near the source of the data. IoT applications are
ideally suited for edge computing architectures, especially in
emerging IoT applications such as self-driving, drone, aug-
mented reality (AR) / virtual reality (VR), and robotics.
Edge computing can be performed in a large computing
device or a local network of many small and medium-sized
computing devices or devices because it is operated in a net-
work environment close to the data source where the network
is located. These computing devices can be a personal device
(such as a smartphone or wearable device), an end device
(such as a gateway, surveillance camera, bank ATM), or other
IoT devices. As the computing device and device close to each
other, the network transmission more direct, such as through
Table 1 Comparison between
Cloud computing and Edge Cloud computing
computing
Complex analytics
Big Data mining
Sources of business logic
Machine learning rules
Advanced visualizations
Long term data storage/warehousing
Wi-Fi, Bluetooth, etc., so delivery of information becomes
fast [17, 18]. Table 1 compare the different of Cloud and
Edge Computing.
2.2 OpenStack
OpenStack [19] is a free and open-source software platform
for cloud computing, mostly deployed as infrastructure-as-a-
service (IaaS), whereby virtual servers and other resources are
made available to customers. The software platform consists
of interrelated components that control diverse, multi-vendor
hardware pools of processing, storage, and networking re-
sources throughout a data center. Users either manage it
through a web-based dashboard, through command-line tools
or through RESTful web services [20]. OpenStack began in
2010 as the project of Rackspace Hosting and NASA. As of
2016, it is managed by the OpenStack Foundation, a non-
profit corporate entity established in September 2012 to pro-
mote OpenStack software and its community. More than 500
companies have joined the project [21].
2.3 Docker
Docker is an open platform for developing, shipping, and
running applications. Docker enables us to separate our appli-
cations from our infrastructure so we can deliver software
quickly. With Docker, we can manage our infrastructure in
the same ways we manage our applications. By taking advan-
tage of Docker’s methodologies for shipping, testing, and
deploying code quickly, we can significantly reduce the delay
between writing code and run it in production. Docker Engine
is a client-server application with these major components:
& A server which is a type of long-running program called a
daemon process (the docker command).
& A REST API which specifies interfaces that programs can
use to talk to the daemon and instruct it what to do.
& A command line interface (CLI) client (the docker
command).
& Templates can also specify the relationships between re-
sources (e.g., this volume is connected to this server). The
template enables Heat to call out to the OpenStack APIs to
Edge computing
Basic data visualization
Basic data analytics and short term data historian features
Datacaching, buffering and streaming
Data pre-processing, cleansing, filtering and optimization
Some data aggregation
Device to Device communications/M2M

Fig. 1 System architecture
create all of our infrastructures in the correct order to
launch our application ultimately.
& Heat manages the whole lifecycle of the application -
when we need to change our infrastructure, modify the
template and use it to update our existing stack. Heat
knows how to make the necessary changes. It will delete
all of the resources when we are finished with the appli-
cation, too.
& Heat primarily manages infrastructure, but the templates
integrate well with software configuration management
tools such as Puppet and Chef. The Heat team is working
on providing even better integration between
2.4 Microservices
Microservices is a software architecture style based on Small
Building Blocks that focus on a single responsibility and func-
tion, using a modular approach to assemble complex, large-
scale applications. Each of the functional areas Blocks com-
municate with each other using the Language-Independent /
Fig. 2 Device architecture
Mobile Netw Appl
Language agnostic APIs. One of the concepts that
microservice architecture applies to software architectural
style is Dew Computing, which refers to the computing power
that comes together from much small dew (the functional
elements that represent microservices).
2.5 Kubernetes
Kubernetes is a powerful system, developed by Google, for
managing containerized applications in a clustered environ-
ment [22]. It aims to provide better ways of managing related,
distributed components across the varied infrastructure.
Kubernetes is a system for managing containerized applica-
tions across a cluster of nodes.
When developing maintenance, the best hope is to monitor
it continuously after deployment. When any server is
overloaded, it is expected to expand automatically. Servers
will be automatically reduced when traffic is sluggish.
Besides, the most important thing is that when a server hangs
point, we hope it can be automatically repaired, and at any
time must maintain the number of service instances. Many of
Mobile Netw Appl
Fig. 3 Edge Architecture
the above functions Docker can help we do, and use the con-
tainer to provide services. This type of service is called CaaS
(Container as a Service), and it looks like everything is fine.
But Docker containers are not a panacea. How to manage the
life cycle of clusters and services in the ever-increasing num-
ber of containers/services will be the orchestration capability
of each container. One of Google’s Kubernetes is a very so-
phisticated container layout management tools [22].
Kubernetes is particularly well-suited for microservices
such architectures. Combining several containers into a
single service, Kubernetes also provides a good service
discovery mechanism for each service to communicate
with one another. Most importantly K8S powerful pro-
gramming can automatically expand services, and even
for large-scale containers for rolling updates (Rolling
update) and rollback (Rolling back / Undo), but also
can integrate CI / CD and other DevOps tools,
Absolutely allow users to manage the largest system
with the least effort [23].
Fig. 4 Data Flow (Device and
Edge)
Kubernetes is a distributed system; the main compo-
nents are: Master as the master, can be used as the
master node. Node is a worker, which runs many con-
tainers. May want to make a virtual machine. K8S can
handle up to 1000 nodes or more, Kubernetes uses mas-
ters and nodes to deploy cluster. Master contains three
basic components, Etcd, API Server, Controller Manager
Server. A node consists of four basic components
Kubelet, Proxy, Pod, Container.
2.6 MQTT
MQTT is a protocol developed in 1999 by Dr. Andy
Stanford-Clark, IBM, and Dr. Arlen Nipper, Arcom
(Eurotech). MQTT stands for Message Queuing
Telemetry Transport [24], a set of dedicated push-to-
talk systems for Machine-to-Machine (M2M) systems
that can be imagined as the Internet for IoT machine
communication Socket or Line. Because the content of

Fig. 5 Data Flow (Cloud)
the MQTT protocol is streamlined, it is ideal for IoT
devices with limited processor resources and network
bandwidth.
The underlying MQTT and HTTP are TCP / IP, that
is, IoT devices can use existing network infrastructure
and equipment, but the message format that flows over
the Internet and the application processing mechanisms
are different. HTTP request instructions have too much
header information, which is redundant for IoT applica-
tions and consumes network bandwidth. MQTT is the
same as HTTP header description, MQTT header digital
encoding, the entire length of only two bytes, equivalent
to two characters, followed by the message of the topic
and content(payload).
2.7 Related works
The Cisco introduced Fog Computing Systems as a new mod-
el to ease wireless data transfer to distributed devices in the
Internet of Things (IoT) network paradigm. Cisco defines Fog
Computing as a paradigm that extends Cloud computing and
services to the edge of the network. Similar to Cloud, Fog
Fig. 6 Lora Node
Mobile Netw Appl
provides data, compute, storage, and application services to
end-users. The distinguishing Fog characteristics are its prox-
imity to end-users, its dense geographical distribution, and its
support for mobility. Services are hosted at the network edge
or even end devices such as set-top-boxes or access points. By
doing so, Fog reduces service latency, and improves QoS,
resulting in superior user-experience. Fog Computing sup-
ports the emerging Internet of Everything (IoE) applications
that demand real-time/predictable latency (industrial automa-
tion, transportation, networks of sensors and actuators).
Thanks to its wide geographical distribution the Fog paradigm
is well positioned for real-time big data and real-time analyt-
ics. Fog supports densely distributed data collection points,
hence adding a fourth axis to the often mentioned Big Data
dimensions (volume, variety, and velocity) [25].
Tsai et al. propose a fog computing platform for distributed
analytics [26]. This platform integrates resources from the
data center to end devices. Their study leverage the devices
to run IoT and multimedia applications and take different sen-
sor data, including a camera, microphone, and so on. In this
paper, they focus on implementing a platform, which supports
complicated analytics, such as Deep Learning to avoid
Fig. 7 Lora Gateway
Mobile Netw Appl
Fig. 8 Lora Network Setting
sending a large amount of raw data to powerful data centers
for analyzing. They adopt three known open-source projects,
including TensorFlow, Docker, and Kubernetes to implement
a fog computing platform. They conduct extensive measure-
ment and quantify performance.
Aqsa Malik et al. evaluated the ability of four OpenStack
Neutron plugins, OpenDaylight, OFAgent, ML2, and Ryu, to
continue to provide useful services in the face of network
perturbance on the management network. This analysis was
performed using OpenStack’s tenth release, named Juno.
They devised two different experiments. The first experiment
evaluated performance regarding spawning time of batches of
Fig. 9 Data visualization on ThingSpeak
VMs of different sizes. Results showed that OpenDaylight is
consistently the fastest of all other plugins under test under all
tested network conditions [27].
3 System design and implementation
In this section, we present our system and implementation in
two stages. First, we design the cloud-edge computing archi-
tecture and the monitoring system performance. Second, we
deploy cloud-edge computing on the campus as our case
study.
 Mobile Netw Appl

Fig. 10 Raspbian Jessie

3.1 System design 3.1.1 System architecture

In this section, we present system architecture, device archi- In this work, we establish an edge computing architec-
tecture, edge architecture, cloud architecture, and system mon- ture for IoT applications [28]. The main system is di-
itoring and evaluation tool. vided into three parts: device side, cloud side and edge

Fig. 11 MySQL
Mobile Netw Appl
Fig. 12 Docker version
ramp (as shown in Fig. 1), the device side is the sensor
and alarm side, can accommodate many different IoT
devices and transport agreements. Part of the cloud for
the virtual machine and Kubernetes master side, but also
deal with data backup, complex operations, and data
visualization and other applications do not need to re-
spond quickly. Finally, the part of the ramp is mainly
responsible for running services and applications, such
as data reception and exception notification. Data recep-
tion is pre-processed via the edge ramp, which greatly
reduces the amount of cloud transmission and storage
load. All services run on the Docker container for real-
izing the microservices architecture. Besides, informa-
tion is also stored on the edge gateway, such as the
cloud storage of unexpected conditions, but also through
the edge of the ramp data to stabilize service operation.
Fig. 13 Kubernetes cluster
3.1.2 Device architecture
Devices such as sensors, Raspberry Pi, Arduino, Linkit
One and so on. The platform also applies to different
transmission protocols such as ZigBee, MQTT, WiFi,
NB-IoT, Bluetooth and so on. The overall architecture
is shown in Fig. 2, mainly using Arduino LoRa Shield
module and Pms5003t sensor as the overall sensing
module and LoRa sensing data collected by LoRa
Nodes will be transmitted to LoRa Gateway through
LoRa. In this process, LoRa offers the Low Power
Wide Area Network (LPWAN) and the star-topology,
and the LoRa Gateway is also connected to the minia-
ture ramps made by the Raspberry Pi 3 for final unified
delivery to the data center(cloud).
3.1.3 Edge architecture
In the Edge section, the main design is the Instant
Provision of Services. The overall environment is set up
on the Raspberry Pi, which is a Linux-based PC that dif-
ferentiates itself from other IoT-based control chips. On
the Linux operating system, we can create the required
software as needed. This article first set up Kubernetes
Minion to provide services and applications, so the ser-
vices are all deployed in Docker containers [10]. The con-
tainerized deployment brings Lightweight, fast and easy to
migrate. Kubernetes treats multiple containers as one pod.
Whereas a single pod contains multiple containers.
Running the services, they perform simultaneously, pro-
viding a copy mechanism to allow the container to run a
specified number of new containers if it is corrupted; If
there are multi-run container will be automatically deleted,
saving the complexity of Victoria transport, significantly
improve service stability, known as HA (High Available).
 Mobile Netw Appl

Fig. 14 MQTT Push Notification

The existence of a copy mechanism also brings another key
advantage, called the rolling update mechanism (Rolling up-
date). Under the condition of running multiple containers, if
we need to replace the version of the application, we can issue
a rolling update instruction. Kubernetes replaces the container
versions one by one, allowing the service to continue running
without downtime. Kubernetes brings the effect of clustering.
We can build multiple Raspberry Pi as Kubernetes minions.
The clustered architecture can provide load balancing.
Kubernetes automatically uses the node with less resource
utilization as the target host for pod startup. Find the best
trade-off between high usage and high load. At the edge, the
first services are receiving data from LoRa, transferring data to
the cloud, MySQL RDB as data storage and MQTT Broker as
an alert push system. The overall architecture is shown in
Fig. 3.
The Fig. 4 below shows the system information flow of
equipment end and edge end. With this Unified Data
Modeling (UML) system, we can show the application flow
and data flow of each stage.
OpenStack environment, only need to face a single storage
system. Moreover, the Cloud side mainly deals with complex
operations [31]. We built Apache Spark to provide In-memory
decentralized computing [32–34], and finally visualize the
r e s u l t s o f t h e op e r a t i o n th r o u g h t h e E L K S t a c k
(Elasticsearch, Logstash, and Kibana) [35]. The figure below
is the operation process after entering the cloud [11]. We can
follow Fig. 5 to understand the data processing method in the
cloud.
3.1.5 Monitoring and evaluation tool
We developed a set of cloud-edge and IoT architecture using
container-based virtualization on the edge side. From this ar-
chitecture, we can measure and analyze:
1. Booting Time of Kubernetes Pods
2. Cluster Load (loads/procs)

3.1.4 Cloud architecture

On the Cloud side, it mainly builds OpenStack cluster infra-

structure services and provides the servers needed by the
Internet of Things through virtualization technologies.
OpenStack also provides network virtualization through the
neutron suite. This dissertation integrates with OpenDaylight
as the SDN Controller used by OpenStack, OpenDaylight
communicates with the Neutron Server via the northbound
interface Neutron Bound. ODL leverages OpenFlow and
OVSDB to manage Open vSwitches for OpenStack compute
nodes, providing L2, L3, LoadBalanceService, and
FirewallService. With Ceph Storage on storage, Ceph pro-
vides decentralized storage and supports both Block Storage
and File Storage, which replaces OpenStack’s Cinder and
Swift [29], as well as storing images Files and some comput-
ing resources. We deploy Ceph [30], and users in the entire Fig. 15 OpenStack login
Mobile Netw Appl

Fig. 16 OpenStack Overview

Fig. 17 OpenStack VM
 Mobile Netw Appl

Fig. 18 OpenStack network topology(1)

3. The CPU and Memory status of Cluster & Buildings: There are many buildings on the campus, and
4. The Network I/O of Cluster the shielding of buildings will cause the interference of
5. The power consumption of 64 Booting Pods data transmission signals
6. The CPU Utilization of 64 Booting Pods & Distance: The campus covers an area of 1,333,096 square
7. The Memory Utilization of 64 Booting Pods meters, in such a large application environment, resulting
8. The Network Utilization of 64 Booting Pods in transmission distance restrictions
& Power Consumption: The general transmission mecha-
nism plus the sensor needs to consume a certain amount
3.2 System implementation of power. However, it is not necessary to have a power
supply system at the place where the display is installed. If
The air-quality monitoring system made on the campus. There the battery is powered, the new battery needs to be re-
are many aspects to consider in terms of the sensors. placed frequently.

Fig. 19 OpenStack network topology(2)

Mobile Netw Appl
Fig. 20 OpenStack VNC Console
Under such conditions, we have chosen LoRa as
shown in Fig. 6. LoRa has a long-distance and low-
power wireless network technology. Through this tech-
nology, we can build Low Power Wide Area (LPWA),
low power consumption features make IoT applications
have excellent endurance, and long-distance advantages
of a single gateway or base station can cover a whole
city or hundreds of square kilometers of area. In the
future will be able to better scalability of the system
[36]. The hardware assembly, LoRa Shield and
Arduino Uno development board integration, and
Fig. 21 OpenDaylight UI
connect four in one sensor. We tested the transmission
distance and stability, and calibration of the sensor fi-
nally, hardware module will facilitate the rapid deploy-
ment of the future. To prevent moisture, we used the
3D printer to make a waterproof shell.
LoRa Gateway in Fig. 7 has a total of three antennas, de-
signed as two LoRa Shield and a WiFi receiver module. Two
sets of LoRa Shield provide dual-band reception, can bring
more stable signal reception. The WiFi module assembled on
Arduino, which can provide WiFi data transmission to Edge
and Cloud side.

Fig. 22 OpenDaylight Monitor
We set the transmission rate, ramp selection, and network
offload selection in LoRa Gateway as the network channel of
all the nodes. The following figure shows the setting interface
after setting up the LoRa Gateway, with the transmission fre-
quency, transmission protocol, transmission rate, and so on,
Then provide the system Log information, by observing the
data in the Log can confirm the current state of the machine,
but also to know whether Node has a stable connection and
transmission. The Lora Network setting can be described in
Fig. 8.
All LoRa Nodes transmit data to the cloud via
Gateway. Through the integration with Arduino, LoRa
will integrate the temperature and humidity and PM2.5
sensor module into the LoRa Node and send it regularly
to the Thingspeak IoT cloud database for further data
acquisition and visual analysis. Figure 9 describes ou
data visualization on ThingSpeak.
On Edge gateway, we first need to create a Raspberry Pi 3,
a Raspberry Pi 3 operating system, and Raspbian Jessie to
build the required kits and applications on this operating
Table 2 LoRa Node and LoRa
Gateway LoRa Node
LoRa Module: GlobalSat LM-130
Arduino Uno
PMS5003T G5T
Mobile Netw Appl
system. Figure 10 shows the setting environment of
Raspbian Jessie.
We built a MySQL relational database to serve as sensor
data storage for the Edge Gateway. The picture in Fig. 11
shows the data collected by four LoRa nodes, including tem-
perature, humidity, PM2.5, and PM10.
On the Edge Gateway, we set up a Kubernetes minion to
provide services. The service runs on Docker as shown in
Fig. 12, controlled by Kubernetes, and Kubernetes is deployed
as a clustered version in Fig. 13, including one master and two
minions.
Moreover, we create an MQTT Broker container, through
the Kubernetes HA mechanism, to provide stable anomaly
notification mechanism, Fig. 14 shows MQTT message push
test.
On the Cloud side, OpenStack clustering was set up to
provide infrastructure services through virtualization technol-
ogies. Nova computing services, Neutron network
virtualization, Keystone authentication management, Glance
image management, Cinder block storage services, Heat cloud
LoRa Gateway
LoRa Module: GlobalSat LM-230
WiFi: Dual Band 2.4GHz/5GHz 802.11 a/b/g/n
PMS5003T G5T,Ethernet: Gigabit Ethernet(RJ45)
Internal Storage: 4GB
CPU: ARM CORTEX A9 Dual Core (Std. 850 MHz)
Mobile Netw Appl

Table 3 Raspberry Pi 3
Item Description

OS Raspbian Jessie
CPU 4 Cores, Broadcom BCM2837 64bits ARMv8 Processor, 1.2GHz
Power 5 V@2.4A(with,MicroUSB)
Storage microSD
Network Ethernet (RJ45)
WiFi BCM43143 WiFi
Bluetooth BCM43438 wireless LAN and Bluetooth Low Energy (BLE) on board

deployment and Magnum Container Services, Horizon
Dashboard. Figure 15 shows the Horizon login interface for
establishing a secure authentication mechanism for the
OpenStack private cloud that we deployed.
Horizon provides cluster monitoring such as vCPU, RAM,
floating IP, number of virtual machines, cloud storage, secu-
rity groups, cloud drives, and so on. Figure 16 presents
OpenStack overview.
OpenStack can use the Horizon dashboard to facilitate the
OpenStack API’s operation of virtual machines with simple
settings. Figure 17 shows the state of the virtual machines and
virtual machines we set up, and we can also take further steps
such as Snapshot, Live Migration, and so on.
Network virtualization also provides network topology as
shown in Figs. 18 and 19 for users to efficiently operate virtual
networks, virtual subnets, and virtual routers.
We can use OpenStack boots instance (virtual machine)
through the web page directly into the machine’s Console
operation as shown in Fig. 20.
Our research also integrates OpenDaylight as the SDN
Controller used by OpenStack, OpenDaylight communicates
with Neutron Server through the northbound interface
Neutron Bound, and ODL manages the Open vSwitch of
OpenStack compute nodes by using OpenFlow and OVSDB
to provide L2, L3, LoadBalanceService and FirewallService.
Figure 21 shows OpenDaylight UI.
We can also check the current OpenStack machine com-
munication through the OpenDaylight UI as shown in Fig. 22.
For example, there are packets exchange and network com-
munication between the machines. This interface shows the
host location and IP.
4 Experimental results and analysis
In this section, first, we present the running of cloud-edge
computing architecture along with the implementation of air
quality monitoring using sensors. Second, we describe the
result of monitoring and analysis performance testing using
Ganglia Monitoring System.

4.1 Experimental environment

4.1.1 Hardware environment

At the device side, there are currently five

machines,LoRaGateway · 1 and LoRaNode · 4, which are cur-
rently deployed on our campus. LoRa node and LoRa
Gateway contains the main components as listed in Table 2:

Table 4 OpenStack Cluster

Host name CPU Memory Disk OS

Openstack Controller 12 cores 64GB 2 TB Ubuntu 16.04.02

Openstack Network 25 cores 64GB 2 TB Ubuntu 16.04.02
Openstack Compute01 64 cores 96GB 2 TB Ubuntu 16.04.02
Openstack Compute02 64 cores 96GB 2 TB Ubuntu 16.04.02
Openstack Block Storage 64 cores 48GB 8 TB Ubuntu 16.04.02
Fig. 23 Hardware environment

Table 5 LoRa Deployment
ID Location Type
A ITService in the First campus LoRa Node
B The main gate of the first campus LoRa Node
C The Luce Chapel LoRa Gateway
D ITService in the second campus LoRa Node
E Department of Life Science LoRa Node
Raspberry Pi 3 is set up on the Edge side as shown in
Table 3.
On the Cloud, OpenStack clustering was set up to provide
infrastructure services through virtualization technologies
[37]. In this study, five nodes were set up as Controller,
Compute01, Compute02, Block01 and Network respectively.
Services were established for Nova, Neutron, Keystone,
Glance, Cinder, Magnum and Horizon. Figure 23 and
Table 4 shows our hardware environment setting.
4.2 Experimental results
This study builds the system on campus with four LoRa nodes
and a LoRa Gateway in the following Table 5.
Fig. 24 The distance of sensors
Mobile Netw Appl
LoRa theoretical transmission distance of 15 to
20 km, and we deploy the farthest two nodes distance
of about 2 km. The LoRa transmission distance limit is
not exceeded. Figure 24 describes the distance of the
sensors implemented.
However, factors such as slope type and building masking,
LoRa Gateway in Fig. 25 is placed at the center of the four
sensors to provide the most stable data transmission. The data
is collected and visualized on the website.
Real-time data from individual LoRa nodes is sent to
MySQL on the Edge and displayed on the web, while a large
amount of historical data is backed up in the cloud, and we
also incorporate it into the page to provide visualization as
shown in Fig. 26.
At the edge, Kubernetes provides a great advantage. This
paper establishes a three-node Kubernetes, Master · 1 and
Node · 2 respectively, all of which are operating systems of
Ubuntu 16.04.1. Figure 27 shows our Kubernetes cluster
setting.
Through this command kubectl scale deploy nginx –rep-
licas = 64, the service will be automatically extended to run
the number of copies set, where the number of copies set to 65,
the results are as shown in Fig. 28.
In this result as shown in Fig. 29, we also found that
the number of pods on the master is 21, the number of
pods on node1 is 22, the number of pods on is 22. We
Mobile Netw Appl

Fig. 25 LoRa Deployment in Campus

Fig. 26 Data visualization

 Mobile Netw Appl

Fig. 27 Kubernetes Cluster

can confirm Kubernetes to Load Balance mechanism, we went a step further to test the Pod’s startup time. As
automatically pods evenly dispersed to each node, the the number of containers started increases with time, it
choice of lower utilization of nodes Open the pod. Also, is also found that as the number of starts increases, the

Fig. 28 auto-scaling.
Mobile Netw Appl
Fig. 29 Booting Time of Kubernetes Pods
slope of the polyline gradually becomes higher, indicat-
ing that too many containers are started, resulting in
delays in the communication and scheduling of the clus-
ter network.
Besides, this paper sets up Ganglia in the Kubernetes
cluster as shown in Fig. 30, a scalable distributed mon-
itoring system suitable for high-performance computing
systems such as cluster architecture. It uses the XDR
transport protocol for data storage and RRDtool for
Fig. 30 Ganglia Monitor
Table 6 Kubernetes Cluster
Hostname CPU
Kubernetes_master Intel(R) Core(TM) i7-6950X CPU @ 3.00GHz
Kubernetes_node01 Intel(R) Core(TM) i7-6950X CPU @ 3.00GHz
visualization. Ganglia uses very lightweight data struc-
tures and algorithms to achieve very low resource over-
head, making it ideal for Kuberentes cluster architecture.
In this experiment, we deployed two nodes Kubernetes
clusters in Table 6. We limit the number of Kubernetes nodes
to one. In this way, we can measure resource utilization of
Kubernetes node more directly.
We test the performance when Kubernetes boots the differ-
ent number of pods. The main testing items include CPU
usage, memory consumption, network usage. In addition, we
use Power Distribution Unit (PDU) to monitor the energy
consumption of servers during Pods are booted. At First, we
boot 1, 2, 4, 8, 16, 32 and 64 Pods to observe the utilization of
resources, each booting Pods experiments will delete all Pods
before next experiment. We can observe the cluster status in
Ganglia, as shown in Fig. 31.
In Fig. 32, we can see that as Pods are booted. The
load is a rise, but almost status is low. Moreover, the
usage utilization which is generally high is 13 s. It is
time to boot 16 pods. Therefore, we can know that boot
16 pods need to use more resources to load process in
the two nodes architecture.
Ram Disk
128G 2T
128G 2T

Fig. 31 Ganglia Status
There are no significant changes in CPU and Memory.
CPU resources are consumed when booting Pod, and more
notably CPU utilization starts to increase significantly when
17 Pods are booted, as shown in Fig. 33.
It can be seen in Fig. 34, as the number of booted Pods
increases, the network utilization (Bytes/s) also gradually in-
creases. The reason for this may be presumed that when the
cluster architecture starts to perform distribution actions, the
master and node also need to communicate with each other
through the API. With the reduction of the request, the utili-
zation will down again. Therefore, the result will bee like a
wave.
Besides monitoring the total resource consumption of the
cluster. In this paper, we observe the resource utilization of
Kubernetes node and add the energy consumption to observe.
Fig. 32 Cluster Load (loads/procs)
Mobile Netw Appl
We collect the data during booting 64 Pods. In the energy
consumption side, with 64 Pods of booting. Power consump-
tion is Up to about 71 watts. Figure 35 presents the graph of
power consumption of 64 Booting Pods.
When the node is booting 64 pods, CPU utilization is still
very low, the highest utilization rate of the power consump-
tion is only 2% and an average of 97.7% idle time. We can see
the feature of a lightweight container. It can be said that Nice
time is 0%. It means when Docker and Kubernetes are running
in Linux, and the CPU does not use the nice command to
adjust the execution priority. Figure 36 shows the graph of
CPU utilization of 64 Booting Pods.
In terms of memory, there is an upward trend when booting
Pods. It can be found that Kubernetes has the highest usage of
Cache, while Swap is not used. Cache brings fast processing
Kubernetes faster and high performance. Figures 37 and 38
shows the memory and network utilization of 64 booting
pods.
On the network side, when booting 64 pods, there is
an upward trend at a relatively high rate of 7629.24 kb
/ s. This trend is because Kubernetes needs machine-to-
machine communication when booting a pod to achieve
command requests and avoid conflicts. Kubernetes ar-
chitecture will deploy Etcd in the master. Etcd is a kind
of key-value distribution storage system. It is used for
storing Kubernetes cluster state and configuration. Etcd
is important to cluster state synchronization. Because all
nodes are from etcd to get other machine states in
Kubernetes nodes. Kubelet can monitor etcd to get
Mobile Netw Appl
Fig. 33 The CPU and Memory status of Cluster
Fig. 34 The Network I/O of
Cluster
information about the pod. If Kubelet finds a new Pod
that is bound to this node. It will create that pod as
required. Kubelet receives and executes instructions
from a master, managing pods and containers in pods.
Kubelet will register the node’s information on the API
Server, and periodically report the resource usage of the
Fig. 35 The power consumption of 64 Booting Pods
node to the master node. Finally, it will monitor the
resources of nodes and containers through cAdvisor.
5 Conclusions and future work
In this section, we stated the concluding remarks and the fu-
ture works of this study.
5.1 Concluding remarks
In this work, we integrate the technologies of
OpenStack, Kubernetes, Docker, and Ceph to implement
a complete set of edge computing architectures and im-
plements the air pollution monitoring system based on
the Internet of Things LoRa and MQTT technologies.
From this proposed architecture, we investigate how sig-
nificantly the architecture of Cloud-Edge computing and
IoT affect the performance of the services using Ganglia
Monitoring System. In this case, we implemented

Fig. 36 The CPU Utilization of
64 Booting Pods
container-based virtualization which constructs
Kubernetes Minion (Nodes) in the Docker container ser-
vice independently for each service on the Edge side.
In the booting process of 1, 2, 4, 8, 16, 32 and 64
Pods, the loading process is increasing, but in our ex-
periment almost of the status are low. The maximum
usage of utilization is 13 s. When boot 16 pods, we
can know that boot 16 pods need to use more resources
to load process in the two nodes architecture. There are
no significant changes in CPU and Memory. CPU re-
sources are consumed when booting Pod, and more no-
tably CPU utilization starts to increase significantly
when 17 Pods are booted as the number of booted
Pods increases, the network utilization (Bytes/s) also
gradually increases. With the reduction of the request,
the utilization will down again. Therefore, The result
will be like a wave.
In the energy consumption term, with 64 Pods of
booting. Power consumption is about 71 watts. When
Node is booting 64 pods, CPU utilization is still very
low, the highest utilization rate is only 2% and an av-
erage of 97.7% when idle time. In this point, we can
see the feature of a lightweight container. Regarding
Fig. 37 The Memory Utilization of 64 Booting Pods
Mobile Netw Appl
memory, there is an upward trend when booting Pods
process. It can be found that Kubernetes has the highest
usage of Cache, while Swap is not used. Cache brings
fast processing Kubernetes faster and high performance.
On the network side, when booting 64 pods, there is an
upward trend at a relatively high rate of 7629.24 kb/s.
5.2 Future work
In the future, open source software is bound to be the
focus of development, hoping to be based on the con-
cept of edge computing in this paper, closer integration
of Hadoop Ecosystem, Spark MLlib and Tensorflow,
clustering and decentralized computing architecture for
ineffective IoT devices, Bring better performance to
show. Moreover, hopefully integrating machine learning
and deep learning technologies can bring higher value
to IoT applications. Finally, we also hope that the effi-
ciency testing will make the power consumption model
suitable for Kubernetes, and find the best trade-off for
Kubernetes in different CPU, memory, Disk, network
and other resources for future Edge Computing architec-
ture applications.
Fig. 38 The Network Utilization of 64 Booting Pods
Mobile Netw Appl
Acknowledgments This work was supported in part by the Ministry of
Science and Technology, Taiwan (R.O.C.), under grants number 107-
2221-E-029-008 and 107-2218-E-029-003.
References
1. Yang C-T, Liu J-C, Chen S-T, Hsin-Wen L (2017) Implementation
of a big data accessing and processing platform for medical records
in cloud. Journal of medical systems 41(10):149
2. Varghese B, Buyya R (2018) Next generation cloud computing:
new trends and research directions. Futur Gener Comput Syst 79:
849–861
3. Deng D-J, Lien S-Y, Lin C-C, Hung S-C, Chen W-B (2017)
Latency control in software-defined mobile-edge vehicular net-
working. IEEE Commun Mag 55(8):87–93
4. Lien S-Y, Deng D-J, Tsai H-L, Lin Y-P, Chen K-C (2017)
Vehicular radio access to unlicensed spectrum. IEEE Wirel
Commun 24(6):46–54
5. Yuan X, Zhang M, Wang Q, Wang Y, Zuo J (2017) Evolution
analysis of environmental standards: effectiveness on air pollutant
emissions reduction. J Clean Prod 149:511–520
6. Chao-Tung Yang, Shuo-Tsung Chen, Chih-Hung Chang, Walter
Den, and Chia-Cheng Wu (2018). Implementation of an environ-
mental quality and harmful gases monitoring system in cloud.
Journal of Medical and Biological Engineering, pages 1–14
7. Kozhirbayev Z, Sinnott RO (2017) A performance comparison of
container-based technologies for the cloud. Future Generation
Computer Systems 68:175–182
8. Yang C-T, Chen C-J, Chen T-Y (2017) Implementation of ceph
storage with big data for performance comparison. Lecture Notes in
Electrical Engineering 424:625–633
9. A Mikula, D Adamová, M Adam, J Chudoba, and J Švec (2016).
Grid site monitoring and log processing using elk. CEUR
Workshop Proceedings
10. Endah Kristiani, Chao-Tung Yang, Yuan Ting Wang, and Chin-
Yin Huang (2018). Implementation of an edge computing architec-
ture using openstack and kubernetes. In International Conference
on Information Science and Applications, pages 675–685. Springer
11. Peng Shu, Rong Gu, Qianhao Dong, Chunfeng Yuan, and Yihua
Huang (2016). Accelerating big data applications on tiered storage
system with various eviction policies. In 2016 IEEE Trustcom/
BigDataSE/ISPA, pages 1350–1357. IEEE
12. Salman Taherizadeh, Andrew C Jones, Ian Taylor, Zhiming Zhao,
and Vlado Stankovski (2017). Monitoring self-adaptive applica-
tions within edge computing frameworks: A state-of-the-art review.
Journal of Systems and Software
13. Sharma V, Song F, You I, Atiquzzaman M (2017) Energy efficient
device discovery for reliable communication in 5g-based iot and bsns
using unmanned aerial vehicles. J Netw Comput Appl 97:79–95
14. Jan Medved, Robert Varga, Anton Tkacik, and Ken Gray (2014).
Opendaylight: towards a model-driven sdn controller architecture. In
World of Wireless, Mobile and Multimedia Networks (WoWMoM),
2014 IEEE 15th International Symposium on a, pages 1–6. IEEE
15. Rabindra K Barik, Rakesh K Lenka, K Rahul Rao, and Devam
Ghose (2016). Performance analysis of virtual machines and con-
tainers in cloud computing. In Computing, Communication and
Automation (ICCCA), 2016 International Conference on, pages
1204–1210. IEEE
16. Deng D-J, Lin Y-P, Yang X, Zhu J, Li Y-B, Luo J, Chen K-C (2017)
Ieee 802.11 ax: highly efficient wlans for intelligent information in-
frastructure. IEEE Communications Magazine 55(12):52–59
17. Kochovski P, Stankovski V (2018) Supporting smart construction
with dependable edge computing infrastructures and applications.
Automation in Construction 85:182–192
18. Yuan Ai, Mugen Peng, and Kecheng Zhang (2017). Edge comput-
ing technologies for internet of things: a primer. Digital
Communications and Networks
19. Openstack. https://www.openstack.org/, 2017
20. Yoji Yamato (2016). Proposal of optimum application deployment
technology for heterogeneous iaas cloud. arXiv preprint arXiv:
1611.09570
21. Yamato Y (2017) Optimum application deployment technology for
heterogeneous iaas cloud. Journal of Information Processing 25:
56–58
22. Netto HV, Lung LC, Correia M, Luiz AF, de Souza LMS (2017)
State machine replication in containers managed by kubernetes.
Journal of Systems Architecture 73:53–59
23. Kubernetes. https://kubernetes.io/, 2017
24. Mqtt. http://mqtt.org/, 2017
25. Ejaz Ahmed and Mubashir Husain Rehmani (2017). Mobile edge
computing: opportunities, solutions, and challenges
26. Pei-Hsuan Tsai, Hua-Jun Hong, An-Chieh Cheng, and Cheng-Hsin
Hsu (2017). Distributed analytics in fog computing platforms using
tensorflow and kubernetes. In Network Operations and
Management Symposium (APNOMS), 2017 19th Asia-Pacific,
pages 145–150. IEEE
27. Malik A, Ahmed J, Qadir J, Ilyas MU (2017) A measurement study
of open source sdn layers in openstack under network perturbation.
Computer Communications 102:139–149
28. Satria D, Park D, Jo M (2017) Recovery for overloaded mobile
edge computing. Futur Gener Comput Syst 70:138–147
29. Ke Zhan and Ai Hua Piao (2016). Optimization of ceph reads/
writes based on multi-threaded algorithms. In 2016 IEEE 18th
International Conference on High Performance Computing and
Communications; IEEE 14th International Conference on Smart
City; IEEE 2nd International Conference on Data Science and
Systems (HPCC/SmartCity/DSS), pages 719–725. IEEE
30. Poat MD, Lauret J (2016) Performance and advanced data place-
ment techniques with ceph’s distributed storage system. Journal of
Physics: Conference Series 762(1)
31. Dipti Shankar, Xiaoyi Lu, and Dhabaleswar K DK Panda (2016).
Boldio: A hybrid and resilient burst-buffer over lustre for acceler-
ating big data i/o. In 2016 IEEE International Conference on Big
Data (Big Data), pages 404–409. IEEE
32. Y Huang, Y Yesha, M Halem, Y Yesha, and S Zhou (2016).
Yinmem: A distributed parallel indexed in-memory computation
system for large scale data analytics. In 2016 IEEE International
Conference on Big Data (Big Data), pages 214–222
33. Mavridis I, Karatza H (2017) Performance evaluation of cloud-
based log file analysis with apache hadoop and apache spark. J
Syst Softw 125:133–151
34. Alexey Svyatkovskiy, Kosuke Imai, Mary Kroeger, and Yuki
Shiraito (2016). Large-scale text processing pipeline with apache
spark. In 2016 IEEE International Conference on Big Data (Big
Data), pages 3928–3935. IEEE
35. Pingkan PI Langi, Warsun Najib, Teguh Bharata Aji, et al. (2015)
An evaluation of twitter river and logstash performances as
elasticsearch inputs for social media analysis of twitter. In 2015
International Conference on Information & Communication
Technology and Systems (ICTS), pages 181–186. IEEE
36. Kumari S, Khan MK, Atiquzzaman M (2015) User authentication
schemes for wireless sensor networks: A review. Ad Hoc Networks
27:159–194
37. Pacevic R, Kaceniauskas A (2017) The development of vislt visu-
alization service in openstack cloud infrastructure. Adv Eng Softw
103:46–56
Publisher’s note Springer Nature remains neutral with regard to jurisdic-
tional claims in published maps and institutional affiliations.