2018 Third International Conference on Fog and Mobile Edge Computing (FMEC)

Virtualization at the Network Edge: A Technology

Perspective.
Syed Rameez Ullah Kakakhel, Lauri Mukkala, Tomi Westerlund, and Juha Plosila
Department of Future Technologies
University of Turku
Turku, Finland.
Email: {srukak,lauri.mukkala,tovewe,juplos}@utu.fi

Abstract — Container-based virtualization offers a very
feasible alternative to heavyweights like KVM or XEN.
Containers are lightweight and offer near-native performance.
They are also easy to deploy because of continuous
integration/development tools and environments. This paper
offers a brief introduction to containers, defines its properties
and provides use-cases in the context of those properties.
Secondly, we look at the live migration of stateful applications via
containers. Live migration is a promising technique at the
network edge to unload computing to other nodes to expand the
use of fog computing / mobile-edge computing. Our experiment
shows that live migration of stateful applications can result in
three different types of errors, namely, resend, reprocessed and
wrong-order errors.
Keywords— Fog Computing, Edge Computing, Virtualization,
Containers, Docker, Live Migration.
I. INTRODUCTION
Virtual machines have a long history of being used for
hardware abstraction. Especially when heterogeneity and
software complexity increases, the use of virtualization
becomes more and more apparent. At the turn of the century,
virtualization was assessed as a solution to the heterogeneity,
complexity, manageability and programmability of wireless
sensor networks [1].
We are on the verge of another evolution of this virtualization
at the network edge. That is, utilizing virtualization to offer
services at the network edge. Services not only targeted to
sensors and actuators but consumers and highly demanding
machines, i.e. vehicles and corporations, even cities for their
smart city initiatives. There are three major virtualization
technologies used nowadays, Xen, KVM and recently
Containers. Other technologies include Microsoft Hyper-V
and solutions from VMware (ESXi, vSphere). KVM relies on
AMD-V or Intel VT-x instructions to facilitate the running of
multiple OSs on a single CPU. Based on its extensive usage in
cloud computing and available expertise, both technical and in
research, it makes sense to bring it down to the edge level.
Micro-datacenters and cloudlets, technologies offering
virtualized services at the network edge, might utilize Xen or
KVM as their virtualization methodology. An example
cloudlet implementation is OpenStack++ [2], based on the
dominant industry cloud platform, OpenStack.
Containers as a form of virtualization are relatively nascent
but gaining a lot of momentum, especially after the
introduction of Docker and its suitability towards DevOps [3].
Standard virtual machines offer a platform that can host
multiple services within itself. On the other hand, containers
facilitate at-scale deployments of one or more applications.
Hence, containers should be complementary to standard
virtual machines and not a full replacement.
In the next sections we will discuss the suitability of
containers to the modern paradigm of fog and edge
computing. First, we will look at containers in more detail,
followed by assessing its suitability to edge computing
platforms. Prominent use-cases and implementations will be
discussed afterwards. Finally, we present the results of our
container migration experiments.
II. WHAT ARE CONTAINERS?
The Linux kernel offers a facility called cgroups, or control
groups, that can isolate or limit the usage of a specific process,
e.g., their access to memory, CPU, and I/O. Cgroups also
offers prioritization, meaning preferential access to resources
for individual processes. Others features of cgroups include,
gauging a groups resource usage and its control, and the
ability to suspend a group. The Linux kernel also has another
feature called namespaces. Namespaces allow the kernel to
isolate a process’ view of the system, e.g., it can isolate a
process’ view of the process IDs, filesystem, hostnames and
network access. Combine cgroups’ ability to partition
resources and the ability of namespaces to isolate a process’
view of the system, and you get effective virtualization. This
OS-level virtualization and separation of groups of processes
are commonly known as containerization or containers. Major
projects that offer container-based virtualization include
Docker, LXC, CoreOS, LMCTFY, and Apache Mesos.

978-1-5386-5896-3/18/$31.00 ©2018 IEEE 87

 2018 Third International Conference on Fog and Mobile Edge Computing (FMEC)
Figure 1 The mist to cloud continuum and multi-layer fog hierarchy.
Being container-based virtualization means that for each
resource or process the kernel and standard libraries are
shared, they do not need to be replicated. This reduces the size
of each container, compared with a virtual machine image.
The drawback is that a container cannot run a different
operating system within itself or utilize a separate set of
drivers from the ones offered by the kernel. However, that is
not an issue when all one wants is to run a thousand copies of
the same application.
Different operating systems have been designed that target the
operation and management of containers. Of interest to this
discussion, are CoreOS1, ResinOS2 and RancherOS3. CoreOS’
Container Linux is an operating system designed from the
ground up to be used with containers. It can be run on public
or private clouds, within a virtual environment or a physical
server within an organization’s premises. Container Linux is
designed to run at massive scale, i.e., clusters. ResinOS is
based on the embedded Yocto Linux. ResinOS allows you to
run Docker containers on embedded devices. It supports a
myriad of embedded computing boards from Raspberry Pi
with ARM processors to UP boards with Intel x86 CPUs.
Lastly, there is RancherOS, which removes most of the parts
of the system and replaces them with containers. This strategy
results in a minimalistic OS ready to run Docker containers
while being less than 30MB in size. Figure 1 shows a mist-
1
https://coreos.com/why/
2
https://resinos.io
3
https://rancher.com/rancher-os/
fog-cloud hierarchy. Each level of this hierarchy is covered by
systems of diverse types and capabilities. The container
operating systems that we discussed can be placed on the fog
and cloud hierarchies at each level. ResinOS and RancherOS
are better suited to the fog environment, while CoreOS is
suited towards cloud computing. Nevertheless, CoreOS can
be used at the telecom base stations or public-private servers.
This view gives us an understanding of how containers can
exist throughout the fog-to-cloud continuum.
III. CONTAINERS AT THE EDGE
The properties and solutions designed for containers that make
them ideal for fog/edge devices will be discussed next. Each
of these would require a thorough discussion, but that would
go beyond the limits of this paper.
A. Lightweight
Standalone applications would always win in relative size
against virtualized ones. However, if virtualization is the
option to go for, containers offer a significant advantage in
size. For example, the virtualized Mi-ESS testbed [4] took
2660 MB, while the containerized version in comparison was
476MB. This drastic difference is because the container
version does not need the full Centos OS (2355 MB) image
and only the required libraries are used (181 MB).
B. Mobility / Migration
Reduced size means easier mobility of containers between
systems or general transfers, e.g., for deployment. Mobility
88
 2018 Third International Conference on Fog and Mobile Edge Computing (FMEC)
might not be a big problem for data centres which have
dedicated gigabit links between their nodes, but on fog/edge
systems this is a big concern. The latency/transfer
requirements become even stricter when we are dealing with
eHealth or Smart Grids or VR/AR games.
C. Performance
Containers offer near-native performance in terms of
computation and network I/O [5]. The only drawback is
significant UDP traffic when using Docker’s NAT feature.
D. Heterogeneity
Handling heterogeneity of computing hardware is common to
virtualization technologies. The added advantage of containers
is that besides the cloud they enable small single board
computers like the Intel NUC to be part of the virtualization
infrastructure [6].
E. Management
Container management is highly automated. Docker Swarm
and Google Kubernetes allow one to set up policies to
automate the deployment of containers over large clusters.
The ability to run a Docker swarm on edge devices has
already been demonstrated [7].
F. Discoverability
Container management solutions come with an extra feature
called ‘registry’, i.e., Docker Registry4 or CoreOS Quay5. A
registry holds all the container images and associated
information. A fog node can request a co-located registry to
download an image instead of requesting from the cloud.
G. Deployability
Containers gained prominence because of their suitability to
DevOps (Development, Operations). These are environments
where fast developments are coupled with faster deployments.
Although currently used in enterprise environments, this is
very well suited to the goals of mobile edge computing
(MEC). As with enterprises, in MEC the deployments are
going to be controlled by the network operators themselves.
H. Dynamicity
As a lightweight VM infrastructure with distributed registries
around a region, containers allow the systems to be
dynamically re-configured. One can switch a text compression
service on a fog node with an image compression service on
the fly.
I. Security
On the cloud front, containers do not offer as much separation
as VMs do. But for edge devices, the ability to verify the local
system, the integrity of a container and its functionality are
more important. Current container solutions offer that level of
security [8].
4
https://docs.docker.com/registry/
5
https://coreos.com/quay-enterprise/
IV. USE-CASES / SCENARIOS
The following example scenarios and use-cases illustrate how
containers can help to alleviate the issues in fog/edge
environments.
A. Resource Management
In the fog-layer, or any edge computing environment, we have
geographically distributed, large-scale resource scattering.
Some of the resources can be self-managed by the
organization, some rented on a cloud-computing style model
from an MEC service provider. Renner et al. [10] discuss the
feasibility of using containers for sharing the distributed
computing resources amongst users à la cloud computing. In
the deployment of such large scale and diverse systems,
management becomes a problem early on. Traditional
management approaches, where much manual effort is put in
by system administrators, would be insufficient. The features
of Manageability, Discoverability and Deployability make
containers an ideal platform to build autonomic [9]
management systems for fog computing.
B. Dynamic Service Delivery
Containers can be restarted, suspended, migrated and updated
on the fly without affecting other containers or the local
system. Combine this ability, with efficient management
strategies and companies can offer dynamic services in the fog
layer. A local gateway can act as a content delivery service
and by merely deploying a newer container can be
transformed into a video transcoding service, based on the
needs of the end-user. Morabito and Beijar [6] assess the
viability of such a platform on the Raspberry Pi 2 and Odroid
C1+ single board computers. Brunisholz et al. [11] have built
a platform that can be re-configured to suit various scenarios
and experiments in the wireless networking domain. Their
platform, called WalT, relies on containers and Raspberry Pi
to achieve adaptability and cost-efficiency.
C. Distributed Edge Platforms
Cheng et al. [12] present a geo-distributed analytics platform
at the network edge that has a dynamic topology. Containers
allow them to not only offer a distributed, dynamic platform
but grant multi-tenancy as well. Recently, building dynamic
and distributed platforms is a lot easier via container
orchestration technologies like Docker Swarm, Apache Mesos
and Google Kubernetes. Javed [7] built a distributed, resilient,
edge cluster based on Kubernetes and Apache Kafka.
With dynamic deployment and clustering, we can also build
ad-hoc edge clusters. Moreover, once the cluster performs its
intended task, it can be reconfigured to perform an entirely
new operation. One scenario would be reconfiguring drone
swarms to perform person-of-interest surveillance or
environmental monitoring or disaster evaluation. Running
containers on a drone and updating it on-the-fly via ResinOS
has already been demonstrated [13].
89
 2018 Third International Conference on Fog and Mobile Edge Computing (FMEC)

D. Cloud-to-Fog Bridge value for performance assessment. In this scenario, the

Edge systems, whether it is fog computing or multi-access application is run on a single node to get the best time estimate
edge computing or cloudlets, rely on the power of cloud for our test. In traditional migration, the worker container is
computing as their backup. Fog devices rely on cloud servers stopped, transferred to another node and simply restarted,
for long-term storage and analytics. Cloud servers can rely on resuming task processing. In enhanced migration, we start the
fog nodes to cover service outages. This makes fog and cloud transfer of the worker container. Once this transfer is
computing complementary technologies. For instance, in case complete, the local container is stopped, and the most recent
of a cloud outage or DDOS attack, the service could be changes (task statuses) are transferred as well. Upon the
transferred to multiple fog nodes. In the same manner, fog completion of both container transfer and recent task list, the
services can be transferred to near-neighbors or the cloud in remote container is started, and it resumes processing tasks
case of an overload. Virtualization of services is the answer to from where the previous one had left off. The enhanced
make this scenario a reality. In addition, containers offer version adds an extra bit of delay as the transferred container
lightweight, near-native performance. From a technology does not resume operations right away but waits for the
perspective, containers offer the ingredients needed to achieve previous host to also send the last active task status and
that. Operating systems (CoreOS, ResinOS), cloud APIs, associated changes.
container orchestrators, repositories (registries), and
performance. What is needed, is combining them into an B. Testbed
efficient, self-managing, and self-assessing solution.
Figure 2 represents our experimental testbed. There are two
identical worker VMs and the control node connected via
localhost. The ProcessingUnit (PU) is our application
V. CONTAINER LIVE MIGRATION container that handles the task assigned to it. In our case, the
task was writing a thousand lines of lorem ipsum to a text file.
Migration is needed, whether it is for load balancing or
Starting, stopping, suspending, migrating and status checks on
offering the best quality of service (i.e. fault tolerance,
the ProcessingUnit are performed via Docker APIs. This is
scalability). Migration is also a key element in the scenarios
Docker Handler’s (DH) job. The TestDriver application
discussed in the previous section (from cloud-to-fog service
running on the local system is responsible for assigning tasks
transfers or vice versa or application migration/deployments
to the two worker nodes. Its operations are governed by
between drones and local nodes). One of the issues with
Algorithm 1 presented below.
traditional (stop, move, restart) migration is service downtime.
Live migration is a solution to reduce the service downtime.
Container live migration has been demonstrated for
applications that are stateless [14]. But for stateful
applications, it is, in theory, possible but difficult in practice.
ClusterHQ, a company offering stateful container based
database services, wound up in 2016 [15]. A detailed
explanation of the complexities and nuances of live migration
with containers are available at [16]. As of writing, Jelastic6, a
commercial cloud services provider offers live migration of
containers. However, work is still going on to make live
migration more feasible. CRIU7 is a software tool utilized for
live migration of containers. IBM demonstrated the use of
CRIU in 2016 [17]. CRIU and associated technologies have
been included into Docker distribution on an experimental
basis. We performed our own live migration experiment with
Docker to understand the nuances associated with it and as a
step towards building our edge computing platform.
A. Experiment
The experiment is based on two worker containers and a
central command node. The identical worker applications are Figure 2 Test Environment Setup
assigned a task and mid-execution one of the containers is
instructed to migrate. The experiment is done in three phases,
1) no migration 2) traditional migration 3) enhanced
migration. The ‘no migration’ scenario gives us the baseline

6
https://jelastic.com/docker/
7
https://criu.org/Main_Page

90
 2018 Third International Conference on Fog and Mobile Edge Computing (FMEC)

Algorithm 1: TestDriver is responsible for during the enhanced migration remains relatively constant,
maintaining list of online and active DockerHandlers between 12-15 seconds (Figure 5). The reason is that with the
(DH) and distributing tasks. enhanced algorithm the PUs are down only during the time it
takes to switch the active PU. Downtime in case of the
standard algorithm increases relative to the migration time.
Data: List of unprocessed tasks
The downtime is higher because both the host and receiver
Result: All tasks have been processed by
workers are offline during the transfer process. The enhanced
ProcessingUnits.
algorithm also reduces the total amount of errors (Figure 6).
No resend, and reprocessed errors occur in case of the
initializeSockets();
enhanced algorithm because at-least one PU is active all the
activeProcessingUnit = null;
time. There are though more re-processed errors potentially
WHILE Test is not completed DO
because one PU is executing a task and upon migration, the
getListOfActiveProcessingUnits();
same task is re-sent to the newer PU.
readNextTaskFromList();
IF 60 seconds since last migration THEN
StopCurrentProcessingUnit();
y = getRandomProcessingUnit();
sendStartToNewProcessingUnit (y);
END
IF 10 seconds since last task sent THEN
sendTaskToActiveProcessingUnit();
END
processReceivedMessagesAndSendResponse();
END

C. Results
We have three performance matrices for our experiment: Figure 3 Algorithm run times in seconds.
Migration times, this is the total time it took to migrate
containers; Downtime, measures how long no processing
occurred during this migration; Error Count, errors that were
encountered. During the experiment, we track three types of
errors: 1) Re-send errors, if the TestDriver had to send a task
again, this was considered as a re-send error. The cause of
such errors could be an offline PU or loss of data during live
migration. 2) Reprocessed errors are the ones where a task
was performed more than once. An example case is that when
mid-execution a container is transferred, sometimes it would
cause it to process that task more than once. 3) ‘wrong-order
errors’, the tasks from TestDriver need to be executed in a
“First In First Out” manner. Reasons for lack of in-order
execution can be improper migration, memory corruption or Figure 4 Total migration time for each algorithm per iteration
choosing to execute the next task in the queue while failing to
finish the current one.
How each error corresponds to a stateful application varies.
For example, re-send errors in the case of a webserver are not
critical, in fact they are somewhat expected. The same does
not hold true for database operations. Wrong order errors for
an asynchronous application are okay but not for financial
systems. Similarly, re-processing a user-login attempt might
be acceptable, but re-processing a user’s order (from an online
shopping cart) is not. The effects of each error are thus
dependent upon the application’s architecture.
Figures 3-6 represent the test results. The standard migration
method increases the overall execution time by 5x, while the
enhanced method increases it by 2.1x (Figure 3). The total
migration time is almost similar as in each case almost the
same amount of data is transferred (Figure 4). The downtime Figure 5 Downtime in seconds for each algorithm, per Iteration

91
 2018 Third International Conference on Fog and Mobile Edge Computing (FMEC)
Figure 6 Number of error encountered for each test type.
VI. CONCLUSION
Compared to solutions like KVM and XEN, containers offer
virtualization at nearly every level of the cloud-fog hierarchy
and are thus very suitable for being used at the edge.
Regarding migration, one technical issue preventing
containers from adoption in active applications is live
migration. There are solutions to avoid migrations in the first
place, i.e., optimal workload division from the beginning, but
that is not always possible. Also, in the case of fog computing,
where a device or a fog node might malfunction, it would be
necessary to migrate the contents from that node. In our tests,
migration presented a 2.1x increase in overall execution time.
More importantly, migration of a stateful application also
resulted in unexpected errors. However, these tests are not a
final authority on evaluating migrations strategies. Academia
and industry are both working towards making live migration
encompassing. CRIU, the container migration utility, is also
being actively improved. Once these technical issues are
hammered out, containers will become an even more viable
option as a virtualization platform at the network edge.
ACKNOWLEDGEMENT
Funding from KAUTE Foundation partially supported this
research.
REFERENCES
[1] N. Costa, A. Pereira, and C. Serôdio, “Virtual
machines applied to WSN’s: The state-of-the-art and
classification,” Second Int. Conf. Syst. Networks
Commun. ICSNC 2007, no. Icsnc, 2007.
[2] K. Ha and M. Satyanarayanan, “OpenStack++ for
Cloudlet Deployment,” no. August, 2015.
[3] L. Bass, I. M. Weber, and L. Zhu, DevOps : a
software architect’s perspective. .
[4] M. B. A. Karim et al., “Extending Cloud Resources to
the Edge: Possible Scenarios, Challenges, and
Experiments,” in 2016 International Conference on
Cloud Computing Research and Innovations
(ICCCRI), 2016, pp. 78–85.
[5] W. Felter, A. Ferreira, R. Rajamony, and J. Rubio,
“An updated performance comparison of virtual
machines and Linux containers,” in ISPASS 2015 -
IEEE International Symposium on Performance
Analysis of Systems and Software, 2015, pp. 171–172.
[6] R. Morabito and N. Beijar, “Enabling Data Processing
at the Network Edge through Lightweight
Virtualization Technologies,” vol. 2016, no. 607728,
2016.
[7] A. Javed, “Container-based IoT Sensor Node on
Raspberry Pi and the Kubernetes Cluster Framework,”
2016.
[8] “What Trusted Computing Means to Users of CoreOS
and Beyond | CoreOS.” [Online]. Available:
https://coreos.com/blog/coreos-trusted-
computing.html. [Accessed: 24-Jan-2018].
[9] J. O. Kephart and D. M. Chess, “The vision of
autonomic computing,” Computer (Long. Beach.
Calif)., vol. 36, no. 1, pp. 41–50, Jan. 2003.
[10] T. Renner, M. Meldau, and A. Kliem, “Towards
Container-Based Resource Management for the
Internet of Things,” 2016 Int. Conf. Softw. Netw.,
2016.
[11] P. Brunisholz, E. Duble, F. Rousseau, and A. Duda,
“WalT: A reproducible testbed for reproducible
network experiments,” in Proceedings - IEEE
INFOCOM, 2016, vol. 2016–Septe, pp. 146–151.
[12] B. Cheng, A. Papageorgiou, F. Cirillo, and E. Kovacs,
“GeeLytics: Geo-distributed edge analytics for large
scale IoT systems based on dynamic topology,” in
Internet of Things (WF-IoT), 2015 IEEE 2nd World
Forum on, 2015, pp. 565–570.
[13] “How we updated a drone while flying - and how you
can too!” [Online]. Available:
https://resin.io/blog/how-we-updated-a-drone-while-
flying-dockercon2016/. [Accessed: 18-Jan-2018].
[14] S. Nadgowda, S. Suneja, and A. Kanso, “Comparing
Scaling Methods for Linux Containers,” in 2017 IEEE
International Conference on Cloud Engineering
(IC2E), 2017, pp. 266–272.
[15] “ClusterHQ? More like Clusterfscked: Cluster bombs,
says it came to market too early • The Register.”
[Online]. Available:
https://www.theregister.co.uk/2016/12/23/clusterhq_b
ombs/. [Accessed: 23-Jan-2018].
[16] “Live migrating a container: pros, cons and gotchas.”
[Online]. Available:
https://www.slideshare.net/Docker/live-migrating-a-
container-pros-cons-and-gotchas. [Accessed: 23-Jan-
2018].
[17] “OpenStack Summit: IBM Demonstrates Live
Migration of Docker Containers - The New Stack.”
[Online]. Available: https://thenewstack.io/openstack-
summit-ibm-demonstrates-live-migration-docker-
containers/. [Accessed: 23-Jan-2018].
92