An Architectural Perspective for Cloud Virtualization
Yiming Zhang Qiao Zhou Dongsheng Li
School of Computer School of Computer School of Computer
NUDT NUDT NUDT
Changsha, China Changsha, China Changsha, China
ymzhang@nudt.edu.cn zhouqiao3518@sina.com dsli@nudt.edu.cn
Yaozheng Wang Jinyan Wang Ping Zhong
School of Computer School of Computer Central South University
NUDT NUDT Changsha, China
Changsha, China Changsha, China ping.zhong@csu.edu.cn yqx@microsoft.com Changsha, China
wyaozheng@gmail.com wjy0213@vip.qq.com
Abstract—Virtual machines (VMs) and processes are two
important abstractions for cloud virtualization, where VMs
usually install a complete operating system (OS) running
user processes. Although existing in different layers in the
virtualization hierarchy, VMs and processes have overlapped
functionalities. For example, they are both intended to provide
execution abstraction (e.g., physical/virtual memory address
space), and share similar objectives of isolation, cooperation
and scheduling. Currently, neither of them could provide the
benefits of the other: VMs provide higher isolation, security
and portability, while processes are more efficient, flexible and
easier to schedule and cooperate. The heavyweight virtualiza-
tion architecture degrades both efficiency and security of cloud
services.
There are two trends for cloud virtualization: the first is to
enhance processes to achieve VM-like security, and the second
is to reduce VMs to achieve process-like flexibility. Based on
these observations, our vision is that in the near future VMs
and processes might be fused into one new abstraction for cloud
virtualization that embraces the best of both, providing VM-
level isolation and security while preserving process-level effi-
ciency and flexibility. We present a reference implementation,
dubbed cKernel (customized kernel), for the new abstraction.
Essentially, cKernel is a LibOS based virtualization architec-
ture, which (i) removes traditional OS and process layers and
retains only VMs, and (ii) takes the hypervisor as an OS and
imitates processes’s dynamic mapping mechanism for VMs’
pages and libraries.
1. Introduction
Currently, virtual machines (VMs) and processes are two
important abstractions for cloud virtualization. Hardware
virtual machines [1] have been widely used to guarantee
isolation [2, 3] and improve system reliability and securi-
Kai Yu Chengfei Zhang
School of Computer School of Computer
NUDT NUDT
Changsha, China Changsha, China
yukaigoforit@126.com chengfeizh@gmail.com
Yongqiang Xiong Huaimin Wang
MSRA School of Computer
Beijing, China NUDT
whm w@163.com
ty [4]. VMs usually emulate existing hardware architectures
that run a complete operating system (OS) providing the
environment for executing various applications in the form
of user processes.
Although existing in different layers in the virtualization
hierarchy, VMs and processes have overlapped functionali-
ties. They are both intended to provide execution abstraction
(e.g., memory address space), and share similar objectives
of isolation, cooperation and scheduling. On the other hand,
however, neither of them could provide the benefits of the
other: VMs provide higher isolation, security and portability,
while processes are more lightweight and easier to schedule
and cooperate (with multi-process APIs like fork, IPC and
signals). Therefore, both VMs and processes are needed by
the mainstream software stacks, which not only decreases
efficiency but also increases vulnerability [4, 5].
Recently, commodity clouds (like Amazon’s Elastic
Computing Cloud [6] and Alibaba’s Aliyun [7]) have com-
pletely changed the economics of large-scale computing [8],
providing a public platform where tenants run their specific
applications (e.g., a web server or a database server) on
dedicated VMs. The highly-specialized VMs require only a
very small subset of the overall functionality provided by a
standard OS.
This has motivated two trends for cloud virtualization:
the first is to enhance processes to achieve VM-like security,
and the second is to reduce VMs to achieve process-like flex-
ibility. For example, picoprocesses [9] augment processes
so as to address the isolation and efficiency problem of
conventional OS processes, and Unikernels [4] statically
seal only the application binary and requisite libraries into
a single bootable appliance image so that Unikernel VMs
could be lightweight and flexible.
Based on these observations, our vision is that in the near
future the two abstractions of VMs and processes might be
fused into one new abstraction for cloud virtualization that
embraces the best of both VMs and processes, providing
VM-level isolation and security while preserving process-
level efficiency and flexibility. We argue that a promising
approach for this fusion is to start from the VM abstraction
and add desirable processes’ features following the library
operating system (LibOS) [10] paradigm, which has been
widely adopted in the virtualization literature [4, 11, 9, 12,
13, 14, 15, 16].
We present a reference implementation, dubbed cKernel
(customized kernel), for the new abstraction. cKernel is
essentially a LibOS based virtualization architecture, which
(i) removes traditional OS and process layers and retains on-
ly VMs, and (ii) takes the hypervisor as an OS and imitates
processes’s dynamic mapping mechanism for VMs’ pages
and libraries. By drawing an analogy between conventional
processes and VMs, cKernel has partially (i) realized the
fork API for VMs, a basis for conventional multi-process
programming abstractions, and (ii) supported dynamic lib-
rary loading and linking for the minimized VMs.
The rest of this paper is organized as follows. Sec-
tion 2 further introduces the background and motivation.
Section 3 introduces a promising approach (cKernel) to re-
alizing the fusion of processes and VMs. Section 4 presents
cKernel’s reference implementation. Section 5 introduces
related work. Section 6 discusses future work. And finally
Section 7 concludes the paper.
2. Background
2.1. VMs Vs. Processes
VMs rely on the virtual machine hypervisor, like Xen [1]
and Linux KVM [17], to provide an application with all its
dependencies in a virtualized, self-contained unit. As shown
in Fig. 1 (left), the hardware dependency is provided through
the abstraction of an emulated machine with CPU kernel
mode, multiple address spaces and virtual hardware devices,
which is able to provide the software dependency by running
a conventional operating system together with various lib-
raries, perhaps slightly modified for para-virtualization [1].
VMs have been widely used for cloud computing since
they provide desirable compatibility of major applications
by reusing existing OS functions including the support of
various hardwares.
However, running a conventional OS in a VM brings
substantial overhead, since the guest OS may run many du-
plicated management processes and each guest OS may de-
mand gigabytes of disk storage and hundreds of megabytes
of physical memory. This has recently motivated the revival
of LibOS for VM appliances [4, 18, 11]. For example,
Unikernel [4] implements the OS functions (e.g., networking
system and file system) as a set of libraries that can be
separately linked into the VM appliance at compile time.
The Unikernel (system libraries), application libraries and
application binary are statically sealed into a single bootable
image, which can run directly on the virtual hardware pro-
vided by the hypervisor. These VM oriented LibOS designs
Figure 1: Conventional VMs/processes vs. cKernel VM
appliances.
effectively reduce the image size and memory footprint,
taking the first step towards making a virtual machine more
like a process.
Meanwhile, another trend is to improve processes with
VM-like isolation and security. For instance, the picopro-
cess [9, 19] is a process-based isolation abstraction that
can be viewed as a stripped-down virtual machine with-
out emulated CPU kernel mode, MMU or virtual devices.
Picoprocess-oriented library OSs [12, 15, 13, 14, 16] provide
an interface restricted to a narrowed set of host kernel
ABIs [12], implementing the OS personalities as library
functions and mapping high-level APIs onto a few interfaces
to the host OS.
For example, Drawbridge [12] refactors Windows 7 to
create a self-contained LibOS running in a picoprocess yet
still supporting rich desktop applications. Graphene [15]
broadens the LibOS paradigm by supporting multi-process
APIs in picoprocesses. Bascule [13] allows OS-independent
extensions to be attached safely and efficiently at runtime.
Tardigrade [14] utilizes picoprocesses to easily construct
fault-tolerant services. And Haven [16] leverages the hard-
ware protection of Intel Software Guard Extensions (SGX)
to achieve shielded execution in picoprocesses against at-
tacks from a malicious host. Similar designs of enhancing
processes include Linux Container [20], Docker [21], Open-
VZ [22], and VServer [23].
2.2. Vision
Currently, neither VMs nor processes could provide the
benefits of the other, and thus both of them are needed
by the mainstream software stacks. This not only decreases
efficiency but also increases vulnerability of cloud services.
The two treads (of enhancing processes to achieve VM-
like security and reducing VMs to achieve process-like
flexibility) have motivated our vision that in the near future
VMs and processes might be fused into one new abstraction
for cloud virtualization, with the goal of making the best of
both VMs and processes.
Considering the great success of VMs in security and
backward compatibility, in this paper we will discuss a
promising approach for the fusion. We propose to start from
the VM abstraction, strip off any unused functions of the
accommodated applications, and add desirable processes’
features like inter-process communication and dynamic lib-
rary linking to the minimized VMs. We follow the technical
and industrial tread to base our design on the state-of-the-art
LibOS paradigm, making virtual machines on the hypervisor
be comparable to processes on the OS, as depicted in Fig. 1
(right).
3. Fusion of VMs and Processes
This section introduces a promising approach (called
cKernel) to realizing the new abstraction, a LibOS based
virtualization architecture that fuses VM and process and
makes the best of both for the cloud.
3.1. cKernel VMs
Conventional general-purpose OSs are heavyweight and
contain extensive features, so processes have been proposed
to act as “lightweight VMs” that are cheaper to create,
schedule and configure. In cloud environments, however,
most cloud services only run a specific application in a
single-purpose VM, which needs only a very small portion
of conventional OS supports. Therefore, we expect to reduce
the OS and make the VM become (even) as lightweight as
a process.
Based on this observation, cKernel proposes to start
from the VM abstraction and take minimalism to the ex-
treme, sealing only the application binary into a single
VM. Previous work [5] has showed that a VM could be
booted/run in less than 3 milliseconds, which is comparable
to process fork/exec on Ubuntu Linux (about 1 millisecond).
This proves the feasibility for cKernel to realize high-
performance process-like VM creation and scheduling (such
as VM fork, IPC-like inter-VM communication, and dynam-
ic library linking) by taking the hypervisor as an OS. By
drawing an analogy between VMs and processes, cKernel
naturally relies on the hypervisor for multicore scheduling,
following the multikernel mechanism [24] of running one
VM per vCPU.
In the conventional virtualization architecture, threads
have been proposed as “lightweight processes”. Multiple
threads running in one process could provide more efficient
scheduling and communication. However, many production
systems have showed that threads are difficult to rein in
large-scale distributed environments [25] due to problems
like data race, deadlock, and livelock, which are usually
nondeterministic and counter-intuitive. On the other hand,
in modern clouds one VM usually runs only one service
instance, making threads less attractive than before. There-
fore, cKernel adopts the simplified coroutine model [26] for
concurrency, where many logical coroutines (each of which
serves one client connection) run in the same VM/process so
as to achieve fast sequential performance while alleviating
the problems of threads.
Next, we will in turn introduce how to add the two im-
portant features of conventional processes to cKernel VMs,
namely, dynamic mapping of pages and dynamic linking of
libraries.
3.2. Process-Like Fork of VMs
Since the process abstraction has been striped off from
the simplified cKernel architecture, cKernel VMs need to
realize IPC (inter-process communication) to support con-
ventional multi-process applications. cKernel lets a multi-
process application have the view that its processes run with-
out any change, while these processes are actually cKernel
VMs collaboratively running on the hypervisor.
The fork API is the basis to realize IPC. It is straight-
forward to leverage the hypervisor’s memory mapping func-
tions (like Xen’s page sharing mechanism) to implement
fork. The fork API will create a child VM by duplicating the
calling parent VM and return its ID to the parent. cKernel
needs to maintain all the information of the parent and
child VMs, including the IDs, offsets within shared pages,
references of the grant table, etc.
3.3. Process-like Library Linking of VMs
Full VMs could easily realize dynamic library link-
ing by leveraging their rich-featured OS functionalities. In
the cKernel architecture, however, the minimized VMs are
monolithically sealed at compile time and thus libraries
cannot be dynamically linked as before. Static library linking
brings great inefficiency to cKernel. For example, the entire
VM image has to be re-compiled and re-deployed to update
each of its libraries.
Therefore, it is necessary to design a dynamic LibOS
for cKernel’s minimized VMs, which supports to map the
shared libraries (along with their states) onto the VMs’ vir-
tual address space at runtime. To achieve this goal, cKernel
follows the “shell-core” mechanism [2], sealing only the
application binary into a single VM (shell) and dynamically
linking libraries (core) at runtime. It (i) adds a dynamic
segment to its virtual address space and (ii) dynamically
loads and maps requisite libraries, in a similar way to the
procedure of dynamic linking of conventional processes by
the language runtime (like glibc) [27].
Specifically, cKernel initializes a VM and then jumps
to an entry function provided by the hypervisor. cKernel
adopts a single virtual address space where the basic OS
functions (e.g., bootstrap and memory allocation), system
libraries (including the language runtime), application binary
and libraries, and global data structures co-locate to run
without kernel/user space separation or process scheduling.
4. Reference Implementation
We present a reference implementation of cKernel by
modifying MiniOS [11], a para-virtualized LibOS for user
domains on Xen [1], where the application binary and
necessary libraries are statically sealed into a single bootable
image like Unikernel [4]. MiniOS has implemented the
basic functionalities needed to run an appliance on Xen. It
has a single address space and a simple scheduler without
preemptive scheduling/threading.

reserved by Xen
program heap program stack
virtual address space
boot stack
console
page tables
external I/O xenstore
arch‐specific  6:
structures start info
grant table p2m table
dynamic seg
front driver info
kernel seg
text and data
Figure 2: cKernel memory layout. The blue region is where
dynamic libraries are loaded.
cKernel leverages the GNU cross-compiling develop-
ment tool chain (including the GCC compiler) to support
source code (mainly C in our current prototype) compatibil-
ity, so that a legacy Linux application could be re-compiled
to a cKernel appliance that can run directly on Xen. It
incorporates the lightweight RedHat Newlib [28] to provide
the standard C library functions.
4.1. Fork
cKernel fork is designed to duplicate the VM that ac-
commodates the caller application. When the fork API is
invoked in the parent VM, cKernel uses inline assemblies
to get the values of CPU registers. Dom0 is responsible for
duplicating and starting the child VM.
The parent domain uses the grant table to selectively
share its data with its child, including the stack, heap, .bss
and data sections. This improves the security by avoiding
unauthorized access. For instance, the parent may choose not
to share its stack if the child never access it; and in a key-
value store application a child domain servicing a specific
user should be given access to only that user’s data. Writable
data is shared in a copy-on-write mode.
The forked child VM can access the authorized data of
the parent VM. The child and parent may also use shared
pages to communicate with each other, i.e., realizing con-
ventional inter-process communication APIs in the cKernel
architecture, such as pipe, signal, and message queue. The
Xen-based implementation of inter-VM page mapping will
be studied in our future work.
4.2. Dynamic Library Linking
The original design of MiniOS cannot support dynamic
libraries. Fig. 2 shows the augmented memory layout of
cKernel, which is divided into multiple subspaces for regions
of text and data, frontend driver info, grant table, external
I/O, heap, and Xen-reserved.
The text and data region contains (i) the kernel segment
including the pre-compiled basic OS functions, (if any) static
Algorithm 1 Dynamic Library Linking
1: procedure XC DOM MAP DYN
2: Create domain boot loader
3: Parse kernel image file
4: Initiate boot memory
5: Build image in memory
Read dynamic lib info from program hdr table
7: for each dynamic lib do
8: Retrieve info of dynamic sections
9: Load dynamic sections
10: Relocate unresolved symbols
11: end for
12: Jump to the program entry
13: end procedure
libraries and application binary, (ii) the dynamic segment
where all the dynamic libraries are loaded, and (iii) the struc-
tures needed by cKernel such as the physical-to-machine
(p2m) mapping table, arch-specific structures, page tables
and stacks.
The front driver info region is for realizing the split
driver model of Xen. The grant table region is used to
share pages with dom0 and other domUs. The external
I/O region maps the external devices to the virtual address
space to facilitate I/O operations. The heap region is used
to allocate memory growing in 4KB page chunks. And the
Xen-reserved region is used by Xen for hypervisor-level
management and not available by cKernel.
By emulating the procedure of dynamic mapping in
UNIX processes that supports shared libraries, we have
implemented cKernel’s dynamic mapping mechanism of
shared libraries in the Xen control library (libxc), which is
used by the upper-layer control tool stacks like XL (XM),
Libvirt/VIRSH, and XAPI/XE.
After creating the domain boot loader, XL will build a
para-virtualized domU, during the process of which it (i)
parses the kernel image file, (ii) initiates the boot memory,
(iii) builds the image in the memory, and (iv) boots up the
image for domU. We modify the above 3rd step (of building
image) and add a function xc dom map dyn(), which maps
the dynamic libraries at runtime to the virtual address of
dynamic segment in the text and data region depicted in
Fig. 2.
5. Related Work & Discussion
This section discusses the related work of the fused vir-
tualization architecture, including virtualization techniques
of virtual machines and containers (Section 5.1), modular
kernels and library OSs (Section 5.2), and security issues
(Section 5.3).
5.1. Virtual Machines & Containers
Currently full virtual machine hypervisors like Xen [1]
and Linux KVM [17] are the dominant virtualization tech-
nique that provides an application with all its dependen-
cies on both hardware and software in a virtualized self-
contained unit [29].
The full VM and rich-featured OS architecture is heavy-
weight and inefficient. Therefore, researchers have proposed
container-based virtualization techniques, such as picopro-
cesses [9], Linux Container [20], Docker [21], OpenVZ [22],
and VServer [23]. A container is essentially an OS process
plus its executing environment, so it is much smaller than a
traditional full VM and boots up much faster.
The picoprocess [9] is a process-based isolation con-
tainer. It is a highly restricted process that is prevented from
making kernel calls, and provides a single memory-isolated
address space with strictly user-mode CPU execution and
a narrow interface for implementing specific OS personali-
ties [12].
Linux Container [20] (LXC) runs multiple isolated con-
tainers over a single Linux host, each of which creates its
own virtual spaces for its processes. It uses Linux kernel
cgroups [30] to multiplex and prioritize resources such as
CPU cycles and memory, and leverages Linux namespace to
isolate an applications’ view of users, processes, file systems
and networking [31].
A Docker [21] container is made up of a base image
plus multiple committed layers, and also uses resource
isolation features of Linux to allow multiple independent
dockers to run within a single Linux host. Traditionally,
Docker supports a single process only, so a separate process
manager like supervisor is needed to run multiple processes
in Docker.
Although containers are smaller and have better perfor-
mance, full VMs are still dominant in public commodity
clouds and production platforms due to better isolation,
security, portability and compatibility.
5.2. Modular Kernels & Library OSs
A microkernel [32] provides the basic functions such as
virtual memory management and interprocess communica-
tion (IPC), while many other functions like file systems,
drivers and protocols, are run in the user space. Micro-
kernels have mainly argued for extensibility and flexibili-
ty [33, 34, 35, 36]. For example, the SPIN [33] microker-
nel allows applications to make policy decisions by safely
downloading extensions into the kernel, and Scout [34],
Vino [35] and Nemesis [36] efficiently realize the commu-
nication, database and multimedia applications on top of
microkernels, respectively.
The exokernel architecture [10] achieves higher flexibil-
ity than microkernels by providing application-level secure
multiplexing of physical resources and enabling application-
specific customization of conventional OS abstractions. A
small exokernel securely exports all hardware resources
through a low-level interface to untrusted library OSs, which
use this interface to implement system functions. Exok-
ernels provide applications with higher performance than
traditional microkernel systems. For example, ExOS [10]
and JOS [37] provide much more efficient application-level
virtual memory and inter-process communication primitives,
and SPACE [38] achieves better I/O performance while pro-
viding low-level kernel abstractions defined by the trap and
architectural interface. Cache Kernel [39] provides a low-
level kernel that supports multiple application-level OSs,
and thus it can be viewed as a variation of the exokernel.
The difference between Cache Kernel and Exokernel is
that Cache Kernel mainly focuses on reliability rather than
secure multiplexing of physical resources.
With the proliferation of cloud computing and single-
purpose VM appliances, Unikernels [4] target the com-
modity clouds and compile the applications and libraries
into a single bootable VM image that runs directly on a
VM hypervisor (e.g., Xen). Mirage [4] is an OCaml [40]
based unikernel system where the applications are written
in high-level type-safe OCaml code and are compile-time
statically sealed into VM appliances against modification,
which provides better security than previous library OSs
like Singularity [41]. Mirage eschews backward compatibil-
ity [12] to achieve higher performance, but it is nontrivial to
rewrite the numerous legacy services and applications (that
are written in C, Java, Python, etc.) using OCaml.
Jitsu [42] uses Unikernel to design a power-efficient
and responsive platform for hosting cloud services in the
edge network, providing a new Xen toolstack that satisfies
the demands of secure multi-tenant isolation on resource-
constrained embedded ARM devices.
MiniOS [11] designs and implements a unikernel-style
library OS to run as a para-virtualized guest OS within
a Xen domain. MiniOS has better backward compatibility
and supports some single-process applications written in C.
However, the original MiniOS also statically seals a mono-
lithic appliance and suffers problems similar to Mirage.
Libra [43] provides a library OS abstraction for the JVM
directly over Xen and relies on a separate Xen domain to
provide networking and storage, which is similar to the
design of MiniOS.
Library OSs may also run in picoprocesses [12, 15, 13,
14, 16]. The picoprocess-based library OSs implement the
OS personalities as library functions and map high-level
APIs onto a few interfaces to the host OS. Picoprocess-
based LibOS is more like a traditional OS process and nat-
urally supports dynamic shared libraries. However, currently
picoprocess-based library OSs run only a handful of custom
applications on small research prototypes, mainly because
they require much engineering effort to achieve the same
security and compatibility as full VMs.
5.3. Security Issues
cKernel VMs are expected to provide network-facing
services in the public multi-tenant cloud and run inside full
virtual machines above the Xen hypervisor layer, therefore
we treat both the hypervisor and the control domain (dom0)
as part of the trusted computing base (TCB). Software
running in cKernel VMs is under potential threat from both
other tenants in the cloud and malicious hosts connected
via Internet. The full VM architecture facilitates cKernel to
use the hypervisor as the guard for isolation against internal
attacks from other tenants, and the use of secure protocols
like SSL and SSH helps cKernel appliances trust external
entities.
Some modern OSs apply type-safe languages (e.g.,
OCaml [4], Haskell [44], and C sharp [41]) to achieve type-
safety. For example, Mirage [4] uses OCaml to implement
a set of type-safe protocol libraries and provides static type-
safety with a single address-space layout that is immutable
via a hypervisor extension. But there are still non-type-safe
components (e.g., the garbage collector) in Mirage. Clearly,
cKernel is written in C and thus is not type-safe. However,
the cKernel architecture is orthogonal to the language for
implementation and it is possible to implement a “type-safe”
cKernel using type-safe languages. Note that to achieve
complete type-safety not only the OS and applications but
also the hypervisor and dom0 should not allow the type-safe
property to be violated.
Another approach to improving security is to reduce the
TCB of a VM through disaggregation. For example, Ander-
son et al. propose to run security sensitive components in
separate Xen domains for MiniOS [11], leveraging the fully
virtualized processors and the Trusted Computing Group’s
trusted platform module (TPM) provided by the hypervisor.
Murray et al. implement “trusted virtualization” for Xen and
improve the security of virtual TPM by moving the domain
builder, which is the most important privileged component,
into a minimal trusted compartment [45]. Currently the TCB
of cKernel includes the hypervisor, the privileged dom0 and
the necessary management tools that is hosted on top of a
full Linux OS running dom0, therefore it is worth studying
the disaggregation approach to reduce the TCB of cKernel
in future work.
6. Future Work
The fusion of VMs and processes provides the cloud a
desirable virtualization architecture that embraces the best
of both, but a lot of work remains to do in this promising
direction.
First, currently cKernel supports fork but still cannot
support IPC APIs like pipe, signal and message queue. So
it is urgent to implement the complete IPC abstractions for
cKernel. It is also necessary to study the subtle difference
between VMs and processes. For example, it is common
for a web server to fork a worker process to service a
user request, but a worker VM will have a new IP and
lose connection to the user. The fork API could also be
extended to support the spawn() primitive for fast booting
a large-number of VM appliances in OpenStack
Second, the current cKernel implementation relies on
the (unmodified) shared memory mechanism of Xen, whose
networking system, file system and threading model are
simple and inefficient. This might result in relatively poor
performance [46]. For example, the current networking im-
plementation of Xen is insufficient and we could use the
ClickOS [46] modification to improve it. It is also desirable
to incorporate the MUSL library [47] instead of RedHat
Newlib for better runtime performance.
Third, it is important to develop demonstrating applica-
tions of the new architecture. For example, it is interesting to
build a cloud platform on which multiple large-scale cKernel
applications are deployed securely and quickly. It is chal-
lenging to build a cKernel pool for fast VM booting, which
accommodates sleeping (empty) VMs that can be waken
up by dynamically linking to the designated applications
as shared libraries. It is also useful to build basic cloud
services adopting the fused architecture, such as the large-
scale persistent in-memory key-value store [48, 49].
Fourth, currently customers of a cloud must trust the
cloud provider with both the hypervisor and the cKernel
VMs, while recent advances in hardware protection (Intel
SGX [16]) provides opportunity to shield cKernel VMs not
only against unforseen bugs of the compiler, runtime, hyper-
visor or toolchain, but also against attacks from malicious
hosts and operators.
Last, it is also vital to implement the fused virtualization
architecture for other VM hypervisors like Linux KVM and
VMWare vSphere.
7. Conclusion
In this paper we propose a new virtualization architec-
ture which fuses the abstractions of VMs and processes
for the cloud. We present a LibOS based reference im-
plementation (cKernel) by modifying MiniOS on top of
Xen, which seals only the application binary into a bootable
VM and supports dynamic library linking and conventional
process-like fork. The new virtualization architecture has a
number of advances. For instance, it (i) allows online library
update by leveraging dynamic library mapping, (ii) supports
conventional multi-process applications by realizing VM
fork, and (iii) improves the flexibility of VMs’ management
by emulating processes’ scheduling and operations. Some
key components of cKernel are already available as open-
source [50].
Acknowledgement
This work was supported in part by the National Ba-
sic Research Program of China (973) under Grant No.
2014CB340303, the National Natural Science Foundation
of China (NSFC) under Grant No. 61772541, 61379055 and
61379053. This work was performed when the first author
was visiting the Computer Lab at University of Cambridge
between 2012 and 2013. The authors would like to thank
Huiba Li, Ziyang Li, Yuxing Peng, Ling Liu, Jon Crowcroft,
the iVCE Group, the NetOS group of Computer Lab at
University of Cambridge, and the anonymous reviewers for
their insightful comments.
References
[1] P. Barham, B. Dragovic, K. Fraser, S. Hand, T. Harris,
A. Ho, R. Neugebauer, I. Pratt, and A. Warfield, “Xen
 and the art of virtualization,” ACM SIGOPS Operating
Systems Review, vol. 37, no. 5, pp. 164–177, 2003.
[2] X. Lu, H. Wang, and J. Wang, “Internet-based virtual
computing environment (ivce): Concepts and architec-
ture,” Science in China Series F: Information Sciences,
vol. 49, no. 6, pp. 681–701, 2006.
[3] X. Lu, H. Wang, J. Wang, and J. Xu, “Internet-based
virtual computing environment: Beyond the data center
as a computer,” Future Generation Computer Systems,
vol. 29, no. 1, pp. 309–322, 2013.
[4] A. Madhavapeddy, R. Mortier, S. Smith, S. Hand, and
J. Crowcroft, “Unikernels: Library operating systems
for the cloud,” in ACM ASPLOS. ACM, 2013, pp.
461–472.
[5] F. Manco, C. Lupu, F. Schmidt, J. Mendes, S. Kuenzer,
S. Sati, K. Yasukata, C. Raiciu, and F. Huici, “My vm
is lighter (and safer) than your container,” 2017.
[6] “https://aws.amazon.com/ec2/.”
[7] “http://www.aliyun.com/.”
[8] A. Madhavapeddy, R. Mortier, R. Sohan, T. Gaza-
gnaire, S. Hand, T. Deegan, D. McAuley, and
J. Crowcroft, “Turning down the lamp: software spe-
cialisation for the cloud,” in Proceedings of the 2nd
USENIX conference on Hot topics in cloud computing,
HotCloud, vol. 10, 2010, pp. 11–11.
[9] J. R. Douceur, J. Elson, J. Howell, and J. R. Lorch,
“Leveraging legacy code to deploy desktop applica-
tions on the web.” in OSDI, vol. 8, 2008, pp. 339–354.
[10] D. R. Engler, M. F. Kaashoek et al., “Exokernel:
An operating system architecture for application-level
resource management,” in Fifteenth ACM Symposium
on Operating Systems Principles. ACM, 1995.
[11] M. J. Anderson, M. Moffie, C. I. Dalton et al., “To-
wards trustworthy virtualisation environments: Xen
library os security service infrastructure,” HP Tech
Reort, pp. 88–111, 2007.
[12] D. E. Porter, S. Boyd-Wickizer, J. Howell, R. Olinsky,
and G. C. Hunt, “Rethinking the library os from the
top down,” ACM SIGPLAN Notices, vol. 46, no. 3, pp.
291–304, 2011.
[13] A. Baumann, D. Lee, P. Fonseca, L. Glendenning,
J. R. Lorch, B. Bond, R. Olinsky, and G. C. Hunt,
“Composing os extensions safely and efficiently with
bascule,” in Proceedings of the 8th ACM European
Conference on Computer Systems. ACM, 2013, pp.
239–252.
[14] J. R. Lorch, A. Baumann, L. Glendenning, D. T.
Meyer, and A. Warfield, “Tardigrade: Leveraging
lightweight virtual machines to easily and efficiently
construct fault-tolerant services.”
[15] C.-C. Tsai, K. S. Arora, N. Bandi, B. Jain, W. Jannen,
J. John, H. A. Kalodner, V. Kulkarni, D. Oliveira,
and D. E. Porter, “Cooperation and security isola-
tion of library oses for multi-process applications,”
in Proceedings of the Ninth European Conference on
Computer Systems. ACM, 2014, p. 9.
[16] A. Baumann, M. Peinado, and G. Hunt, “Shielding
applications from an untrusted cloud with haven,” in
USENIX Symposium on Operating Systems Design and
Implementation (OSDI), 2014.
[17] “http://www.linux-kvm.org/.”
[18] “http://osv.io/.”
[19] J. Howell, B. Parno, and J. R. Douceur, “How to
run posix apps in a minimal picoprocess.” in USENIX
Annual Technical Conference, 2013, pp. 321–332.
[20] “https://linuxcontainers.org.”
[21] “https://www.docker.com.”
[22] “http://openvz.org.”
[23] “http://http://linux-vserver.org.”
[24] A. Baumann, P. Barham, P.-E. Dagand, T. Harris,
R. Isaacs, S. Peter, T. Roscoe, A. Schüpbach, and
A. Singhania, “The multikernel: a new os architecture
for scalable multicore systems,” in Proceedings of the
ACM SIGOPS 22nd symposium on Operating systems
principles. ACM, 2009, pp. 29–44.
[25] A. Ho, S. Smith, and S. Hand, “On deadlock, livelock,
and forward progress,” Technical Report, University of
Cambridge, Computer Laboratory (May 2005), 2005.
[26] S. R. Maxwell, “Experiments with a coroutine execu-
tion model for genetic programming,” in Evolutionary
Computation, 1994. IEEE World Congress on Compu-
tational Intelligence., Proceedings of the First IEEE
Conference on. IEEE, 1994, pp. 413–417.
[27] Y. Zhang, D. Li, and Y. Xiong, “Enabling online library
update for vm appliances in the cloud,” NUDT, Tech.
Rep., 2016.
[28] “https://sourceware.org/newlib/.”
[29] J. Rutkowska and R. Wojtczuk, “Qubes os architec-
ture,” Invisible Things Lab Tech Rep, vol. 54, 2010.
[30] “https://en.wikipedia.org/wiki/Cgroups.”
[31] R. Rosen, “Resource management: Linux kernel
namespaces and cgroups,” Haifux, May, 2013.
[32] P. B. Hansen, “The nucleus of a multiprogramming
system,” Communications of the ACM, vol. 13, no. 4,
pp. 238–241, 1970.
[33] B. N. B. S. S. Przemysław, P. E. G. Sirer, M. E. F. D.
Becker, C. Chambers, and S. Eggers, “Extensibility,
safety and performance in the spin operating system,”
in Fifteenth ACM Symposium on Operating Systems
Principles. ACM, 1995, p. 46.
[34] A. B. Montz, D. Mosberger, S. W. O’Mally,
L. L. Peterson, and T. A. Proebsting, “Scout: A
communications-oriented operating system,” in HotOS.
USENIX, 1995, p. 58.
[35] C. Small and M. Seltzer, “Vino: an integrated platform
for operating systems and database research,” Harvard,
Tech. Rep., 1994.
[36] I. M. Leslie, D. McAuley, R. Black, T. Roscoe,
P. Barham, D. Evers, R. Fairbairns, and E. Hyden, “The
design and implementation of an operating system to
support distributed multimedia applications,” Selected
Areas in Communications, IEEE Journal on, vol. 14,
no. 7, pp. 1280–1297, 1996.
[37] “http://pdos.csail.mit.edu/6.828.”
[38] D. Probert, J. Bruno, and M. Karaorman, “Space:
A new approach to operating system abstraction,” in
 Object Orientation in Operating Systems, 1991. Pro-
ceedings., 1991 International Workshop on. IEEE,
1991, pp. 133–137.
[39] D. R. Cheriton and K. J. Duda, “A caching model of
operating system kernel functionality,” in Proceedings
of the 1st USENIX conference on Operating Systems
Design and Implementation. USENIX Association,
1994, p. 14.
[40] “https://ocaml.org.”
[41] G. C. Hunt and J. R. Larus, “Singularity: rethinking
the software stack,” ACM SIGOPS Operating Systems
Review, vol. 41, no. 2, pp. 37–49, 2007.
[42] A. Madhavapeddy, T. Leonard, M. Skjegstad, T. Gaza-
gnaire, D. Sheets, D. Scott, R. Mortier, A. Chaudhry,
B. Singh, J. Ludlam et al., “Jitsu: Just-in-time sum-
moning of unikernels,” in 12th USENIX Symposium on
Networked System Design and Implementation, 2015.
[43] G. Ammons, J. Appavoo, M. Butrico, D. Da Silva,
D. Grove, K. Kawachiya, O. Krieger, B. Rosenburg,
E. Van Hensbergen, and R. W. Wisniewski, “Libra:
a library operating system for a jvm in a virtualized
execution environment,” in VEE, vol. 7, 2007, pp. 44–
54.
[44] T. Hallgren, M. P. Jones, R. Leslie, and A. Tolmach, “A
principled approach to operating system construction
in haskell,” in ACM SIGPLAN Notices, vol. 40, no. 9.
ACM, 2005, pp. 116–128.
[45] D. G. Murray, G. Milos, and S. Hand, “Improving
xen security through disaggregation,” in Proceedings
of the fourth ACM SIGPLAN/SIGOPS international
conference on Virtual execution environments. ACM,
2008, pp. 151–160.
[46] J. Martins, M. Ahmed, C. Raiciu, V. Olteanu, M. Hon-
da, R. Bifulco, and F. Huici, “Clickos and the art
of network function virtualization,” in 11th USENIX
Symposium on Networked Systems Design and Imple-
mentation (NSDI 14). USENIX Association, 2014,
pp. 459–473.
[47] “http://www.musl-libc.org.”
[48] Y. Zhang, C. Guo, D. Li, R. Chu, H. Wu, and Y. Xiong,
“Cubicring: Enabling one-hop failure detection and
recovery for distributed in-memory storage systems,”
in 12th USENIX Symposium on Networked Systems
Design and Implementation, NSDI 15, Oakland, CA,
USA, May 4-6, 2015, 2015, pp. 529–542.
[49] J. Ousterhout, P. Agrawal, D. Erickson, C. Kozyrakis,
J. Leverich, D. Mazières, S. Mitra, A. Narayanan,
G. Parulkar, M. Rosenblum et al., “The case for
ramclouds: scalable high-performance storage entirely
in dram,” ACM SIGOPS Operating Systems Review,
vol. 43, no. 4, pp. 92–105, 2010.
[50] “http://www.kylinx.com/.”