1350 IEEE TRANSACTIONS ON COMPUTERS, VOL. 70, NO.

9, SEPTEMBER 2021

Zweilous: A Decoupled and Flexible Memory

Management Framework
Guoxi Li, Student Member, IEEE, Wenzhi Chen , Member, IEEE, and Yang Xiang , Fellow, IEEE

Abstract—Currently, with the booming growth of cloud computing, workloads from broad ranges of functions and demands are
crammed into a single physical machine. They lay considerable stress on the need of evolution of the operating system underneath,
especially the memory subsystem. Even enhancing large pages with main memory compression is not intuitively straightforward due to
rigid rules imposed by the state-of-the-art manager Buddy System from the beginning of the design. To relieve the aforementioned
problems and provide broader design space for system designers, we propose Zweilous, a clean slate physical memory management
framework. It is self-contained, highly decoupled, and thus can co-exist with the vanilla memory manager. Separate self-contained
metadata/functions guarantee a flexible extension with little modification to current frameworks. To show it is easy to add enhanced
functions that accelerate the evolution of the memory management subsystem, we implement Hzmem, a new large page memory
manager redesign enhanced with the function of main memory compression. Our method achieves competitive performance compared
with native and virtualized large page support, effective memory size increased and fewer impacts on other parts of the operating
system.

Index Terms—Operating systems, memory management, memory architecture

1 INTRODUCTION underpinned during the time and make memory manager

memory management subsystem is an indispensable
T HE
part of any modern commodity operating system. Histor-
ically, memory management has evolved from a naive
implementation to a very sophisticated paged memory man-
agement framework hand in hand with the evolution of
memory hardware. The most famous memory manager of
paged memory, Buddy System, is pervasive in most mod-
ern commodity operating systems nowadays like Linux or
FreeBSD. However, the algorithm behind it can trace back
into the 1960s [1].
Along with the enhancement of the basic paged memory
management, many subsystems or features are added in
succession. They have become heavily coupled with the
Buddy System, complicating the codebases.
Nowadays, memory management subsystems of operat-
ing systems are notoriously difficult to develop and main-
tain. Kernel developers are in continuous battles against
faults, suboptimal and unpredictable performance and
increasing complexity for development [2].
The root cause is the simplified assumptions or premises
of the underlying physical memories which are centered
around from the beginning of design, such as a single physi-
cal address space or a constant page size [3]. They are

Guoxi Li and Wenzhi Chen are with the School of Computer Science and
Technology, Zhejiang University, Hangzhou, Zhejiang 310027, China.
E-mail: {guoxili, chenwz}@zju.edu.cn.

Yang Xiang are with the School of Software and Electrical Engineering,
Swinburne University of Technology, Hawthorn, VIC 3122, Australia.
E-mail: yxiang@swin.edu.au.
Manuscript received 13 Oct. 2019; revised 14 Apr. 2020; accepted 10 June 2020.
Date of publication 14 July 2020; date of current version 8 Sept. 2021.
(Corresponding authors: Wenzhi Chen and Yang Xiang.)
Recommended for acceptance by A. Sivasubramaniam.
Digital Object Identifier no. 10.1109/TC.2020.3009124
rigid and bug-prone.
Page descriptors as the key metadata for memory subsys-
tems are the very embodiment of these assumptions. For
example, the fact that one page descriptor contains data for
one base page of the constant page size. When it comes to
supports for large pages, developers have no alternative but
to combine page descriptors of continuous base pages as a
compound page for one large page. That inevitably introdu-
ces large overheads in time and space. What’s worse, large
pages become second-class objects with limited functions
relying on the first-class objects, base pages. These obsolete
assumptions impose constraints on the developments and
hinder the further evolution of the memory management
subsystem.
However, it is difficult to eliminate these constraints. One
solution for system developers or researchers is to make
radical changes of metadata/functions at the low levels of
the systems. Because these metadata/functions are also
responsible for many other subsystems, the modifications
are also entangled with them. Therefore the costs of labors
are prohibitive even for experienced developers as shown
in many previous works [4], [5], [6], [7], [8], [9].
Another solution is to use separated, customizable and
self-contained metadata/functions. Because the modifications
are isolated, developers are given as much as freedom as pos-
sible without considering constraints from other subsystems.
They can use memory resources to maximum advantage with
the knowledge they possess. However, excessive overhauls
for the memory subsystems, which is at the heart of the oper-
ating system and functionally interconnected with many
other subsystems, bring the system instability.
Therefore, we propose a general-purpose memory man-
agement framework abstraction from scratch called Zweilous.

0018-9340 ß 2020 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
See ht_tps://www.ieee.org/publications/rights/index.html for more information.

Authorized licensed use limited to: National University Fast. Downloaded on November 23,2022 at 10:54:26 UTC from IEEE Xplore. Restrictions apply.
LI ET AL.: ZWEILOUS: A DECOUPLED AND FLEXIBLE MEMORY MANAGEMENT FRAMEWORK
The key techniques for Zweilous are as follows a) well-
defined placeholders of self-contained, customizable meta-
data/functions that are necessary for a full-featured memory
manager, and b) well partitioned underlying physical mem-
ories that are managed by specific managers.
Through these, Zweilous can run simultaneously with
the vanilla memory manager, leaving critical data like ker-
nel data managed by the vanilla memory manager and
bringing down instability when developers deploying their
experimental implements of new ideas. It gets rid of rigid
assumptions with customizable metadata/function and
hides many non-trial, general details of developments of
physical memory managers. It is the first-ever memory
management framework in a commodity OS with three
benefits:

Decoupled and flexible extension, co-existing with the
vanilla and prevention from the “burden of history”. It is a
decoupled and flexible methodological support of
memory manager implementation with a clean slate
by utilizing separated, customizable self-contained
metadata/functions and management from the vanilla
memory manager. Therefore it enables multiple mem-
ory managers to co-exist in one operating system co-
operatively. That will bring down much instability
from the newly-designed memory manager.

Non-intrusive ways with little modification to implement
new ideas. The framework remodels the current oper-
ating system memory infrastructure in less intrusive
ways. Zweilous enables efficient development of new
ambitious/aggressive physical memory managers
managing the resource on their own for higher per-
formance and/or richer features with little efforts.

More programmability for ambitious developers. The
framework is based on a general understanding of
memory management developments rather than pre-
cise definitions. In order to help developers easily to
introduce and modify the memory management serv-
ices, and to focus on the new ways of managing mem-
ory resources without taking care of details, the
framework hides many non-trial general details and
provides programmability at two levels. First, the
framework can provide userspace application compat-
ible API at the application level. Second, the frame-
work can manage memories for some well defined
partitions of the physical memory at the level of pages.
Therefore, ambitious developers need only pay atten-
tion to the designs of metadata/functions, like page
descriptors and page fault handlers, and fill up the pla-
ceholders for the targeted memory allocator.
On the other hand, both industry and academia have wit-
nessed a growing movement towards cloud computing in
recent years. In response, circumstances and scenarios have
evolved to the point where intricate applications are very
likely to deplete the entire memory of a single physical
machine. Traditional databases, NoSQL stores, and large
router machines fall into such categories of memory-hungry
workloads where large memory footprint fails to bring
about good temporal locality [4], [10], [11], [12], [13].
Two existing techniques are already used for the afore-
mentioned workloads: a) large pages also called hugepages
Authorized licensed use limited to: National University Fast. Downloaded on November 23,2022 at 10:54:26 UTC from IEEE Xplore. Restrictions apply.
1351
to increase TLB reaches and thus reduce page faults and
TLB misses of big memories; b) main memory compression
to increase effective memory size so that it enables a single
machine to accommodate more workloads, thus reducing
costs. They are used frequently in these use cases separately.
However, the costs of labors of applying them in one situa-
tion are prohibitive because of some obsolete assumptions.
For example, a single base page size is assumed in the
current page reclaiming subsystem on which the vanilla
main memory compression framework is based. All that
hinders the enhancement of large pages with main memory
compression.
Motivated by the above, we have implemented a new
physical memory manager called Hzmen for memory compress-
ible hugepages. It takes advantage of Zweilous to show how it
is easy to add enhanced functions that accelerate the pace of
evolution of the memory subsystem. Hzmem is another new
data and control path of physical memory management run-
ning in parallel with the regular path. Hzmem includes a self-
contained hugepage allocator without utilities provided by
Buddy System. Hzmen have competitive performance over
the vanilla large page support in the benchmarks of frequent
and heavy hugepage allocations. Also, we believe that
Hzmem is the first physical memory manager that inherently
allows main memory compression over large pages. Hzmem
can increase effective memory size and impacts little on the
rest of the operating system.
This research makes the following contributions:

We analyze the implementation and execution char-
acteristics of current physical memory management.
We find out the root cause of why its design is rigid
and not open to extension. Therefore, a new clean-
slate idea of managing memory with the current one
in an independent way is pioneered. It opens up
another direction of memory subsystem evolutions
for system designers and researchers to experiment
with their new ideas or algorithms with less effort,
which is difficult before.

We propose a new physical memory management
framework abstraction named Zweilous which is
self-contained and decoupled from the current mem-
ory framework. To the best of our knowledge, this is
the first effort ever to design a physical memory
framework that is independent of and co-exists with
the unmodified one.

We design and implement new large page support
with the main memory compression feature named
Hzmem, which is to enhance large page support
with the function of main memory compression with
moderate efforts.

We demonstrate a thorough evaluation of Zweilous
and Hzmem showing the framework and its specific
appliance can achieve competitive performance,
increasing effective memory and impacting little on
other subsystems.
The remainder of the paper is structured as follows:
section 3 outlines the high-level design and the architec-
ture of Zweilous, section 4 describes the implementation of
Hzmem, section 5 presents the evaluation of Zweilous and
Hzmem, and section 6 concludes.
1352
2 RELATED WORK
Although virtual memory is an active research area, our
work of new memory management framework over com-
mercial operating systems is the first effort to the best of our
knowledge.
2.1 Memory Management Architecture
Recently, lots of researches [2], [4], [5], [6] with great ambi-
tions make efforts to review or improve the memory
subsystem.
Some studies give us an overview of the evolution of
Linux memory subsystems. Huang et al. [2] conduct a quan-
titative survey of virtual memory’s development process
over five years (2009-2015). Insights of this work help sys-
tem designers and developers track hot functions and build
more reliable and efficient memory management systems.
The study with great patience gives us lessons and start
points for diving into this research area.
Clements et al. [5] propose a new design called RadixVM
that makes mmap, munmap and page faults scale perfectly
on non-overlapping memory regions. The drawbacks of
RadixVM are its intensive labor interconnected with other
subsystems if applied to a commodity operating system.
Therefore the only implementation on a Unix-like research
kernel is presented. This study again shows that modifying
the memory subsystem of commercial operating systems
needs to be thought twice and that our work to avoid the
risk by using Zweilous is valuable.
Park et al. [7] propose a new physical memory manager
call lazy iBuddy system. They eliminate overheads of splits
and coalescing by managing pages individually and reduce
lock contentions by exploiting a fine-grained locking mecha-
nism. However, this work has to replace vanilla memory
manager while our work can co-exist with Buddy System.
Considering the complexity of a memory manager, our
work is more reliable.
2.2 Huge Page Support
Large page supports on commercial operating systems are
pioneered very early at the beginning of this century. Navarro
et al. [8] pioneer the official large page support in FreeBSD.
They introduce techniques of reservation-based allocation
and fragmentation control on the vanilla memory manager
while Hzmem is built upon the new memory framework
Zweilous that is independent of the vanilla base page alloca-
tor. Kwon et al. [9] propose Ingens, a framework for transpar-
ent huge pages. They use tracking utilization and access
frequency of memory pages. Panwar et al. [14] present a nice
research of the impacts of fragmentation with huge pages in
the Linux kernel and an efficient memory manager called illu-
minator to make cost-effective allocations. The problems these
works try to resolve, like fragmentations of base pages are the
focus of our work.
2.3 Main Memory Compression
Most researches of main memory compression focus on
hardware modification. Ekman et al. [15] propose a main
memory compression scheme that reduces access costs by
exploiting a highly-efficient structure for data locating and
a hierarchical memory layout to vary the compressibility
Authorized licensed use limited to: National University Fast. Downloaded on November 23,2022 at 10:54:26 UTC from IEEE Xplore. Restrictions apply.
IEEE TRANSACTIONS ON COMPUTERS, VOL. 70, NO. 9, SEPTEMBER 2021
with fewer overheads. Pekhimenko et al. [16] propose Line-
arly Compressed Pages (LCP) that compresses all the cache
lines with the same pages to the same size using special
hardware. Unlike these works, Hzmem does not need any
hardware modifications.
Tuduce et al. [17] propose a main memory compression
solution that can resize the compressed data automatically.
This work gives us much insight into how to manage com-
pressed data in software, and we get much inspiration from
it to implement our part of compression management.
2.4 Limitation of the Previous Works
Previous works improve or enhance the memory manage-
ment subsystem by introducing heavy software or hardware
modifications. Although main ideas are simple such works
require lots of domain-specific knowledge in implementa-
tion. This paper does not focus on the traditional method of
memory subsystem improvement, but rather it focuses on a
general-purpose memory management framework abstrac-
tion to facilitate the implementation of new ideas.
3 DESIGN
In this section, we describe the design and architecture of
Zweilous. We focus on the motivation of redesign and the
multiple choices made during the process of design. To clar-
ify such problems we are giving examples of Linux, but the
idea is not specific to Linux.
3.1 Motivation
The rapid development of cloud computing and thus the
advent of emerging demands impact tremendously the evo-
lution of software and hardware infrastructure. For exam-
ple, Non-volatile memory (NVM) technologies which are
receiving increasing attention in academia and industry
blurs the line between main memory and persistent storage
[18], [19], [20], [21] while big memory workloads of hun-
dreds of GBs call for much more elaborate and adaptive
overcommitment functions of the underlining memory
management. These new situations bring big challenges for
the current memory management whose obsolete design
can date back to decades ago.
Our efforts to build Zweilous are first motivated by
improving memory-hungry workloads without good tempo-
ral locality. In this section, we discuss two exiting techniques:
large pages and main memory compression. We discuss the
current challenges in combining them in one situation, and
briefly mention how we address these challenges. Finally, we
show some more examples of why a framework is needed so
that memory subsystems of the operating system can evolve
independently and quickly.
3.1.1 Large Pages
Most modern commodity operating systems support large
pages. For example, Linux has support for large pages since
around 2003 [22]. The large pages are called hugepages in
Linux. Unlike other operating systems, Linux starts to sup-
port large pages more explicitly via hugetlbfs filesystem.
Developers need to map some sections of memory from
files of hugetlbfs. The process first reads or writes that
LI ET AL.: ZWEILOUS: A DECOUPLED AND FLEXIBLE MEMORY MANAGEMENT FRAMEWORK
sections and thus triggers a page fault. While handling page
faults, the virtual memory mapping of the hugetlbfs
makes the operating system associate the virtual memory
with one free large page instead of one free base page. How-
ever, the free large page is not directly allocated from physi-
cal memory manager (i.e., Buddy System) but large page
memory pools preserved in advance by administrators.
The large page that it maintains is indeed contiguous
base pages preserved from Buddy System. Thus, the mem-
ory pool acts as a “broker” or “middle person” between the
process demanding large pages and Buddy System of the
physical memory manager.
Why processes cannot use large pages directly from the
physical memory manager? It is a burden of history. The
legacy physical memory manager has been running without
trouble for long before operating systems start to support
larges pages. The design of the physical memory manager
in Linux from the very beginning bases on the simplifying
assumption that the page size is constant (i.e., 4 KB of the
base page in Linux). Meanwhile, a lot of enhanced features,
like swapping or slab, have been added to it not long after-
wards. They all exclusively handle base pages using the
page descriptors for base pages, which underpins the
assumption and make the memory management system
rigid and complicated.
Therefore, Linux kernel developers have to take a lengthy
detour and implement the large page support based on the
vanilla memory manager at the expense of overheads and
maintainability problems. In Linux, one large page is deemed
to be 512 contiguous base pages in an aligned 2 MB memory
region but consuming the same amount of page descriptors.
In Linux, it is the key metadata called struct page, a data
structure per base page holding meta-information used by the
physical memory manager.
3.1.2 Main Memory Compression
A compressed-memory system reserves some sections of
memory to hold pages in compressed form to make the
effective memory larger instead of disk read/write for
swapping. Main memory compressions like zRAM [23],
zswap [24] and zcache [25] are supported in Linux, but
the implementation is leveraging page reclaiming subsys-
tem. The page reclaiming is centering around Buddy Sys-
tem and expects a page size of 4 KB. Therefore, hugepages
of legacy memory management are detached from the page
reclaiming for practical reasons.
The constraints that make it difficult to reuse current
infrastructure to complete our goal: 1) current page reclaim-
ing subsystem expects a single base page size exclusively,
not for multiple page sizes. It is no trivial to change the situ-
ation and even some simple key assumptions Linux has
held for a long time have to be challenged. 2) Current large
pages supports do not involve in the page reclaiming sub-
system. Furthermore, The fact that large pages in Linux
have no backing storage devices leads to the impossibility
of swapping and writing back to persistent storage devices.
One solution is to modify base codes of page reclaiming
or even the page descriptors, the metadata, to change this
situation. As one of the most used metadata in the kernel,
the size of page descriptors has swollen largely. So many
Authorized licensed use limited to: National University Fast. Downloaded on November 23,2022 at 10:54:26 UTC from IEEE Xplore. Restrictions apply.
1353
other subsystems are using them that a single byte modifica-
tion impacts much on other parts of the operating system.
The task is time consuming and laborious because of the
complex invariants of memory systems.
We choose another solution that self-contained, custom-
izable metadata/functions are used to eliminate these con-
straints. The implementation can have as much freedom as
possible without the consequent impacts on other subsys-
tems if implemented by the first solution. Developers need
only to pay attention to the specific parts of the system and
not other parts.
3.1.3 Slow Evolutions of Memory Management
The memory subsystem is a mature and core kernel subsys-
tem. Over decades, it has much more functionalities and
interconnected more inextricably with other subsystems. Its
codebases get large and complex. Therefore, the recent
developments are mainly focused on bug fixes, code main-
tenance, and optimizations, but new features [2]. To get a
general understanding of the memory subsystem with other
subsystems in terms of new features added recently, we do
a survey of important features of Linux kernel from version
4.0 to 5.5. That is a time range from Apr 2015 to Jan 2020.
The sources are “LinuxChanges” from the site kernelnew-
bies.org. They introduce kernel changes whenever a major
revision of the kernel is released. We pay attention to the
title in the section of “prominent features” or “coolest
features” after checking its descriptions. We observe that 18
are associated with memory subsystems, which accounts
for only 7.9 percent of the 228 important features in total.
Memory management changes are very few. For example, 4
of them are related to system calls like mmmap, madvise.
They are deemed as important because some in-kernel serv-
ices are exposed to user space, which has a greater impact
on others. Therefore, if we raise the standards, the percent-
age of new and important features that are recently added
to memory subsystems can be lower.
We find that some large and crazy memory manage
changes finding their way into the mainline often take
many years like userspace page fault handling [26], or some
incomplete functions take a couple of years to complete like
using persistent memory as RAM [27]. Although our early
motivation is contingent, these examples show that a frame-
work should be built for hiding non-trial, general details
and make it more extensible for developers. The program-
mability it achieves can help memory subsystems evolve
more quickly. Developers can use the framework to fulfill
them for these benefits:

Zweilous aims to generalize. Components of our
framework are as follows a) well-defined placeholders
of self-contained, customizable metadata/functions
that are necessary for a full-featured memory man-
ager, and b) well partitioned underlying physical
memories that are managed by to-be-implemented
memory managers. It is easy for developers to start
designing the very core of the memory manager with-
out learning or referring to the details that memory
managers have in common. Our work shows that a
developer without any experience of kernel develop-
ing can start to work in 2 weeks.
1354

Zweilous can run with the vanilla memory manager
in one operating system. Zweilous achieve program-
mability at two levels: pages and API levels. Chang-
ing and debugging memory management is difficult
as many in-kernel debugging/trace tools rely on
memory to run. The case is different in Zweilous.
The underlying physical memory is well partitioned:
important data in the operating system, like kernel
data, is managed by the vanilla one which is proved
to be robust and stable in a long time while the mem-
ory of testing applications is handled by the newly-
implemented memory manager based on Zweilous.
Developers can easily introduce and modify the new
memory management services with much fewer
times of kernel crashes, which greatly reduce devel-
opment time and labors.
Therefore, we conclude that enhance the large page with
main memory compression using existing infrastructure
and tools is even more difficult and convoluted, and that it
is advisable to redesign a general-purpose physical memory
management framework based on much more flexible and
extensible assumptions without bringing instability to the
Linux base code.
3.2 Goals and Challenges of Design
We want the code modification both of kernel space and
user space to be as small as possible. Thus, 1) in user space,
framework interface should be compatible with the legacy
hugepage API; 2) in kernel space, a self-contained frame-
work without using existing in-kernel utilities (i.e., function
or API provided by the kernel) makes no overlapping with
old stable kernel codes.
Thus, the main goals are to make our framework decoupled
and flexible, and systems based on Zweilous do not compro-
mise functionality and performance. The new systems can also
run simultaneously with other subsystems and/or even the
vanilla memory management. We take hardware virtualiza-
tion support into consideration to make our framework more
versatile in cloud computing.
A large, old and complicated system is retrofitted with a
new feature that affects or crosscuts all functions. This kind
of retrofit does not make a better new system. There are
many cases in the system design domain. For example,
Buddy System, the allocator that was originally designed
to manage a single page size, is to be retrofitted with a new
feature for large pages. Though the new feature can destroy
the old principles or assumptions, it has to be subordinate
to the old ones by reusing the existing page descriptors.
That makes the design phase left with less freedom and
results in an awkward implementation.
The solution is to level the new features with the old
ones. It is also the first-class principle. The key points of our
framework are as follows: a) well-defined placeholders of
self-contained, customizable metadata/functions that are
necessary for a full-featured memory manager, and b) well
partitioned underlying physical memories that are man-
aged by specific managers. We decide to implement the
huge page physical memory management with a clean slate
based on a redesigned memory management framework
Zweilous.
Authorized licensed use limited to: National University Fast. Downloaded on November 23,2022 at 10:54:26 UTC from IEEE Xplore. Restrictions apply.
IEEE TRANSACTIONS ON COMPUTERS, VOL. 70, NO. 9, SEPTEMBER 2021
From the practical development with high-level design,
we derive the following metadata/functions which a new
memory management framework need to consider:

invisibility from the vanilla memory management
subsystem.

new page descriptor represents the physical page
states.

page fault handler branch.

page reclaiming mechanism.
In need of considering details and implementations of
these aspects, system designers and developers should not
ignore optimization.
3.2.1 Invisibility From the Vanilla Memory Management
Subsystem
To make our framework as decoupled as possible from the
vanilla memory management subsystem in Linux, we decide
to make the framework completely invisible from the vanilla
memory subsystem but not to make any significant modifica-
tion to other parts of the operating system. During the boot-
ing process of the operating system, some memories need to
be removed from the management of the vanilla memory
management subsystems. The reserved memory region is,
thus, invisible from the operating system but visible to our
new memory framework. Applications access memory
through the framework. Then our new memory framework
can manipulate the reserved memory region exclusively
without any interference of any other Linux subsystems.
3.2.2 New page descriptor represents one physical
page’ s states
A normal page descriptor is used for representing one physi-
cal page’s states and the key metadata for memory manager.
These general metadata are modified by many subsystems
in a frequent and complicated way during the whole life of
the operating system. Therefore, They contain various states
concerning many different subsystems in just one data
structure.
For example, the complete definition of a struct page
has a size of over 4 double words in x86-64 architecture
though it has been optimized by developers leaving almost
no means untried. The union of C language is seen
throughout its definition. What parts can be put together
into one union is quite a domain-specific knowledge, and
there is no guarantee that the length of the structure will be
unchanged. Many states concerning different subsystems
are tangled together, which causes a problem of readability
and maintainability in this overused data structure. More-
over, the assumption of a constant page size originates in
the general page descriptors.
The new memory management is focused on the mem-
ory regions that are detached and invisible from the vanilla
memory management subsystem. Due to the main design
goals of decoupling and flexibility, reusing of old page
descriptor with readability and maintainability problems
and basing on the obsolete assumption of page size of 4 KB
show no merits or even necessity. But a page descriptor
used for holding information of one physical page is indis-
pensable, therefore we decide to use a completely new page
LI ET AL.: ZWEILOUS: A DECOUPLED AND FLEXIBLE MEMORY MANAGEMENT FRAMEWORK
descriptor and make its size as small as possible. It only
holds the information needed by our new memory manage-
ment and free of readability and maintainability problems
with custom page size it can present.
3.2.3 Page Fault Handler Branch
Our new framework aims to run on top of commodity hard-
ware. When it comes to memory management, it is difficult
to avoid covering page faults that greatly concern the hard-
ware mechanism. For example, in x86-64 architecture dur-
ing translation from virtual memory address to the physical
memory address (we call it page walk), if a state of the phys-
ical page in the page table entry indicates ”not present”, a
page fault will be triggered by hardware without software
interference. A page fault handler registered by the operat-
ing system during booting will be invoked to deal with the
exception, which is responsible for bringing back the con-
tents of pages and setting up the correct physical memory
address in the page table entry.
The new framework is to be designed as decoupled and
flexible as possible. However, the page fault handler is global
and only one in the operating system, which is a fact deter-
mined by hardware and cannot be changed through software
methods. Therefore, still in decoupled and flexible manners,
we decide to make a branch in the old page fault handler that
deals with page faults exclusive to the memory region we
reserved in advance. That requires only minor modifications
to old page fault handler including a branch check and a new
branch dealing with the new page fault code path for the
reserved memory region. All this does not even touch the old
code path, keeping it stable and decoupled.
3.2.4 Page Reclaiming Mechanism
As a full-featured memory management framework, the
page reclaiming mechanism should not be omitted. There
are various kinds of subsystems in Linux based on the old
page reclaiming mechanism, like swapping, page migra-
tion, page writing back, main memory compression, etc.
They are all running simultaneously according to page
states provided by the aforementioned page descriptor.
They are highly dependent on the page states that are
crammed into one page descriptor, meaning that these page
descriptors are subject to race conditions. Introducing locks
and state checks make codes concerning page reclaiming
lengthy and complicated, thus bug-prone.
However, our new memory management framework
gives up using the old page descriptor, which deprives us
of the aforementioned problems.
To be decoupled and simple, we decide to design a new
page reclaiming mechanism that is completely detached
from the old page reclaiming mechanism and based on the
page states provided by the new page descriptor.
3.3 Architecture of Zweilous
To sum it up, the decoupled and flexible memory manage-
ment framework is shown in Fig. 1. Zweilous contains a
placeholder for a physical memory allocator that can be
implemented from academic or industrial requirements or
ideas, and the whole framework is self-contained and co-
Authorized licensed use limited to: National University Fast. Downloaded on November 23,2022 at 10:54:26 UTC from IEEE Xplore. Restrictions apply.
1355
Fig. 1. The architecture of decoupled and flexible memory management
framework.
exists with the vanilla one at the same time in a decoupled
and flexible way.
There are three layers of the whole system: user space,
kernel space, and hardware. Zweilous gets decoupled from
the vanilla in all these three layers:

In the hardware layer, physical page frames are divided
into two regions: one managed by the vanilla and the
other by Zweilous. To make it decoupled, there is no
overlapped region managed by Zweilous and the
vanilla together at any point in time.

In kernel layer, Zweilous uses its own page descriptors,
physical memory allocator and page reclaiming mech-
anism in as self-contained ways as possible, i.e., there
are no data structures or code paths tangled with the
vanilla memory management subsystem except a
branch in the global page fault handler (because most
hardware platform specifications stipulate one global
page fault handler for all physical memories).

In user space, Zweilous provides a specially-designed
APIs or compatible ones. Through these, administra-
tors decide which part of the virtual memory for a
certain process would be backed up by the physical
memory managed by Zweilous. Other processes or
other parts of virtual memory is still backed by the
vanilla, which makes the most processes run in a sta-
ble and decoupled way.
Between layers of User and Kernel, specific system calls can
be used for specifying parts of virtual memories. Between
layers of Hardware and Kernel, there is a constant one to one
match of physical page frames and page descriptors. There-
fore Zweilous is vertically decoupled and isolated.
4 IMPLEMENTATION OF HZMEM
Leveraging Zweilous, we implement Hzmem, a new hugepage
manager with the function of main memory compression and
hugetlbfs API compatibility. Zweilous can take Hzmem
and put it in the placeholder of the New Memory Allocator
shown in Fig. 1. Physical memories allocated from different
memory managers in Zweilous can be used by applications
differently through normal API or hugetlbfs API.
Hzmem consists of four parts: physical large memory
manager, page fault handler for large pages, page reclaiming
for compressing large pages, and large page compression
data management. Fig. 2 shows workflows in four parts.
A complete workflow involves two paths — the com-
pression path and the decompression path — and two
1356
Fig. 2. Workflows of Hzmem: two paths, two entities, and one end.
entities — a fault handler and a compression daemon — one
end — a compression data manager.
Compression Path. a periodically-running compression
daemon selects pages with less prospect to be used in the
future (we call them cold pages) and passes them to com-
pression data management component. The compression
data manager compresses them in a lossless way.
Decompression Path. This path starts from a user space
application accessing a page that is compressed in the above
path before. When that happens, a special page fault is trig-
gered and the corresponding handler restores the compressed
pages through compression data manager in a fast way.
We implement 3196 lines of C code (LOC). It runs well in
Linux kernel of versions with the vanilla Linux memory
manager Buddy System.
4.1 Hugepage Physical Memory Allocator
We take a clean-slate approach to implement the hugepage
physical memory allocator from the ground up. Overall, it
uses the design principles described in Section 3.
4.1.1 New Page Descriptors
Currently, one important data structure regarding memory
management in Linux is struct page since many subsys-
tems or functionalities of memory management, like page
allocation/free of Buddy System, page reclaiming, etc.
using it for getting/setting page states. Each struct page
represents one base page in the initial designing phase. As
memory capacity increases, it means a large number of page
descriptors, forcing developers to rack their brains cram-
ming more into a single page descriptor just to make its size
small at the expense of readability and maintainability [28].
Hugepages (Linux’s large page support) use this data
structure to reuse page allocation/free from Buddy System
by merging 512 base pages into one compound page. Its
states are stored in one page descriptor at the head, leaving
other page descriptors unused. 511 of 512 page descriptors
are just wastes of spaces.
We introduce new page descriptors exclusively for
Hzmem. It has two benefits:

In Hzmem, vanilla large page support that treats one
large page as 512 contiguous base pages is dropped. A
Authorized licensed use limited to: National University Fast. Downloaded on November 23,2022 at 10:54:26 UTC from IEEE Xplore. Restrictions apply.
IEEE TRANSACTIONS ON COMPUTERS, VOL. 70, NO. 9, SEPTEMBER 2021
Fig. 3. Management of hugepage in Hzmem. In the figure, there are 3
sections in one node. Each node has two lru list for holding allocated
pages.
newly-customized page descriptor is introduced in
Hzmem. One page descriptor strictly represents one
large page and co-exists with the vanilla page descrip-
tors that represent base pages for other parts of the
operating system. The one-to-one mapping gets rid of
any “compound” and space-wasting problems.

The new page descriptors are completely self-contained
and specialized without being used by other parts of
the operating system. It contains fewer fields of struc-
ture with much more readability and maintainability.
The decreasing number and size of the new page descrip-
tors reduce memory footprint enormously, which is ampli-
fied by machines with a large capacity of physical memory
under severe pressure.
4.1.2 Deallocated and Allocated Page Management
Each node of a NUMA system usually have tens of GBs of
memory on servers. To manage such large memory effi-
ciently, one NUMA node should not be managed in the
same way as Linux. Zweilous gets rid of Buddy System,
and thus many its key assumptions cannot hamper Hzmem
from flexible management of memory.
As is shown in Fig. 3, on top of Zweilous, hugepage
memory of one NUMA node contains several sections. Cur-
rently, we assume the size of one section as 4 GB heuristi-
cally. Thus, free pages are scattered into different sections,
reducing lock contention among CPUs. A smaller granular-
ity memory management achieves better scalability and
parallelism.
The state-of-the-art Buddy System manages contiguous
blocks of base pages in the number of 2n , where n is called
order. A n þ 1 order of physical memory block is exactly
twice the size of a n order of physical memory block. To
best fit the size of requested memory large order of block
split into halves of one order lower when necessary. Both
are called a buddy to each other, where the name of Buddy
System comes from. To make contiguous physical memory
as large as possible, both buddies, when free, are coalesced
into one block size of one order higher.
Blocks of excessively large orders are seldom requested
and bring lots of overhead when allocated/free. The upper
limit of order in Linux is empirically 10, which makes the
largest contiguous block 4 MB, a size of 2 contiguous huge-
pages. Thus, it is not advisable to maintain free huge pages
in contiguous blocks. To make the implementation of
LI ET AL.: ZWEILOUS: A DECOUPLED AND FLEXIBLE MEMORY MANAGEMENT FRAMEWORK
Hzmem simple, we arrange free hugepages in linked lists.
The non-trivial splitting and coalescing of “Buddy” are
averted. In Hzmem, allocation/free of hugepages are done
in Oð1Þ time with faster speed. .
4.2 Page Fault Handler
Page faults are categorized into hard and soft. They are dis-
tinguished by whether reading contents from disk to fill the
free page returned from memory manager, which means
that hard page faults make the performance of the system
suffer.
Since huge pages are not mapping persistent storage we
pay attention only to soft page faults that trigger page table
operations in our implementation of Hzmem, which greatly
simplifies the code path of page faults.
There are two solutions when processing page faults in
Hzmem:
1) The most common one is the virtual page is accessed
when not resident in memory for the first time. The
huge page fault is a soft page fault, therefore the new
physical memory manager allocates a zeroed huge
page.
2) The special one is protection violations are triggered.
Either a shared page has to be returned from page
caches or a compressed page has to be decompressed
by the compression data manager.
Hardware-based virtualization helps to improve virtuali-
zation performance and simplify guest OS implementation.
We also extend Hzmem to virtualization supports in terms
of X86-64 CPU and Intel VT-x for practical purposes.
Intel VT-x introduces many new features: In term of CPU
virtualization, root and non-rood mode guarantee different
privileges of CPU executions for isolation and trap-and-
emulate; in terms of memory virtualization, nested paging,
called Extended Page Table (EPT) is used for mapping from guest
physical memory address to host physical memory address.
However, it is not possible to apply Hzmem directly. The
challenge comes from different page sizes of host and guest.
Both host and guest can have 2 sizes of pages: 2 MB and
4 KB. That forms 4 kinds of combinations mathematically,
but only 3 kinds exit in reality—a combination of 2 MB in
guest and 4 KB in host does not exist in Linux.
Ideally, everything just works fine when page sizes
match both in host and guest. An EPT fault is a VM exit,
and thus the execution falls back to the VMM. In the han-
dling of the EPT fault to fill the EPT table, the VMM triggers
a host page fault which brings in the “real page”. Both page
fault handlers can simply use the two corresponding EPT
page entry and host page entry with the same size, which
can be perfectly dealt with by Hzmem or the vanilla.
The situation is not straightforward in the third
combination—4 KB in the guest and 2 MB in the host.
1) Managing Two Sizes in One Memory maznager. In this
case, one host page entry of 2 MB can be involved in
many other EPT faults of 4 KB whose fault address
falls in the same 2 MB memory range. Hzmem mem-
ory manager has to deal with these two kinds of
pages at the same time while Hzmem is inherently
designed for managing pages with a size of 2 MB
Authorized licensed use limited to: National University Fast. Downloaded on November 23,2022 at 10:54:26 UTC from IEEE Xplore. Restrictions apply.
1357
2) More Complicated Page Fault hanzdling. When EPT
faults in the same 2 MB range occur successively,
only the first one triggers a host page fault and loads
the 2 MB memory pointed by the page table entry.
The rests do not have to trigger a host page fault due
to the already resident 2 MB memory. However,
they cannot simply point to the same 2 MB page
since the fault addresses falls in different offsets
within the 2 MB range.
Therefore, some tweaks of page faults exist in Hzmem to
support nested paging in a hardware-based virtualization
environment.
Our solution to the mismatch of page size shows great
flexibility in our design. We add one field to the new page
descriptor for containing the offset within the 2 MB block.
Because of our self-contained design, it is convenient to add
one field without impacts of other subsystems.
We make the page descriptor available to the EPT page
fault handler. When the first EPT fault occurs within one
2 MB memory block, the VMM setups EPT page table and
host page table correctly and loads a free 2 MB page. Other
EPT faults within the same 2 MB block can be handled
through the offset carried by the page descriptor without
manipulating the host page table. The offset in page descrip-
tor is transient, and it is reusable in every EPT fault within.
4.3 Page Reclaiming and Compression Data
Management
The allocated pages are identified as cold or hot. The cold
ones are reclaimed instead of hot ones, which helps reduc-
ing the throttles of the operating system when memory is
not enough. Based on Zweilous we have our own page
descriptors for storing states of hugepages and thus a self-
contained page reclaiming can be established. In every
node, a daemon monitors the usage of huge pages periodi-
cally with a watermark indicating whether the memory is
enough or else triggers page reclaiming. We use the second
chance algorithm [29] taken from vanilla Linux and further
investigation of hot/code page identification for specialized
optimization is left for future work.
Hugepage compression data manager takes a role in com-
pression/decompression of hugepages. We choose LZ4 for a
lossless data compression algorithm with better decompres-
sion speed [30]. Since the not fixed size in the compressed
form we store them in base pages for less fragmentation.
5 EVALUATION
We measure Zweilous and Hzmem with many user applica-
tions and benchmarks, comparing against the state-of-the-art
hugetlbfs in Linux. Experiments are performed on a
64-GB-memory server having 16 Intel Xeon E7520 1.87 GHz
CPUs. Linux 3.10 and Centos 7 are for both host and guest
environments. Base pages are 4 KB and large pages are 2 MB.
After describing some details of our development experi-
ence to show the programmability, we first test Zweilous in
native, non-virtualized and virtualized platforms using
SPECjvm2008[31], SPECCPU2017[32] for overheads and
STREAM [33] for throughputs. Then datasets of Yelp Data-
set Challenge [34] are used to evaluate effective memory
increased in Hzmem. Finally, a series of user applications
1358 IEEE TRANSACTIONS ON COMPUTERS, VOL. 70, NO. 9, SEPTEMBER 2021

are conducted to test the page fault overhead exclusively for

performance isolations. Parameters for hot and cold pages
are consistent throughout the experiments: watermark as
80 percent and detecting period as 10 seconds.

5.1 Programmability of Zweilous

The major benefit of our framework is that developers can
easily introduce and modify the memory management serv-
ices. We evaluate how long it takes one to implement a new
memory manager in Zweilous by report the time it took one
student of our lab to add the compression/decompression
to Hzmem. The student has no prior experience of kernel
development. Therefore, the student needs to do a survey
of compression/decompression techniques in Linux kernel
and some basics of kernel coding styles. Our framework
gets rid of the needs of learning kernel API as all the meta-
data/functions are self-contained. The student had only to
learn how to put the invocation of customized compres-
sion/decompression services in the branch of the page fault
handler. Therefore, it took 2 weeks to start actually coding,
and less one week to finish the prototype. We believe more
experienced developers can get quicker to start while the
finishing time depends on the complexity of the design
which is out of the interest of our work.

5.2 Overheads of Zweilous

We want to measure the overheads introduced by Zweilous
in SPECCPU benchmarks. We utilize a tool called hugectl
in libhugetlbfs [35]. It is a wrapper for hosting applica-
tions by remapping the text and data section of applications
in large pages. The heap sections used by libc can also be
hooked by redirecting to mapping of large pages. All this
can be achieved without any modification of source code.
From the results shown in Fig. 4a, we can find that Zwei-
lous slows down in the most benchmarks, but no worse
than 2.32 and 0.58 percent on average. Since SPECCPU is
CPU-intensive benchmark suite, benefits of the new design
do not appear obvious. The performance reduction can be
explained from the extra page fault handling with decom-
pression/compression and lack polishness of the code like
the vanilla one. Though Zweilous is not showing many
advantages, it works competitively and not too badly in
non-memory-insensitive environments.
Fig. 4b shows the overheads in Zweilous from SPECjvm
benchmarks. SPECjvm benchmarks focus on the perfor-
mance of a Java Runtime Environment and reflect the perfor-
mance of the hardware processor and memory subsystem.
We configure the SPECjvm to make the Java heap using large
pages.
The results show that Zweilous works better in most
cases with 3.56 percent best. The worst case is -4.88 percent.
Zweilous runs much better in SPECjvm than in SPECCPU.
Since SPECjvm is a data-intensive benchmark suite, benefits
of the new design come to show its advantage. Using Zwei-
lous, the management of large pages eliminates much
redundancy of memory footprint by reducing the number
of page descriptors and simplifies the execution of page
fault handling of large pages thanks to the new code path of
page fault handling.
Fig. 4. Overhead of Hzmem in SPECCPU and SPECjvm. All values are
normalized to the baseline of unmodified Linux.
5.3 Throughputs of Hzmem
We want to measure the throughput of memory managed
by Zweilous and compare it with the unmodified vanilla
manager and how it scales with large memory usage.
Table 1 compares the throughput of STREAM bench-
mark in Hzmem to unmodified Linux. STREAM is a syn-
thetic benchmark of memory bandwidth with simple vector
operations: Copy, Scale, Add, and Triad. The STREAM
benchmark is suitable for datasets that are beyond the CPU
cache. We run STREAM with different vector sizes which
are indicated by the x-axis in units of millions.
The results show that with vector sizes ranging from 20 mil-
lion to 40 million the throughput of Hzmem and unmodified
Linux is not different wider than 0.2 percent. On the other
hand, with vector sizes from 80 million on, the gap between
them gets wider. Hzmem achieves at most 0.6 percent better
throughput and even on average 0.5 percent can be reached.
All this proves that Zweilous scales better when working sets
increase greatly.
When memory usage is much larger than the cache
capacity and fragmented after long-running without the fit
size of free large pages, the vanilla memory allocator has to
coalesce memory to get required sizes, thus bringing over-
heads and throughput degradation. However, Zweilous
uses a new metadata for each large page and eliminates the
overheads of splitting or coalescing neighboring memory
from Buddy System of the vanilla memory manager, which
shows great advantages on the large scale.

Authorized licensed use limited to: National University Fast. Downloaded on November 23,2022 at 10:54:26 UTC from IEEE Xplore. Restrictions apply.
LI ET AL.: ZWEILOUS: A DECOUPLED AND FLEXIBLE MEMORY MANAGEMENT FRAMEWORK 1359

TABLE 1
Throughput (MB/s) of Benchmark STREAM in Different szizes (millions) of Vectors

COPY SCALE ADD TRIAD

Size(m) Hzmem Unmod Hzmem Unmod Hzmem Unmod Hzmem Unmod
20 3031 3035 2595 2592 2825 2823 2352 2352
40 3024 3018 2588 2582 2814 2807 2321 2319
60 3030 3013 2591 2578 2816 2801 2324 2315
80 3028 3009 2591 2575 2810 2795 2323 2310
100 3030 3010 2590 2576 2824 2809 2342 2332

5.4 Virtualized Platform Section 4.2. In both cases, the QEMU in host creates and
We test our Zweilous in the virtualized platform using lib- maps the main memory of the virtual machine using 2 MB
virt 3.9.0 and QEMU 1.5.3. Both guest and host use Linux page size. In guest, we configure the benchmark SPECjvm
3.10. QEMU [36] is a generic and open source machine emu- using two page sizes to stress memory subsystems through
lator and virtualizer that can leverage hardware virtualiza- different page fault code path.
tion on x86-64 platform. It is a user-mode virtual machine The first case is that the benchmark uses 4 KB page size
launch and monitor. To use Hzmem to manage the memory inside the VM. In this case, EPT faults return execution to
used by VM, we launch and map the memory of QEMU via the VMM and the VMM handles the EPT faults in 4 KB size
the aforementioned API hugetlbfs. which further triggers normal page faults in 2 MB in the
host. If successive EPT faults occur within the same 2 MB
5.4.1 Start Time memory block, no host page faults are triggered and correct
offsets within the 2 MB memory block are set without trou-
We want to measure how long it takes to create and boot a vir- ble. Fig. 6a shows the results of this case. The best case is
tual machine using management from Zweilous, and how it 3.37 percent and the worst is -0.91 percent. Most results are
scales as the capacity of the memory of running VMs increases, within the range of plus/minus 1.0 percent. Zweilous
and how these compare to an unmodified virtual machine. achieve comparative performance with the unmodified one.
In our test, we increase the memory capacity from 2 GB to
32 GB at one step of 2 GB. The results are shown in Fig. 5. An
unmodified virtual machine with 2 GB memory capacity
starts about 1.5 seconds, scaling almost linearly to a maxi-
mum of 16 seconds with 32 GB memory capacity. On the
other hand, Zweilous curve in the Figure starts at 3.7 seconds
up to 14 seconds. Before 22 GB memory capacity, Zweilous
is inferior to the unmodified in start time. However, from
22 GB up to 32 GB, Zweilous shows a great advantage over
the unmodified one.
Zweilous contains only a prototype code base without
the polishness of many superior engineers like the unmodi-
fied one. However, it has a Smore advanced design to
reduce the redundancy of the vanilla large page manage-
ment. Therefore, it scales greatly compared to the unmodi-
fied one in the start time of virtual machines.

5.4.2 Overheads of Different Combinations

We want to measure overheads of combinations for differ-
ent page sizes in host and guest, which is detailed in

Fig. 6. Overheads of benchmark using different page size combinations.

Fig. 5. Start time of VMs using Zweilous and unmodified. All values are normalized to the baseline of unmodified Linux.

Authorized licensed use limited to: National University Fast. Downloaded on November 23,2022 at 10:54:26 UTC from IEEE Xplore. Restrictions apply.
1360
Fig. 7. Effective memory of Hzmem and unmodified Linux.
The second case is that the benchmark uses 2 MB page size
inside the VM. In this case, page fault handling is not so com-
plicated as the first case. Every EPT fault goes hand in hand
with one host page fault, which simply manipulates two
entries in two page tables. Fig. 6b show the results of this sec-
ond case. Zweilous performs much better than in the first
case. Only three benchmarks show slower performance than
the unmodified one without over 1.0 percent worse. Four
benchmarks run better with over 2.0 percent improvement
and the best is 5.0 percent. This case goes simple code path in
page fault handling can bring the best of design in reducing
the redundancy of the vanilla memory subsystems.
5.5 Evaluation of Hzmem
In this section, we test our experimental huge page enhanced
with main memory compression based on our memory
framework Zweilous.
5.5.1 Effective Memory Increasing
Effective memory size is the actual amount of memory
applications can access. It depends on the datasets and aver-
age ratio of the compression algorithm. More importantly,
the management area of compressed data in the main mem-
ory also contributes to this and reflects the optimization of
implementation.
Applying LZ4 algorithms to datasets of Yelp Dataset, an
average compression ratio of 0.23 can get. We want to mea-
sure the effective memory increased by Hzmem. Both
Hzmem and unmodified Linux have the same size of large
memory (shown in the x-axis of Fig. 7) before evaluation.
The benchmark keeps requesting memory and accessing
the allocated memory. Until the benchmark fails due to
memory exhaustion the amount of memory it can access is
the effective memory size.
A comparison of effective memory increased by Hzmem
with unmodified Linux is shown in Fig. 7. It is clear that
unmodified Linux fails to increase effective memory size
when the amount of memory set up before running is
depleted completely. Hzmem is capable of getting the effec-
tive memory size increased by 4465 percent while the gap
narrows as the memory set up before gets close to the capac-
ity of hardware memory. This may seem perplexing but the
explanation is simple and clear. Data of compressed form
are stored in the rest of the main memory. The smaller the
amount of memory is set up in advance the larger the
remaining space to store the compressed data. However, as
Authorized licensed use limited to: National University Fast. Downloaded on November 23,2022 at 10:54:26 UTC from IEEE Xplore. Restrictions apply.
IEEE TRANSACTIONS ON COMPUTERS, VOL. 70, NO. 9, SEPTEMBER 2021
Fig. 8. Throughput of page fault using a stress test. Overcommitted is the
case where working sets are beyond huge page capacity while the non-
overcommitted case is not. When beyond capacity unmodified Linux that
cannot increase effective memory has no throughput at all and the
benchmark stalls.
more data get compressed the overheads increase when
decompressed. The balance between effective memory size
and performance is left for future works to study.
5.5.2 Overhead and Performance Isolation
of Page Fault
We want to measure the throughputs of page faults, the cor-
rectness of decompressing and CPU utilization introduced
by Hzmem.
Overheads of memory access mostly introduced by page
walks from TLB misses and page fault handling from page
table entry missing. And the performance penalty is larger
for the latter as page fault handling is resolved in a more
complicated software way. Thus, the efficiency of page fault
handling in commercial operating systems is critical to
accelerating memory accesses.
We want to evaluate the page fault handling of Hzmem in
an isolated way. We write a benchmark that has heavy page
faults. It first requests certain amounts of large pages and
accesses pages with random numbers followed by page faults.
Two cases depending whether the working sets of the bench-
marks is larger than the amount of memory set up before run-
ning. Main memory compression introduced by Hzmem can
reclaim huge pages in the compressed areas so that it makes
room for more free huge pages to be requested by applica-
tions. Therefore, applications with more working sets than the
huge page capacity can still run without stopping.
In Fig. 8, we call overcommitted when working sets are
larger the huge page capacity and non-overcommitted oth-
erwise. For the overcommitted cases, the huge page capac-
ity is set up as 4 GB. Overheads of data compressing from
LI ET AL.: ZWEILOUS: A DECOUPLED AND FLEXIBLE MEMORY MANAGEMENT FRAMEWORK
page reclaiming devalue the results. Therefore, we let the
benchmark sleep for 30 seconds to cool down the pages and
subtract this period of time for fairness. And in case of
cooled down pages, the results can reflect the exclusive
throughput of page fault handling.
First, the correctness of decompressing is proved in our
experiments.
Fig. 8a shows throughputs of Hzmem and unmodified
Linux when non-overcommitted. Hzmem has larger through-
put but the gap is within 9 MB/s, 4 percent in the best case.
Fig. 8b shows throughputs of Hzmem and unmodified
Linux when overcommitted. When working sets are less
than 4 GB there is almost no difference. When requesting
more than 4 GB there is no throughput of unmodified Linux
as the effective memory size hits the ceiling while that of
Hzmem is not zero and decreasing no more than 27 percent.
The performance is quite higher than disk accesses while
swapping.
The CPU utilization of compressing daemons is mea-
sured in the meantime: 16.6 percent at most and 11.0 percent
in average, which can be optimized in future works.
5.5.3 Summary
Zweilous is a general-purpose memory management frame-
work abstraction based on decoupled, flexible and self-
contained metadata/functions, co-existing with the vanilla
memory manager. Therefore, the implementation of Hzmen
is achieved in more isolated and non-intrusive ways on com-
mercial operating systems. Native and virtualized Hzmem
both achieve competitive performance and throughput
against the vanilla equivalent.
6 CONCLUSION
With developments of cloud computing and AI, and improve-
ment of hardware architecture, workloads that require big-
memories become pervasive. In the future, nobody can pre-
dict memory patterns of dominant workloads. What we make
sure is that workloads are becoming more diverse. These
workloads stress memory heavily, forcing improvement of
memory management. However, the obsolete assumptions of
memory subsystems become burdens and obscure atrociously
the evolution of general-purpose operating systems.
A new memory management framework abstract called
Zweilous is proposed to address this problem. It is decoupled
and flexible with customizable self-contained metadata/func-
tions and can co-exist with the vanilla memory manager in a
separated and independent way. Leveraging Zweilous,
researchers can experiment and implement their creative
ideas of memory manager designs safely and robustly, which
helps evolve memory subsystems of OSes more rapidly.
In one way of realizing our ideas of combining the tech-
niques of large pages and main memory compression, we
implement Hzmem with little modification of Linux kernel
codebase. Using the new memory framework, we overcome
many challenges which seem hopeless otherwise. Hzmem
achieves competitive performance compared with native
and virtualized large page supports, effective memory size
increased and fewer impacts on other parts of the operating
system.
Authorized licensed use limited to: National University Fast. Downloaded on November 23,2022 at 10:54:26 UTC from IEEE Xplore. Restrictions apply.
1361
ACKNOWLEDGMENTS
The authors would like to thank the members of ARC Lab of
Zhejiang University for their constructive comments and
help during the work. They would also want to thank the
reviewers for their feedback to improve this article.
REFERENCES
[1] K. C. Knowlton,“A fast storage allocator,” Commun. ACM, vol. 8,
no. 10, pp. 623–624, 1965.
[2] J. Huang, M. K. Qureshi, and K. Schwan, “An evolutionary study
of linux memory management for fun and profit,” in Proc. USE-
NIX Conf. USENIX Annu. Tech. Conf., 2016, pp. 465–478.
[3] N. Ganapathy and C. Schimmel, “General purpose operating sys-
tem support for multiple page sizes,” in Proc. USENIX Annu. Tech.
Conf., 1998, no. 98, pp. 91–104.
[4] A. Basu, J. Gandhi, J. Chang, M. D. Hill, and M. M. Swift,
“Efficient virtual memory for big memory servers,” Proc. 40th
Annu. Int. Symp. Comput. Architecture, 2013, pp. 237–248.
[5] A. T. Clements, M. F. Kaashoek, and N. Zeldovich, “Radixvm:
Scalable address spaces for multithreaded applications,” in Proc.
8th ACM Eur. Conf. Comput. Syst., 2013, pp. 211–224.
[6] S. Gerber, G. Zellweger, R. Achermann, K. Kourtis, T. Roscoe, and
D. Milojicic, “Not your parents’ physical address space,” in Proc.
15th Workshop Hot Topics Operating Syst., 2015, Art. no. 16.
[7] H. Park, J. Choi, D. Lee, and S. H. Noh, “IBuddy: Inverse buddy for
enhancing memory allocation/deallocation performanceon multi-
core systems,” IEEE Trans. Comput., vol. 64, no. 3, pp. 720–732,
Mar. 2015.
[8] J. Navarro, S. Iyer, P. Druschel, and A. Cox, “Practical, transparent
operating system support for superpages,” ACM SIGOPS Operat-
ing Syst. Rev., vol. 36, 2002, Art. no. 89.
[9] Y. Kwon, H. Yu, S. Peter, C. J. Rossbach, and E. Witchel,
“Coordinated and Efficient Huge Page Management with
Ingens,” Proc. 12th USENIX Conf. Operating Syst. Des. Implementa-
tion, 2016, pp. 705–721.
[10] J. Ousterhout et al., “The case for RAMCloud,” Commun. ACM,
vol. 54, no. 7, pp. 121–130, 2011.
[11] M. Ferdman et al., “Clearing the clouds: A study of emerging
scale-out workloads on modern hardware,” ACM SIGPLAN Noti-
ces, vol. 47, no. 4, pp. 37–48, 2012.
[12] K. Koh, K. Kim, S. Jeon, and J. Huh, “Disaggregated cloud mem-
ory with elastic block management,” IEEE Trans. Comput., vol. 68,
no. 1, pp. 39–52, Jan. 2019.
[13] C. Chen, K. Li, A. Ouyang, and K. Li, “FlinkCL: An opencl-based in-
memory computing architecture on heterogeneous cpu-gpu clusters
for big data,” IEEE Trans. Comput., vol. 67, no. 12, pp. 1765–1779,
Dec 2018.
[14] A. Panwar, A. Prasad, and K. Gopinath, “Making huge pages
actually useful,” in Proc. 23rd Int. Conf. Architect. Support Program.
Lang. Operating Syst., 2018, pp. 679–692. [Online]. Available:
http://doi.acm.org/10.1145/3173162.3173203
[15] M. Ekman and P. Stenstrom,“A robust main-memory compres-
sion scheme,” ACM SIGARCH Comput. Architecture News, vol. 33,
no. 2, pp. 74–85, 2005.
[16] G. Pekhimenko et al., “Linearly compressed pages: A low-
complexity, low-latency main memory compression framework,”
in Proc. 46th Annu. IEEE/ACM Int. Symp. Microarchitecture, 2013,
pp. 172–184.
[17] I. C. Tuduce and T. R. Gross, “Adaptive main memory comp-
ression,” in Proc. USENIX Annu. Tech. Conf., General Track, 2005,
pp. 237–250.
[18] F. Chen, M. P. Mesnier, and S. Hahn, “A protected block device
for persistent memory,” in Proc. 30th Symp. Mass Storage Syst.
Tech., 2014, pp. 1–12.
[19] D. Xue, L. Huang, C. Li, and C. Wu, “Dapper: An adaptive man-
ager for large-capacity persistent memory,” IEEE Trans. Comput.,
vol. 68, no. 7, pp. 1019–1034, Jul. 2019.
[20] S. Li et al., “RC-NVM: Dual-addressing non-volatile memory
architecture supporting both row and column memory accesses,”
IEEE Trans. Comput., vol. 68, no. 2, pp. 239–254, Feb. 2019.
[21] C. Wang, Q. Wei, J. Yang, C. Chen, Y. Yang, and M. Xue, “NV-
Dedup: High-performance inline deduplication for non-volatile
memory,” IEEE Trans. Comput., vol. 67, no. 5, pp. 658–671,
May 2018.
1362
[22] J. Corbet huge pages part 1 (Introduction), 2010. [Online]. Avail-
able: https://lwn.net/Articles/374424/
[23] N. Gupta, zRAM: Compressed RAM based block devices, Accessed:
Jul. 2020. [Online]. Available: https://www.kernel.org/doc/
Documentation/blockdev/zram.txt
[24] S. Jennings, zswap, Accessed: Jul. 2020. [Online]. Available: https://
www.kernel.org/doc/Documentation/vm/zswap.txt
[25] J. Corbet, zcache: A compressed page cache, 2010. [Online]. Avail-
able: https://lwn.net/Articles/397574/
[26] J. Corbet, User-space page fault handling, 2015. [Online]. Avail-
able: https://lwn.net/Articles/636226/
[27] J. Corbet, Persistent memory for transient data, 2019. [Online].
Available: https://lwn.net/Articles/777212/
[28] J. Corbet, Cramming more into struct page, 2013. [Online]. Avail-
able: https://lwn.net/Articles/565097/
[29] W. Mauerer, Professional Linux Kernel Architecture. Hoboken, NJ,
USA: Wiley, 2010.
[30] P. Zaitsev and V. Tkachenko Evaluating Database Compression
Methods: Update, 2016. [Online]. Available: https://www.
percona.com/blog/2016/04/13/evaluating-database-
compression -methods-update/
[31] SPEC, SPECjvm2008, 2008. [Online]. Available: http://www.spec.
org/jvm2008/
[32] SPEC, SPECCPU2017, 2017. [Online]. Available: http://www.
spec.org/cpu2017/
[33] McCalpin, STREAM benchmark, 2002, [Online]. Available:
http://www.cs.virginia.edu/stream/
[34] Yelp Dataset Challenge, Accessed: Jan. 2017. [Online]. Available:
https://www.yelp.com/dataset_challenge/
[35] libhugetlbfs, 2015. [Online]. Available: https://github.com/
libhugetlbfs/libhugetlbfs
[36] QEMU, the FAST! processor emulator, Accessed: Jul. 2020. [Online].
Available: http://www.qemu.org
Guoxi Li (Student Member, IEEE) received the
BS degree in computer science from Zhejiang
University, he is currently working toward the PhD
degree with Zhejiang University. His research
interests include operating system and system vir-
tualization. He is a student member of the ACM.
Authorized licensed use limited to: National University Fast. Downloaded on November 23,2022 at 10:54:26 UTC from IEEE Xplore. Restrictions apply.
IEEE TRANSACTIONS ON COMPUTERS, VOL. 70, NO. 9, SEPTEMBER 2021
Wenzhi Chen (Member, IEEE) received the PhD
degree from the College of Computer Science and
Engineering at Zhejiang University. He is a profes-
sor with the College of Computer Science and
Technology, Zhejiang University, and the director of
Information Technology Center of Zhejiang Univer-
sity, he used to be the vice dean of the College of
Computer Science and Technology. His current
research interests include embedded system and
its application, computer architecture, and com-
puter system software and information security. He
is a member of ACM and ACM Education Council.
Yang Xiang (Fellow, IEEE) received the PhD
degree in computer science from Deakin Univer-
sity, Australia. He is currently a full professor and
the dean of Digital Research & Innovation Capa-
bility Platform, Swinburne University of Technol-
ogy, Australia. His research interests include
cyber security, which covers network and system
security, data analytics, distributed systems, and
networking. He is also leading the Blockchain ini-
tiatives at Swinburne. In the past 20 years, he has
been working in the broad area of cyber security,
which covers network and system security, AI, data analytics, and net-
working. He has published more than 300 research papers in many inter-
national journals and conferences. He is the editor-in-chief of the
SpringerBriefs on Cyber Security Systems and Networks. He serves as
the associate editor of IEEE Transactions on Dependable and Secure
Computing and IEEE Internet of Things Journal, and the editor of Journal
of Network and Computer Applications. He served as the associate editor
of IEEE Transactions on Computers and IEEE Transactions on Parallel
and Distributed Systems. He is the coordinator, Asia for IEEE Computer
Society Technical Committee on Distributed Processing (TCDP).
" For more information on this or any other computing topic,
please visit our Digital Library at www.computer.org/csdl.