1034 IEEE TRANSACTIONS ON COMPUTERS, VOL. 70, NO.

7, JULY 2021

Contour: A Process Variation Aware

Wear-Leveling Mechanism for Inodes
of Persistent Memory File Systems
Xianzhang Chen , Member, IEEE, Edwin H.-M. Sha , Senior Member, IEEE, Xinxin Wang,
Chaoshu Yang , Weiwen Jiang , and Qingfeng Zhuge , Member, IEEE

Abstract—Existing persistent memory file systems exploit the fast, byte-addressable persistent memory (PM) to boost storage
performance but ignore the limited endurance of PM. Particularly, the PM storing the inode section is extremely vulnerable for the
inodes are most frequently updated, fixed on a location throughout lifetime, and require immediate persistency. The huge endurance
variation of persistent memory domains caused by process variation makes things even worse. In this article, we propose a process
variation aware wear leveling mechanism called Contour for the inode section of persistent memory file system. Contour first enables
the movement of inodes by virtualizing the inodes with a deflection table. Then, Contour adopts cross-domain migration algorithm and
intra-domain migration algorithm to balance the writes across and within the memory domains. We implement the proposed Contour
mechanism in Linux kernel 4.4.30 based on a real persistent memory file system, SIMFS. We use standard benchmarks, including
Filebench, MySQL, and FIO, to evaluate Contour. Extensive experimental results show Contour can improve the wear ratios of pages
417.8
and 4.5
over the original SIMFS and PCV, the state-of-the-art inode wear-leveling algorithm, respectively. Meanwhile, the
average performance overhead and wear overhead of Contour are 0.87 and 0.034 percent in application-level workloads, respectively.

Index Terms—Persistent memory file system, wear leveling, process variation, persistent memory, metadata management

1 INTRODUCTION [18]. Hence, the lack of wear-leveling mechanism in existing

the development of persistent memory (PM)
W ITH
technologies, such as Phase Change Memory
(PCM) [2] and Intel’s Optane DC persistent memory [3],
many persistent memory file systems [4], [5], [6], [7], [8], [9]
are proposed to exploit the advanced characteristics of PM,
especially non-volatility, low latency, and byte-addressability.
Existing persistent memory file systems, such as PMFS [5] and
SIMFS [10], focus on designing highly efficient file I/O stacks
and data consistency guarantees but ignore the critical disad-
vantage of PM, i.e., limited write endurance [11], [12]. For
example, the endurance of PCM is about 108 [13]. What’s
worse, the write endurances of memory cells in a PM show
huge variance due to process variation [14], [15], [16], which
makes the memory cells can only sustain 106 -108 writes [17],
 Xianzhang Chen is with the College of Computer Science, Chongqing
University, Chongqing 400044, China, and also with the School of
Computing, National University of Singapore, Singapore 117417, Singapore.
E-mail: xzchen109@gmail.com.
 Xinxin Wang and Chaoshu Yang are with the College of Computer
Science, Chongqing University, Chongqing 400044, China.
E-mail: {wangxinxcq, yangchaoshu}@gmail.com.
 Edwin H.-M. Sha and Qingfeng Zhuge are with the School of Computer
Science and Software Engineering, East China Normal University, Shanghai
200062, China. E-mail: {edwinsha, qfzhuge}@gmail.com.
 Weiwen Jiang is with the College of Engineering, University of Notre
Dame, Notre Dame, IN 46556 USA. E-mail: jiang.wwen@gmail.com.
Manuscript received 22 Dec. 2019; revised 22 May 2020; accepted 10 June 2020.
Date of publication 15 June 2020; date of current version 9 June 2021.
(Corresponding author: Xianzhang Chen.)
Recommended for acceptance by F. Douglis.
Digital Object Identifier no. 10.1109/TC.2020.3002537
0018-9340 © 2020 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission.
See ht_tps://www.ieee.org/publications/rights/index.html for more information.
persistent memory file systems may cause fatal damage to the
underlying PM device.
Particularly, the inode section of persistent memory file
system is extremely fragile to the wear issues. There are two
roots in the severe wear leveling problem of the inodes in
existing persistent memory file systems. On one hand, the
inode structure is one of the most frequently updated data
structure in a file system and the inode updates require imme-
diate persistency. The inode of a file is responsible for main-
taining the attributes of the file, typically including file size,
last access time, last modification time, and access-control
information. In the POSIX file system interfaces, most file sys-
tem operations either directly update the inodes, such as link()
and unlink() that modify the link count in the inode of the tar-
get file, or write both of the file data and the corresponding
inodes. For example, write() and mkdir() update the file data
and the corresponding inode of a data file or a directory file,
respectively.
What is more important is that the inodes in existing
persistent memory file systems are stored on fixed loca-
tions throughout their lifetime. Existing persistent mem-
ory file systems, regardless of whether they organizes the
inode section by an array structure [10], [19] or tree struc-
ture [5], [9], never move the inode of a file to different
memory cells. Hence, the update operations of each inode
perform all the writes to the same physical memory
address, which can easily wear out the corresponding
memory cells and cause data loss. Unfortunately, most
existing file systems on PMs do not consider wear-leveling
designs for inodes. Previous work for the wear leveling of

Authorized licensed use limited to: R V College of Engineering. Downloaded on December 28,2021 at 16:04:01 UTC from IEEE Xplore. Restrictions apply.
CHEN ET AL.: CONTOUR: A PROCESS VARIATION AWARE WEAR-LEVELING MECHANISM FOR INODES OF PERSISTENT MEMORY FILE...
inodes [1] is unaware of variant endurance caused by pro-
cess variation.
In this paper, we focus on solving the wear-leveling
problem for the inode section of file systems on the PM
device with process variation. We present a process-varia-
tion-aware mechanism called Contour, that balances the
wear of the physical pages storing inodes by fitting the
unevenly updated inodes to the memory pages with var-
ied endurance. Contour consists of two techniques. First,
we enable the migration of inodes by inode virtualization.
Inode virtualization uncouples inodes from “fixed” physi-
cal locations by a “virtual inode” layer between the logical
inodes and the inode slots on physical PM. The mapping
relation between a logical inode and the corresponding
virtual inode is recorded as an offset in a deflection table.
Given the inode number (i-number) of a logical inode, the
file system can access the actual inode slot using the offset
and the virtual address of the corresponding virtual inode.
Therefore, inode virtualization enables a logical inode1
to be moved to different physical locations by changing
its offset.
Second, we tackle the unbalanced inode updates by an
inode migration mechanism. A PM device is divided into
multiple memory domains that show various endur-
ance [14]. The memory cells in the same domain, however,
have the same endurance. Take the features of persistent
memory domains in mind, we propose cross-domain migra-
tion and intra-domain migration for migrating inodes. The
cross-domain migration algorithm controls the balance of
writes on the memory domains by “write budgets”. We
migrate an inode to a proper memory domain according to
the write frequency of the inode and the relative write
budgets of memory domains. In intra-domain migration,
we evenly distribute the writes of inodes to the inode slots
in the same domain by migrating inodes to less-wear slots.
We implement the proposed Contour mechanism in
SIMFS [10], a typical file system for managing PM device, as
a case study in Linux kernel 4.4.30. We evaluate the wear-
leveling effect and overheads of Contour using the typical
benchmarks, including Filebench [20], MySQL [21], and
Flexible I/O [22]. The wear-leveling effect is expressed by
wear ratio, i.e., write counts/endurance.
The experimental results show that the proposed Con-
tour mechanism brings significant wear-leveling improve-
ment for inodes. Compared with the original SIMFS and
PCV [1], the state-of-the-art algorithm for the wear-leveling
of inodes, Contour shows 417.8
and 4.5
lower standard
deviation of wear ratios of pages, respectively. The maxi-
mum wear ratio of pages in Counter can be 2740.3
and
2.5
lower than these of original SIMFS and PCV, respec-
tively. Meanwhile, the average performance overhead and
wear overhead of Contour are 0.87 and 0.034 percent in
application-level workloads, respectively.
The main contribution of this paper includes:
 We propose a process variation aware wear-leveling
mechanism, Contour, to protect PM device from being
damaged by frequently updated inodes.
1. In the following, “inode” is stand for logical inode without special
explanation.
Authorized licensed use limited to: R V College of Engineering. Downloaded on December 28,2021 at 16:04:01 UTC from IEEE Xplore. Restrictions apply.
1035
 We design a process variation aware inode migra-
tion algorithm to balance the wear of inodes across
the memory domains.
 We implement Contour in Linux kernel based on a
real persistent memory file system, SIMFS.
 Extensive experimental are conducted with standard
benchmarks. The experimental results show that
Contour significantly improves the wear leveling of
inodes with negligible overhead.
The reminder of this paper is organized as follows. Section 2
introduces the background and shows the motivational exam-
ple for managing the inodes of file system on persistent mem-
ory. In Section 3, we present the design and implementation of
the proposed Contour mechanism. Section 4 evaluates the
proposed Contour by extensive experiments. In Section 5, we
summarize the studies related to this work. Finally, Section 6
concludes the paper.
2 BACKGROUND AND MOTIVATION
2.1 Endurance Variation of Persistent Memory
A common problem of persistent memory technologies is
limited endurance. To make things worse, the endurance of
persistent memory cells show huge variation due to process
variation [14], [15], [17], [23], [24]. Previous works [14], [18]
divide the memory space into multiple domains that the
memory cells in a domain have the same endurance. Accord-
ing to [17], [18], the endurance of domains in a persistent
memory device approximately follows a linear distribution.
For a 2 GB persistent memory device, the endurance of
domains ranges from 3:0
106 to 1:7
108 . The endurance
of the weakest domain can be 56 times lower than that of the
strongest domain. Thus, the persistent memory device can
fail quickly without protection for the weak persistent mem-
ory cells will be worn out early.
The size of a domain is determined by the architecture of
persistent memory device. The persistent memory device
usually organizes the memory cells in a hierarchical manner,
including sub-arrays, sub-banks, and banks. The memory
cells residing in the same sub-array show the same endur-
ance [14]. Hence, we regard a memory sub-array as a
domain, which is generally 4MB. In this case, the inode zone
in a file system consists of multiple memory domains with
huge endurance variation. For example, the size of an inode
zone is about 2.5GB for a 256GB persistent memory device
since the file system generally reserves 1 percent space as
inode zone. Such an inode zone covers 640 domains with
more than 56
variations from the weakest domain to the
strongest domain. In summary, the inode management of
persistent memory file systems should fully consider the
endurance variation of the memory cells.
2.2 Existing Persistent Memory File Systems
With the development of new persistent memories, a set of
persistent memory file systems have been proposed to break
the traditional I/O bottleneck. The design of existing persis-
tent memory file systems mainly focus on improving perfor-
mance or data consistency [4], [9], [25]. For example, BPFS [4]
proposes “short-circuit shadow paging”, an improved
shadow paging mechanism exploiting the byte-addressabil-
ity of persistent memory, to provide highly efficient data
1036
consistency guarantees. SIMFS [10] presents “file virtual
address space” to boost the file access performance by utiliz-
ing virtual address space and hardware MMU. NOVA [9] is
a log-structured file system that develops efficient atomic
operations for data and metadata. SoupFS [26] optimizes soft
updates on persistent memory taking advantages of the
advanced features of PM. SplitFS [27] and ZoFS [28] seek
user-space file access solutions for improving performance
with consistency and security guarantees.
The limited endurance problem of persistent memory,
however, are not well considered in the design of existing
persistent memory file systems. The underlying persistent
memory can be easily wear out by common file system
operations, especially the pages for storing inodes. It is
because that inodes are the most frequently updated data in
a file system. The inodes are frequently updated by many
commonly-used file operations, including write, read, mkdir,
rmdir, rename, link, symlink, unlink, chmod, chown, open (cre-
ate), and delete. Unfortunately, the wear out problem of
pages for storing inodes is ignored by existing persistent
memory file systems. Therefore, we focus on the wear level-
ing problem of inodes in this paper.
2.3 Wear Leveling Problem of Inodes
There are two types of designs for inode section in existing
persistent memory file systems: tree structures and arrays.
For example, PMFS [5], SanGuo [6], and HiNFS [7] use B-
tree to organize all the inodes. The inodes are stored in the
leaf node of the B-tree. SCMFS [19] and SIMFS [10] organize
the inodes by an array. Beyond these data structures, we
have three observations on the inode management of persis-
tent memory file systems.
Observation 1. The Inodes of Most Existing Persistent Memory
File Systems Never Move Throughout Their Lifetime. In most
existing persistent memory file systems, inode section is
fixed once initialized. Therefore, most of the writes to a
given inode are performed on a single inode slot of a
physical page, which makes the PM fragile to heavily
updated inodes.
Observation 2. The Write Counts of Inodes Generally Vary a
Lot. For example, most system files are read-only and their
inodes are seldom updated, whereas some directory files
and user’s data files are frequently updated. This common
feature of systems result in unbalanced write distribution
over the pages for storing inodes. Since each page can store
multiple inodes, an inappropriate inode combination on a
page can expedite the damage of the page.
Observation 3. The Updates Upon Inodes are Immediately Per-
formed on Persistent Memory. Existing persistent memory
file systems use hardware primitives, such as pm_barrier [5]
and PCOMMIT [9], [29], [30], to ensure writing ordering of
inodes. With these instructions, a file system cannot cache
dirty inodes in CPU cache, DRAM, or even the buffer in
memory controller, but writes the new data of inodes to
PM immediately. As a result, the updates of inodes will
bring inevitable damage to the PM.
We show a motivational example by running the
application-level workload OLTP of Filebench [20] on PMFS [5]
Authorized licensed use limited to: R V College of Engineering. Downloaded on December 28,2021 at 16:04:01 UTC from IEEE Xplore. Restrictions apply.
IEEE TRANSACTIONS ON COMPUTERS, VOL. 70, NO. 7, JULY 2021
Fig. 1. Write distribution of OLTP on PMFS [5] and SIMFS [10].
and SIMFS [10]. The experimental results are shown Fig. 1. In
the two file systems, all the writes of the inodes are performed
on merely tens of pages. OLTP writes each of the few pages
more than 30,000 times in just one minute while left the rest
thousands of pages unwritten. Therefore, it is necessary to
develop wear-leveling mechanism for protecting the memory
cells from the frequently updated inodes. We believe the way
to tackle the wear leveling problem of inodes is to dynamically
migrate the inodes.
3 CONTOUR DESIGN AND IMPLEMENTATION
In this section, we propose a new design called Contour, to
achieve wear-leveling of pages by distributing the writes of
inodes evenly over multiple physical persistent memory
pages. Contour is composed of two parts: Inode virtualiza-
tion enables a logical inode to be remapped to any physical
inode; Inode migration selects and migrates the heavily-
written inodes to less written pages.
3.1 Inode Virtualization
In a file system, each inode has a unique ID called “inode
number” (i-number). In existing persistent memory file sys-
tems, i-number is an index for seeking the corresponding
inode in the inode section. We use the inode section of
SIMFS [10] as an example.
The inode section of SIMFS is shown in Fig. 2a. SIMFS
organizes the logical inodes as an array-structured inode
section. An i-number is the index of an element in the array.
The sizes of inodes are the same. The array of inode section
is stored on multiple physical memory pages. Each physical
page of persistent memory can be viewed as an array of
inode slots. Once the inode section is created, the mapping
from the logical inodes to the inode slots is fixed, i.e., each
inode slot is one-to-one bounded with a logical inode
throughout the lifetime. As a result, the file system is unable
to handle the wear leveling problem of inodes with the per-
manent bind between inode slots and logical inodes.
In this section, we propose to change the permanent bind
to a convertible relation by a new design of inode virtualiza-
tion. As shown in Fig. 2b, Contour adds a new virtual inode
layer between logical inode (represented by i-number) and
inode slot. The Virtual Inodes (VInodes) forms an array-
structured virtual space, “inode virtual address space”, that
each VInode has a unique offset in virtual space. The inode
virtual address space uses page table to map the virtual
inode to the physical pages of inode slots, where each
VInode is bound with an inode slot. The inode virtual
address space has a beginning virtual address. The file
CHEN ET AL.: CONTOUR: A PROCESS VARIATION AWARE WEAR-LEVELING MECHANISM FOR INODES OF PERSISTENT MEMORY FILE...
Fig. 2. Comparison of conventional inode section and the proposed virtualized inode section.
system can get the virtual address of a VInode using its off-
set and the inode virtual address space’s beginning address.
In Contour, we realize the endurance variation of memory
domains in the persistent memory, whereas existing persis-
tent memory file systems are unaware of that.
On the other side, inode virtualization mechanism maps
logical inodes to VInodes via a deflection table. In the deflec-
tion table, each logical inode has an entry indexed by its i-
number. The entry of a logical inode maintains the offset to
the corresponding VInode. With the help of deflection table,
a logical inode can be mapped to different VInode dynami-
cally by changing its offset. Similar to array-structured inode
sections, tree-structured inode sections can also be virtual-
ized by constructing a deflection table for inodes in the leaf
nodes. Using the proposed Contour mechanism, the file sys-
tem access an inode in three steps: (1) The file system gets the
offset of the logical inode via its i-number; (2) The file system
calculates the virtual address of the inode by adding its offset
with the beginning address of the inode virtual address
space; (3) The file system accesses the inode slot using the
virtual address of inode and the hardware memory manage-
ment unit in the processor.
In implementation, the deflection table is an array that
corresponds to the logical inodes of the file system. The
inode virtual address space is a segment of the kernel vir-
tual address space. The overhead for accessing an inode in
Contour is only an additional memory access and an addi-
tion, which is negligible for the file operation.
3.2 Inode Migration
Given a PM page (domain), we define wear ratio as the por-
tion of the write counts performed on the page (domain)
with respect to the corresponding endurance. For an inode
section across the memory domains with endurance varia-
tion, wear leveling basically means that the wear ratio of the
physical pages for storing inodes are the same. In this case,
we propose a two-layer strategy for the wear leveling of
inode zone: first, we design a cross-domain migration algo-
rithm for balancing the wear ratio of different domains; sec-
ond, we present intra-domain migration algorithm to even
distribute the writes on the PM pages in the same domain,
which has identical endurance.
3.2.1 Cross-Domain Migration
Given a wear ratio, the affordable numbers of writes on the
memory domains are various. Assume the writes upon a
Authorized licensed use limited to: R V College of Engineering. Downloaded on December 28,2021 at 16:04:01 UTC from IEEE Xplore. Restrictions apply.
1037
memory domain are evenly distributed over the pages in
the memory domain, a stronger memory domain can bear
more writes than a weaker memory domain with the same
wear ratio. Thus, it is necessary to move the inodes with dif-
ferent writes counts between the domains such that the
wear ratio of memory domains are balanced.
In this section, we propose a cross-domain migration algo-
rithm to balance the wear ratios of different memory domains.
To control the number of wear ratios of memory domains, we
define a Write Budget Line (WBL) for all the domains. WBL is
an adjustable wear ratio constraint. Within the write budget
line, each domain dj has its own “Write Budget” WBj , which
indicates the maximum number of writes can be performed
on all the memory pages in the domain. On one hand, the
write budgets of memory domains are various due to endur-
ance variation. On the other hand, the updating counts of
inodes are diverse for the files have different access features.
The main idea of the proposed cross-domain migration is to
hold the write budget line for all the domains by migrating
inodes between the domains. We now discuss two key con-
cerns of the idea.
Matching Inodes and Domains. A basic observation is that
migration operations actually bring additional writes to the
memory domains. We need to avoid migration operations
as far as possible. From this perspective, to achieve balanced
wear ratios and reduce the potential inode migrations, the
ideal case is to place the inodes with most writes on the
strongest memory domains and so on. To achieve the ideal
case, we should match the inodes and memory domains
according to the potential updating counts of inodes and
the available write budget of memory domains.
The potential updating counts of inodes is related to the
I/O patterns of applications [31], such as open-write-close,
write-seek, and aggregate-write. For example, an applica-
tion adopting open-write-close pattern, such as CESM [32],
generally opens a file, write it, and then close it. Thus, it will
not cause much writes to the same inode. On the contrary,
an application employing aggregate-write pattern, such as
MADCAP [33] and the file log in Webserver [20], opens a
file and contiguously appends it, which will bring lots of
writes to the inode of the corresponding file. Hence, the
potential updating counts of an inode can be reflected by
the write frequency of the inode. We define damaging flow
DF T ðIi Þ as the write counts of inode Ii in a timing period T .
The length of each timing period is set to t.
With the damaging flow, we can judge the criticality of
each inode to the memory domain, i.e., how harmful an
1038 IEEE TRANSACTIONS ON COMPUTERS, VOL. 70, NO. 7, JULY 2021

inode can be to the current memory domain. To determine In period T , the expected maximum write counts of domain
T 1
the criticality of an inode, we use the average damaging dj is t  NWBj  DFavg . Thus, the expected maximum write
flow of previous periods as the reference. The average dam- counts WI ðIi Þmax of inode Ii in memory domain dj is:
T
T
aging flow DFavg of period T is calculated by Equation (1).
T 1
NWBj  DFavg
T
þ T 1 WI T ðIi Þmax ¼ t  : (5)
WItotal =t DFavg NP  NS
T
DFavg ¼ ; (1)
2
It is because that each domain has NP  NS inode slots and
T
where WItotal is the total write counts of all the inodes in we assume the inode slots sustain the same write counts.
0
period T . Initially, DFavg equals zero. Then, we calculate the When the write counts of an inode Ii reaches the expected
T
criticality factor ‘i of inode Ii in period T as follows: maximum write counts, the file system calculates the current
criticality factor of ‘curr
i of Ii . According to Equations (2) and
DF T ðIi Þ (5), ‘curr is calculated by:
‘Ti ¼ T
 NP  NS : (2) i
DFavg
WI T ðIi Þmax  NP  NS t  NWBj
‘curr ¼ T 1
¼ ; (6)
where NP is the number of pages in each memory domain, i
Dti  DFavg Dti
which is 1024 for a memory domain is 4MB and a page is
4KB. NS is the number of slots in each page, which is a fixed where Dti is the time passed from the beginning of period T .
value for any file system. In Equation (2), we amplify the rel- Then, we migrate Ii to the corresponding memory domain
ative damaging flow of an inode by NP  NS times for the crit- according to ‘curr
i and Equation (3). To avoid migrations as
ical factor reveals the importance of an inode to the whole much as possible, we do not proactively migrate the ino-
memory domain. des that do not reach the minimum write counts in the
In each period, we match an inode with a memory period. We simply move it to the best-fit memory domain
domain according to the criticality factor of the inode and when the current memory domain is running out of
the endurance of the memory domains. During a period, if space.
the criticality factor of an inode reaches a higher level, we Adjusting the Write Budgets. The balance of the whole
move it to a stronger memory domain. In the cross-domain write distribution on memory domains is controlled by
migration algorithm, we setup D criticality levels for match- write budget line. Each time we set the new write budget
ing the inodes with memory domains, where D is the num- line, we call it a new round R. At the beginning of round R,
ber of memory domains. The criticality levels are one-to-one the file system calculates the total write budget WBR
total :
matched with the memory domains. The criticality levels is
defined by a step function f : Rþ ! N: X
D

total ¼ WBL
WBR Ej þ ðWBtotal  WItotal Þ;
R R1 R1
(7)
8 j¼1
>
> 1; if ‘Ti 2 A1
<
2; if ‘Ti 2 A2
fð‘Ti Þ ¼ : (3) where Ej , j ¼ 1; 2; ldots; D, is the endurance of domain dj .
>
> :::
: During the round, the file system records the total writes
D; if ‘i 2 AD
T
R
counts WItotal of inodes and estimates the percentage of
remaining total budget. Once the remaining total budget
where Aj (j ¼ 1; 2; ldots; D) are the ranges of the criticality fac-
reaches a threshold, the file system will start a new round. A
tors of inodes. The ranges of criticality factor is determined by
busy system needs a larger threshold to ensure that the file
the relative write budgets of memory domains. First, we take
system has enough time to reset the budgets. In our imple-
the average write budget of the memory domains as the base-
mentation, we always set the write budget line and the new
line for the criticality factor is normalized to the average dam-
round threshold to 1 and 5 percent, respectively. With this
aging flow of the system. Then, we calculate the normalized
configuration and the 2 GB persistent memory device from
write budget value (denoted by NWBj for domain dj ) of each
[18], each round can support about 4:42
108 inode updates.
domain over the baseline. Finally, we set the range of memory
The file system can support another 2:21
107 inode updates
domain dj to Aj ¼ ðNWBj1 ; NWBj , except that A0 ¼ ½0;
before starting a new round, which is enough for most
NWB1  and AD ¼ ðNWBD1 ; þ1Þ.
workloads [9].
Migrating the Inodes. Basically, we do not need to migrate
At the beginning of each round, we also need to adjust the
an inode if its damaging flow matches the located memory
write budget of each memory domain. A memory domain
domain. Thus, there is an expected damaging flow for each
may overuse the prior write budget or have surplus. Thus,
inode. In a period T , we migrate an inode Ii if its write counts
we need to rectify the ranges in Equation (3) according to the
exceeds its expected damaging flow DF T ðIi Þexp in the current
actual write budgets of the memory domains: First, the file
period. DF T ðIi Þexp is based on the expected criticality factor
T 1 system finds out the average write budget WBR avg of the ino-
of Ii and the average damaging flow DFavg in the prior
des in round R; then, the file system calculates the normal-
period. The expected criticality factor of Ii in memory domain
ized write budget NWBR j for each memory domain by
dj is NWBj1 < ‘Ti  NWBj according to Equation (3). Given
T 1 dividing WBR avg ; finally, we reset the boundaries of the ranges
a period T , DFavg is a known variable. Thus, the predicted
in Equation (3) according to the normalized write budgets of
damaging flow can be calculated by Equation (4).
memory domains.
T 1 T 1 In a certain round, a memory domain dj may have a write
NWBj1  DFavg < DF T ðIi Þexp  NWBj  DFavg : (4)
budget equal to or larger than that of the adjacent stronger

Authorized licensed use limited to: R V College of Engineering. Downloaded on December 28,2021 at 16:04:01 UTC from IEEE Xplore. Restrictions apply.
CHEN ET AL.: CONTOUR: A PROCESS VARIATION AWARE WEAR-LEVELING MECHANISM FOR INODES OF PERSISTENT MEMORY FILE... 1039

memory domain djþ1 , i.e., NWBR j  NWBjþ1 . In this case, the

R
a less written inode slot for inode Ii . Here we also consider
file system resets the boundaries of range Aj and Ajþ1 by: the balance of writes on physical pages. First, we check if
there is a free slot in a page that not only the write counts
NWBR þNWBR
j1 < Aj  ;
j
NWBR 2
jþ1
of the pages is less than the reference line but also the slot has
NWBR þNWBR
(8) not reach its write budget (Lines 4-7). If the condition is
j
2
jþ1
< Ajþ1  NWBR
j : failed, we try to find such a free slot in the pages that their
write counts have not reaching its write budget (Lines 8-12).
The new boundaries increase the possibility of accepting ino- If there is still no available slot and total write counts already
des for the weaker domain and place the inodes with higher excess the budget of the domain, we find a free slot from a
damage flow in the stronger domain, through which the domain that has available budget following best-fit principle
usage of write budgets for the domains may be balanced. (Lines 13-16). Otherwise, we swap inode Ii with the least
recently written inode Icand in the domain (Lines 17-20). If we
3.2.2 Intra-Domain Migration find a satisfiable empty slot, we migrate Ii to it.
As aforementioned, the memory cells in the same domain
have the same endurance. Hereby, in-domain wear leveling Algorithm 1. The Intra-Domain Migration Algorithm
means that the write counts of the physical pages for storing
Input: WSðIi Þ: the write counts of the inode slot storing inode
inodes are the same. In this case, the maximum write count
Ii ; WPavg : the average write counts of the physical pages;
of the pages is the minimum and the standard deviation of
Output: The relative storing position Offseti for Ii ;
the write count of pages is zero. Therefore, the in-domain 1: if WSðIi Þ  a then
wear leveling problem of inodes can be defined as follows: 2: continue;
Given n inodes and NP physical pages that each page has 3: else
NS slots for storing inodes, how to place n inodes on NP 4: for Pp has free slot and WPp  WPavg do
pages such that the standard deviation of the write counts 5: if a slot Slots on Pp satisfies WSs < a then
of pages is minimized? The key is to find out a proper com- 6: Slotcand Slots ;
bination of inodes for each page such that the pages can 7: break;
achieve the same write count. The write counts of inodes, 8: if Slotcand ¼ NULL then
however, are dynamically changing in a run-time system. 9: for Pp has free slot and WPp < a  NS do
Therefore, we propose to solve the problem by dynamically 10: if a slot Slots on Pp satisfies WSs < a then
migrating frequently updated inodes during run-time. 11: Slotcand Slots ;
There are two observations for designing the dynamic 12: break;
migration method. First, from the perspective of wear level- 13: if Slotcand ¼ NULL then
ing, it is not necessary to migrate an inode if the write 14: if WPavg  NP  WBj then
counts of the underlying inode slot has not reach its write 15: Slotcand a free slot in the best-fit domain that has
budget. Second, only the future number of writes of an available budget;
inode impacts the page that the inode will be placed after 16: else
migration. Thus, a migration needs to consider the write 17: Icand the least recently written inode;
counts of slots and the features of inodes simultaneously. 18: Slotcand SlotðIcand Þ;
19: Swap Ii with Icand ;
Based on the observations, we propose intra-migration
20: Exchange Offseti and OffsetðSlotcand Þ;
algorithm for migrating the inodes. We use the average write
21: return;
counts of all pages in a memory domain (denoted by WPavg )
22: Migrate Ii to Slotcand ;
as the “reference line”. The main idea of intra-migration 23: Offseti OffsetðSlotcand Þ;
algorithm is to achieve wear-leveling by aligning the write
counts of each page to the reference line. The intra-migration
In each round, the threshold a is a constant. In worst
algorithm determines whether an inode need to be migrated
case, the intra-domain migration algorithm need to traverse
or not by comparing the write counts of the underlying inode
all the inodes in the memory domain. Hence, the timing
slot and the reference line. The detailed algorithm is shown
complexity of the proposed intra-domain migration algo-
in Algorithm 1.
rithm is OðNP  NS Þ.
PR
Ej WBLr
a¼ r¼1
; (9) 3.3 Discussion
NP  NS
The deflection table in Contour is implemented as an array.
The proposed intra-migration algorithm uses threshold a Each entry in the deflection table stores the offset by a 4-
to judge if an inode slot is overly wore. a is the write budget byte integer, which can support 232 inodes. For a file system
of each inode slot in domain dj . We hope that the expected with one million inodes, the space overhead of the deflec-
writes in each round (i.e., write budget) are evenly distrib- tion table is merely 3.82 MB. The wear of the deflection table
uted on the inode slots in the domain. Thus, in each round is much lighter than the wear of inode section since the off-
R, we set a by Equation (9). set is modified only when an inode is migrated. We main-
In Algorithm 1, WSð Ii Þ is the write counts of the inode tain the counters for storing the write counts of each logical
slot that storing inode Ii throughout all the existing rounds inode, each page, each domain, and all the domains in
(Line 1). If WSð Ii Þ achieves a, Ii needs to be migrated to pro- DRAM-based main memory and write the dirty counters
tect the corresponding inode slot. In the following, we find back to the PM at the end of each period. Thereby, the

Authorized licensed use limited to: R V College of Engineering. Downloaded on December 28,2021 at 16:04:01 UTC from IEEE Xplore. Restrictions apply.
1040
TABLE 1
Characteristics of Filebench Workloads and MySQL
# of files Average Threads
(Data/Log) file size
fileserver 200K/0 4KB 50
webserver 200K/100 4KB 100
OLTP 200K/1 10MB 12
MySQL 30M/20 8KB 50
The selected workloads have different read/write ratios and access patterns.
damage of these counters is alleviated to a lower level than
the inodes. Even a system failure occurs, we will only miss
the write counts in a short timing period, which is negligible
in the lifetime of PM.
There is a wear-leveling mechanism called Kevlar [34]
that also deflects virtual and physical pages and migrates
hot data. However, since Kevlar is a universal technology
for applications and Contour is dedicated to the inodes of
persistent memory file systems, the granularity, migration
mechanism, and wear-leveling strategy of Contour and
Kevlar are different. For the wear-leveling of inode section
in persistent memory file systems, Contour is promised to
be more efficient and economic than Kevlar.
4 EVALUATION
4.1 Experimental Setups
In this section, we evaluate the wear-leveling effect and effi-
ciency of Contour. We implement the proposed Contour
mechanism in Linux kernel 4.4.30 based on SIMFS [10], a
typical persistent memory file system. We compare Contour
with two file systems: 1) The original SIMFS that has no
wear-leveling mechanism (denoted by NoWL) for inodes; 2)
PCV, the state-of-the-art wear-leveling algorithm for ino-
des [1], which is also implemented in SIMFS. PCV balances
the write counts of pages by migrating the inodes that meets
the thresholds. We set the period T of Contour to 10 seconds
for it can achieve the best trade-off between domain wear
ratio and migration overhead.
The experiments are conducted on a server that equipped
with an Intel Xeon E5-2640 v4 2.40 GHz processor and
256 GB DRAM. We use PMEM [35] to emulate 64 GB DRAM
as persistent memory of the file systems. To compensate the
difference of write latency between PM and DRAM, we
adopt the RDTSC instruction to read timestamp and add
software latency after the clflush instruction. Similar to [7],
the write latency of PM is set to 200 ns and the read latency is
the same to DRAM. In each file system, the inode section has
32 memory domains that can support one million files. The
system runs on Ubuntu 16.04 with Linux kernel 4.4.30.
In the experiments, we first use Filebench [20], a widely
used benchmark for file systems, and MySQL to evaluate the
effect of wear leveling. We select three typical application-
level workloads from Filebench: fileserver, webserver, and
OLTP. fileserver emulates a file server that hosts 50 threads.
Each thread performs a sequence of file operations, includ-
ing create, write, read, append, delete, and stat. webserver simu-
lates a Web server that uses 100 threads to open, read, and
close multiple files in a directory tree, along with a set of file
appends to simulate the web log. OLTP emulates a TPC-C
Authorized licensed use limited to: R V College of Engineering. Downloaded on December 28,2021 at 16:04:01 UTC from IEEE Xplore. Restrictions apply.
IEEE TRANSACTIONS ON COMPUTERS, VOL. 70, NO. 7, JULY 2021
TABLE 2
The Number of Writes After Running 10 Hours on NoWL
I/O size NoWL fileserver webserver OLTP MySQL
(r/w)
Total writes 6.80E+08 5.00E+08 4.10E+08 4.00E+08
1MB/16KB Max on domains 21345585 20198403 35243334 16559976
1MB/256B Max on pages 25788 20197700 24028600 16559700
2KB/2KB
32B
workload to test the performance of the transactions that
poses small random reads and writes on a file system using
10 database writers, 1 database readers, and one log writer.
Furthermore, we install MySQL on the file systems and eval-
uate wear leveling by the mysqlslap [21] benchmark. The
detailed characteristics of the workloads are shown in
Table 1. Then, we measure the performance overhead by Fil-
ebench and Flexible I/O (FIO) [22] respectively. Finally, we
evaluate the wear overhead caused by inode migration.
4.2 Effect of Wear Leveling
In this subsection, we show the effect of wear leveling from
domain level and page level. First, the domain-level wear
ratios reveal the trend of wear of the whole memory
domains for inodes. Second, the page-level write distribu-
tions show the wear-leveling effect of the approaches at one
point. Table 2 shows the absolute numbers of writes after
running the workloads for 10 hours without support of
wear leveling. For webserver, OLTP, and MySQL, the writes
on the most heavily written domain mainly come from the
most heavily written page, indicating that the write distri-
butions in some domains are highly unbalanced. The max
numbers of writes on pages for webserver, OLTP, and
MySQL already reach 108 , showing that some pages are
closing to be damaged. Hence, it is necessary to evaluate the
wear-leveling effect of different approaches.
4.2.1 Domain-Level Wear Ratio
In the experiments, we capture the wear ratios of domains
after running the workloads for one hour, five hours, and
ten hours, respectively. We define “wear gap” as the differ-
ence by subtracting the minimum wear ratio of memory
domains from the maximum wear ratio. Fig. 3 shows the
experimental results. For all the workloads, Contour stably
maintains balanced domain wear ratios at each moment.
After running the fileserver workload for 10 hours, the
maximum wear ratios of NoWL, PCV, and Contour are 22.2,
27.8, and 9.0 percent respectively, as shown in Fig. 3a. Mean-
while, the Standard Deviation (SD) of the wear ratios by
NoWL, PCV, and Contour are 4.6, 5.7, and 0.64 respectively.
Thus, the wear-leveling effect of Contour is 7.4
and 9.5

better than that of NoWL and PCV respectively. The wear
gaps in Contour are 0.95, 0.74, and 1.64 percent for one hour,
five hours, and ten hours respectively. The wear gaps in
NoWL are 3.4, 12.4, and 17.2 percent respectively, which
means that the wear of domains becomes more unbalanced
along with the use of the system. Similarly, the wear gaps in
PCV also grows from 3.2 percent (at 1h), 12.3 percent (at 5h)
to 21.5 percent (at 10h). The results show that only Contour
maintains balanced writes on the domains along with system
running for Contour is aware of process variation.
CHEN ET AL.: CONTOUR: A PROCESS VARIATION AWARE WEAR-LEVELING MECHANISM FOR INODES OF PERSISTENT MEMORY FILE...
Fig. 3. Comparison of wear ratio of domains in NoWL, PCV, and
Contour.
Figs. 3b, 3c, and 3d show that Contour still achieves bet-
ter wear leveling than other approaches on webserver,
OLTP, and MySQL. For example, the wear gaps in Contour
are less than 1.4 percent by running MySQL. The wear
gaps in NoWL are 6.5, 8.62, and 10.8 percent at 1h, 5h, and
10h, respectively. The wear gaps in PCV also grows from
0.79 percent (at 1h), 3.8 percent (at 5h) to 10.1 percent (at
10h). The SD of the wear ratios by NoWL, PCV, and Con-
tour at 10h are 3.1, 2.7, and 0.39 respectively. It means that
for MySQL, the wear-leveling effect of Contour is 6.9
and
5.8
better than that of NoWL and PCV respectively.
In fileserver and webserver, the lines of wear ratios by
NoWL is quite smooth, whereas the lines in OLTP and
MySQL fluctuate a lot. Both fileserver and webserver gener-
ally commit just few updates to the inode of a single file that
make the variance of write counts of the inodes are natu-
rally small. On the contrary, many files in OLTP and
MySQL have much more updates than other files, which
cause huge variance in the write counts of inodes.
In webserver, OLTP, and MySQL, the lines of wear ratios
by PCV are smoother than that of NoWL. It is because that
PCV balances the write counts of pages by migrating
heavily-written inodes. However, due to the unawareness
of process variation, PCV still cause heavy damage to the
weak domains although the writes are evenly distributed.
4.2.2 Page-Level Write Distribution
Fig. 4 shows the write counts (denoted by dark dots) and
wear ratios (denoted by blue lines) of the pages in the inode
sections after running the workloads for ten hours. In the
sub-figures of Fig. 4, each dot represents the write count of
a page and the line shows the wear ratio of the correspond-
ing page. The write counts of pages in Contour generally
follows the endurance variation of memory domains. The
wear ratio of pages in Contour are highly balanced, e.g., the
SDs of wear ratios are less than 0.95 on all workloads. On
the contrary, the wear ratios of pages in NoWL and PCV
shows large variation.
Authorized licensed use limited to: R V College of Engineering. Downloaded on December 28,2021 at 16:04:01 UTC from IEEE Xplore. Restrictions apply.
1041
Figs. 4a, 4b, and 4c show the experimental results by file-
server. The threads in the fileserver workload manage the
files by a fixed pattern: create, write, close, open, read, close,
and delete. Moreover, as Table 1 shows, fileserver has zero
log file. Hence, the write counts of inodes by fileserver are
nearly uniform. Since we alternatively allocate inode slot
from each page for create file requests, NoWL shows almost
evenly distributed write counts (20392 to 25788) on the
pages in Fig. 4a. PCV further balanced the writes of pages,
as shown in Fig. 4b. However, due to the ignorance of
endurance variations of domains, the SD of wear ratios PCV
reaches 5.71. In this case, Contour shows balanced wear
ratio of pages and the SD of wear ratios is only 0.56, which
is 10.2
lower than that of PCV.
On webserver, OLTP, and MySQL, Contour still shows
balanced wear of pages. The SDs of wear ratios by Contour
are 0.95, 0.81, and 0.49 for webserver, OLTP, and MySQL
respectively. On the contrary, many pages in NoWL are
already worn out for their write counts exceed the endur-
ance of the corresponding pages, even more than 100
that
of several pages. The maximum wear ratios in NoWL is
2740.3
higher than these of Contour on average. Moreover,
the SDs of wear ratios in NoWL are 286.1
, 221.4
, and
417.8
higher than that of Contour for webserver, OLTP,
and MySQL respectively. The highly imbalanced wear of
pages by NoWL is due to the lack of wear-leveling mecha-
nism for the heavily-written files.
Figs. 4e, 4h, and 4k show that PCV also evenly distributes
the writes to the pages on the latter three workloads. The
wear ratios of pages, however, are highly imbalanced: The
SD of wear ratios on webserver, OLTP, and MySQL are 3.1,
3.3, and 2.7 respectively. The wear leveling of Contour out-
performs PCV by 2.2
, 3.0
, and 4.5
on webserver, OLTP,
and MySQL respectively. Meanwhile, the maximum wear
ratios in Contour is 2.4
lower than these of PCV on aver-
age. Hence, Contour can achieve better wear leveling of
pages than PCV.
Different from fileserver, Figs. 4f, 4i, and 4l show that
some pages in Contour has relatively low write counts and
wear ratios by webserver, OLTP, and MySQL. For example,
there are 101 pages that has a wear ratio less than 1 percent
by MySQL. In webserver, OLTP, and MySQL, many files
are heavily written, as Figs. 4d, 4g, and 4j illustrated. In this
case, Contour migrates the related inodes considering the
write budget of the pages to balance the wear ratio. Some of
pages are not tolerating much writes yet for the other pages
still have write budgets. Therefore, the wear ratios of pages
in Contour will be more balanced with the increased num-
ber of writes.
4.3 Performance Overhead
To understand the impact of the proposed Contour mecha-
nism on the performance of file systems, we measure the
throughput of write(), one of the most frequently used opera-
tion that updates inode, by FIO and the Ops/s of application-
level workloads by Filebench. “I/O Size” in Fig. 5 means the
data size of each I/O request.
Fig. 5 shows that Contour and PCV has less than 1 percent
performance variation on both write throughputs and appli-
cation-level throughputs, even though Contour achieves bet-
ter wear leveling. For sequential write, PCV and Contour
1042 IEEE TRANSACTIONS ON COMPUTERS, VOL. 70, NO. 7, JULY 2021

Fig. 4. Comparison of page-level write distribution in NoWL, PCV, and Contour at 10 hours.

show 6.8 and 8.1 percent performance degradation on aver-
age compared with NoWL, which can easily cause data
loss for unaware of wear-leveling. For random write, the
throughput of PCV and Contour drop by 8.9 and 8.6 percent
on average respectively compared to NoWL. The write
throughput degradation of Contour decreases with the
increase of I/O size. When I/O size is larger than 32 KB,
the sequential and random write throughputs of Contour are
between 95.6 and 100.0 percent these of NoWL. It is because
the weight of inode management cost in a write operation
decreases with the increase of the size of data transfer.
In the cases of application-level workloads, we measure the
Ops/s of fileserver, webserver, and OLTP on the file systems.
Fig. 5c shows the experimental results. The performance of
Contour achieves 98.2, 99.3, and 99.9 percent that of NoWL
for fileserver, webserver, and OLTP respectively. It shows
that the performance overhead of Contour is negligible for the
applications with complex file access patterns.
4.4 Wear Overhead
PCV and Contour both have wear overheads caused by
writing the migrated inodes to new locations and updating
the offsets in the deflection table. Table 3 shows the evalua-
tion results of wear overheads. “AWR (%)” is calculated by
dividing the additional writes cause by migration to the
total write counts in the experiments. “Domain wear (%)” is
the average wear ratio of the domains after running the
workload for 10h. “AWER (%)” is the wear ratio caused by
additional writes. The AWR of PCV and Contour are both
less than 1 percent on all the benchmarks. The average

Authorized licensed use limited to: R V College of Engineering. Downloaded on December 28,2021 at 16:04:01 UTC from IEEE Xplore. Restrictions apply.
CHEN ET AL.: CONTOUR: A PROCESS VARIATION AWARE WEAR-LEVELING MECHANISM FOR INODES OF PERSISTENT MEMORY FILE...
Fig. 5. Performance comparison of NoWL, PCV, and contour.
AWR of Contour is 0.06 percent higher than that of PCV. It
is because that Contour protects the weak memory domains
by controlling the write budgets. The AWER of PCV and
Contour are 0.041 and 0.034 percent respectively. The
AWER of Contour is lower than that of PCV for Counter
has lower average wear ratio of domains. Hence, the wear
overhead of Contour is negligible to the system.
4.5 Sensitive Analysis of Tuning Write Budget Line
Contour manages the writes of domains using a Write Bud-
get Line (WBL). Here we analysis the sensitivity of Contour
to the WBL. We set the WBL of each round to 0.5, 1, 1.5, and
2 percent. Then, we run the webserver workload for 5 hours
to evaluate the wear ratios of domains, the wear overhead
and the performance overhead.
Fig. 6a shows the wear ratios of memory domains in Con-
tour with different WBL. The variation of wear ratios grows
larger with the increase of WBL. On one hand, the SDs of
wear ratios of the memory domains are 0.0023, 0.0126, 0.0074,
and 0.018 when WBL is 0.5, 1, 1.5, and 2 percent respectively.
On the other hand, the differences between the maximum
and minimum wear ratio grows from 0.67 percent with
“WBL=0.5” to 4.96 percent with “WBL=2”. Therefore, the
smaller the WBL is, the better wear leveling is guaranteed.
Nevertheless, a smaller WBL will bring larger overhead.
As Fig. 6b shows, the additional write ratio AWR of Con-
tour is decreased from 1.69 percent with “WBL=0.5” to 0.37
percent with “WBL=2”. Accordingly, the additional wear
ratio AWER of Contour is also reduced from 0.056 percent
with “WBL=0.5” to 0.01 percent with “WBL=2”. With a
larger WBL, each domain can tolerant more writes and
thereby reduce the number of migrations. The reduction of
migration result in lower wear overhead and overall perfor-
mance overhead. We normalize the Ops/s with different
TABLE 3
The Wear Overhead Caused by Migrations
in PCV [1] and Contour
Authorized licensed use limited to: R V College of Engineering. Downloaded on December 28,2021 at 16:04:01 UTC from IEEE Xplore. Restrictions apply.
1043
WBL configurations to that with WBL=0.5, as shown in
Fig. 6b. When WBL is 1, 1.5, and 2 percent, the performance
shows 1, 1.1, and 1.4 percent improvement over that with
WBL=0.5, respectively.
In summary, when WBL grows larger than 1 percent, the
wear ratios of domains varies drastically while the overhead
reduces slightly. Thus, we set WBL to 1 percent in the
experiments to balance the trade-off between the wear-
leveling effect and the migration overhead.
5 RELATED WORK
5.1 Wear Leveling in Persistent Memory
File Systems
Several existing persistent memory file systems also consider
wear leveling. BPFS [4] mentions that page-level wear leveling
can be achieved by periodically swapping virtual-to-physical
page mappings. However, swapping page mappings for dif-
ferent data in the file system is a complex mechanism that
requires careful analysis and design. PMFS [5] leaves wear
leveling to hardware for they believe that software-based
wear leveling may be overly complicated. In this paper, we
show that software-base wear leveling only has negligible
overhead to the file system. HMVFS [36] simply improves
wear leveling of metadata and file data by write ordering.
UnistorFS [37] achieves wear leveling by interchanging the
pages between PM-based main memory and storage. The
problem is that existing persistent memory file systems still
maintain the inodes on fixed locations. The inode section of
these persistent memory file systems can be easily damaged
by frequently updated inodes.
5.2 Process Variation Aware Wear Leveling
Process variation is not only a problem for PM but also for
flash memory [38]. Most works for flash memory with pro-
Fig. 6. Sensitivity of tuning write budget line in Contour.
1044
cess variation focus on optimizing the lifetime and perfor-
mance in the SSD controller [39], [40], [41], rather than in
the system software. Besides, the granularity difference
between flash memory and PM brings fundamental discrep-
ancy between their wear leveling techniques. There are also
works that consider process variation in the wear leveling
of PM. Dong et al. [15] propose a hardware migration algo-
rithm for wear rate leveling considering the trade-off
between endurance benefits and migration cost. Yun
et al. [42] present a bloom-filter-based algorithm to achieve
highly efficient fine-grained wear leveling of PCM. Zhao
et al. [24], [43] take advantages of cell morphing to improve
the endurance of MLC PCM cells. Han et al. [44] propose an
architecture-level approach to balance the wear rate of PCM
chips. Zhang et al. [45] evenly put writes on weak or strong
domains according to an endurance-dependent probability.
Most of the process variation aware wear leveling works
focus on architecture level, which bring large hardware cost
and show inflexibility on the granularity of migration. In
this paper, the proposed OS-level approach, Contour, is
easy to deploy with negligible hardware and software costs.
6 CONCLUSION
In this paper, we studied the wear leveling problem of ino-
des, the most frequently updated data in a file system, on
persistent memory considering process variation. We
showed the potential wear risk of inodes by a motivational
example obtained on existing persistent memory file sys-
tems. We proposed a process variation aware wear leveling
mechanism, Contour, to balance the wear of memory pages
for inodes. Contour enables dynamic migration of inodes by
a deflection table. In tackle of varied endurance of memory
domains, we designed cross-domain migration algorithm to
match inodes to proper memory domains. For the memory
pages in the same domain show same endurance, we pre-
sented an intra-domain migration algorithm to balance the
writes on the pages in the same memory domain. We imple-
mented a case of Contour in Linux kernel based on a real per-
sistent memory file system. Experimental results show that
Contour achieves significant wear-leveling improvement
over existing solutions with negligible overhead.
ACKNOWLEDGMENTS
The authors would like to thank the anonymous reviewers for
their valuable feedbacks and improvements on this article.
This work was supported in part by the National Natural Sci-
ence Foundation of China under Grant 61802038, in part by
the Chongqing Postdoctoral Special Science Foundation
under Grant XmT2018003, in part by the China Postdoctoral
Science Foundation under Grant 2017M620412, and in part by
the Joint Sino(Chongqing)-Singapore Post-Doctoral Fellow-
ship Program.
REFERENCES
[1] X. Chen, E. H. -M. Sha, Y. Zeng, C. Yang, W. Jiang, and Q. Zhuge,
“Efficient wear leveling for inodes of file systems on persistent
memories,” in Proc. Des. Autom. Test Eur. Conf. Exhib., 2018,
pp. 1524–1527.
[2] P. Zhou, B. Zhao, J. Yang, and Y. Zhang, “A durable and energy
efficient main memory using phase change memory technology,”
in Proc. Int. Symp. Comput. Archit., 2009, pp. 14–23.
Authorized licensed use limited to: R V College of Engineering. Downloaded on December 28,2021 at 16:04:01 UTC from IEEE Xplore. Restrictions apply.
IEEE TRANSACTIONS ON COMPUTERS, VOL. 70, NO. 7, JULY 2021
[3] J. Izraelevitz et al., “Basic performance measurements of the intel
optane DC persistent memory module,” 2019, arXiv: 1903.05714.
[4] J. Condit et al., “Better I/O through byte-addressable, persistent
memory,” in Proc. ACM Symp. Operating Syst. Princ., 2009,
pp. 133–146.
[5] S. R. Dulloor et al., “System software for persistent memory,” in
Proc. Euro. Conf. Comput. Syst., 2014, pp. 15:1–15:15.
[6] K. Zeng, Y. Lu, H. Wan, and J. Shu, “Efficient storage manage-
ment for aged file systems on persistent memory,” in Proc. Des.
Autom. Test Eur. Conf. Exhib., 2017, pp. 1773–1778.
[7] J. Ou, J. Shu, and Y. Lu, “A high performance file system for non-
volatile main memory,” in Proc. Eur. Conf. Comput. Syst., 2016,
pp. 12:1–12:16.
[8] E. Lee, S. H. Yoo, and H. Bahn, “Design and implementation of a
journaling file system for phase-change memory,” IEEE Trans.
Comput., vol. 64, no. 5, pp. 1349–1360, May 2015.
[9] J. Xu and S. Swanson, “NOVA: A log-structured file system for
hybrid volatile/non-volatile main memories,” in Proc. USENIX
Conf. File Storage Technol., 2016, pp. 323–338.
[10] E. H.-M. Sha, X. Chen, Q. Zhuge, L. Shi, and W. Jiang, “A new
design of in-memory file system based on file virtual address
framework,” IEEE Trans. Comput., vol. 65, no. 10, pp. 2959–2972,
Oct. 2016.
[11] M. K. Qureshi, V. Srinivasan, and J. A. Rivers, “Scalable high perfor-
mance main memory system using phase-change memory tech-
nology,” in Proc. Int. Symp. Comput. Archit., 2009, pp. 24–33.
[12] F. Huang, D. Feng, Y. Hua, and W. Zhou, “A wear-leveling-aware
counter mode for data encryption in non-volatile memories,” in
Proc. Des. Autom. Test Eur. Conf. Exhib., 2017, pp. 910–913.
[13] H. S. P. Wong et al., “Phase change memory,” Proc. IEEE, vol. 98,
no. 12, pp. 2201–2227, Dec. 2010.
[14] W. Zhang and T. Li, “Characterizing and mitigating the impact of
process variations on phase change based memory systems,” in
Proc. IEEE/ACM Int. Symp. Microarchit., 2009, pp. 2–13.
[15] J. Dong, L. Zhang, Y. Han, Y. Wang, and X. Li, “Wear rate level-
ing: Lifetime enhancement of pram with endurance variation,” in
Proc. Des. Autom. Conf., 2011, pp. 972–977.
[16] Z. Sun, X. Bi, and H. Li, “Process variation aware data manage-
ment for SRR-RAM cache design,” in Proc. Int. Symp. Low Power
Electron. Des., 2012, pp. 179–184.
[17] W. Zhou, D. Feng, Y. Hua, J. Liu, F. Huang, and P. Zuo,
“Increasing lifetime and security of phase-change memory with
endurance variation,” in Proc. IEEE Int. Conf. Parallel Distrib. Syst.,
2016, pp. 861–868.
[18] J. Xu et al., “An efficient spare-line replacement scheme to enhance
NVM security,” in Proc. Des. Autom. Conf. Conf. Exhib., 2019,
pp. 91:1–6.
[19] X. Wu, S. Qiu, and A. L. Narasimha Reddy,“SCMFS: A file system
for storage class memory and its extensions,” ACM Trans. Storage,
vol. 9, no. 3, pp. 1–11, 2013.
[20] V. Tarasov, E. Zadok, and S. Shepler,“Filebench: A flexible frame-
work for file system benchmarking,” Login, vol. 41, no. 1, pp. 6–12,
2016.
[21] “mysqlslap,” 2019. [Online]. Available: https://dev.mysql.com/
doc/refman/8.0/en/mysqllap.html
[22] “Fio: flexible I/O tester,” 2014. [Online]. Available: http://freecode.
com/projects/fio
[23] A. P. Ferreira, S. Bock, B. Childers, R. Melhem, and D. Moss,
“Impact of process variation on endurance algorithms for wear-
prone memories,” in Proc. Des. Autom. Test Eur. Conf. Exhib., 2011,
pp. 1–6.
[24] M. Zhao, L. Jiang, Y. Zhang, and C. J. Xue, “SLC-enabled wear
leveling for MLC PCM considering process variation,” in Proc.
Des. Autom. Conf., 2014, pp. 36:1–36:6.
[25] S.-H. Chen, Y.-H. Chang, Y.-M. Chang, and W.-K. Shih, “mwJFS:
A multiwrite-mode journaling file system for MLC NVRAM
storages,” IEEE Trans. Very Large Scale Integr. (VLSI) Syst., vol. 27,
no. 9, pp. 2060–2073, Sep. 2019.
[26] M. Dong and H. Chen, “Soft updates made simple and fast on
non-volatile memory,” in Proc. USENIX Annu. Tech. Conf., 2017,
pp. 719–731.
[27] R. Kadekodi, S. K. Lee, S. Kashyap, T. Kim, A. Kolli, and
V. Chidambaram, “SplitFS: Reducing software overhead in file
systems for persistent memory,” in Proc. ACM Symp. Operating
Syst. Princ., 2019, pp. 494–508.
[28] M. Dong, H. Bu, J. Yi, B. Dong, and H. Chen, “Performance and
protection in the ZoFS user-space NVM file system,” in Proc.
ACM Symp. Operating Syst. Princ., 2019, pp. 478–493.
CHEN ET AL.: CONTOUR: A PROCESS VARIATION AWARE WEAR-LEVELING MECHANISM FOR INODES OF PERSISTENT MEMORY FILE...
[29] Z. Ross, “Add support for new persistent memory instructions,”
2014. [Online]. Available: https://lwn.net/Articles/619851/
[30] “Intel architecture instruction set extensions programming refer-
ence,” 2020. [Online]. Available: https://software.intel.com/sites/
default/files/managed/0d/53/319433-022.pdf
[31] C. Kuo, A. Shah, A. Nomura, S. Matsuoka, and F. Wolf, “How file
access patterns influence interference among cluster applications,”
in Proc. IEEE Int. Conf. Cluster Comput., 2014, pp. 185–193.
[32] J. W. Hurrell et al., “The community earth system model: A frame-
work for collaborative research,” Bull. Amer. Meteorological Soc.,
vol. 94, no. 9, pp. 1339–1360, 2013.
[33] J. Borrill, L. Oliker, J. Shalf, and H. Shan, “Investigation of leading
HPC I/O performance using a scientific-application derived bench-
mark,” in Proc. ACM/IEEE Conf. Supercomputing, 2007, Art. no. 10.
[34] V. Gogte et al., “Software wear management for persistent memo-
ries,” in Proc. USENIX Conf. File Storage Technol., 2019, pp. 45–63.
[35] Intel, “Persistent memory emulation platform (PMEP),” 2016.
[Online]. Available: https://pmem.io/2016/02/22/pm-emulation.
html
[36] S. Zheng, L. Huang, H. Liu, L. Wu, and J. Zha, “HMVFS: A hybrid
memory versioning file system,” in Proc. IEEE Symp. Mass Storage
Syst. Technol., 2016, pp. 1–14.
[37] S.-H. Chen, T.-Y. Chen, Y.-H. Chang, H.-W. Wei, and W.-K. Shih,
“Unistorfs: A union storage file system design for resource shar-
ing between memory and storage on persistent ram-based sys-
tems,” ACM Trans. Storage, vol. 14, no. 1, 2018, Art. no. 3.
[38] M.-F. Chang and S.-J. Shen, “A process variation tolerant embed-
ded split-gate flash memory using pre-stable current sensing
scheme,” IEEE J. Solid-State Circuits, vol. 44, no. 3, pp. 987–994,
Mar. 2009.
[39] L. Shi, Y. Di, M. Zhao, C. J. Xue, K. Wu, and E. H. -M. Sha,
“Exploiting process variation for write performance improvement
on NAND flash memory storage systems,” IEEE Trans, VLSI Syst.,
vol. 24, no. 1, pp. 334–337, Jan. 2016.
[40] Y. Di, L. Shi, K. Wu, and C. J. Xue, “Exploiting process variation
for retention induced refresh minimization on flash memory,” in
Proc. Des. Autom. Test Eur. Conf. Exhib., 2016, pp. 391–396.
[41] Y. Luo, S. Ghose, Y. Cai, E. F. Haratsch, and O. Mutlu, “Improving
3D NAND flash memory lifetime by tolerating early retention loss
and process variation,” ACM Meas. Anal. Comput. Syst., vol. 2, no. 3,
2018, Art. no. 37.
[42] J. Yun, S. Lee, and S. Yoo, “Dynamic wear leveling for phase-
change memories with endurance variations,” IEEE Trans. Very
Large Scale Integr. (VLSI) Syst., vol. 23, no. 9, pp. 1604–1615,
Sep. 2015.
[43] M. Zhao, L. Jiang, L. Shi, Y. Zhang, and C. J. Xue, “Wear relief for
high-density phase change memory through cell morphing con-
sidering process variation,” IEEE Trans. CAD, vol. 34, no. 2,
pp. 227–237, Feb. 2015.
[44] Y. Han, J. Dong, K. Weng, Y. Wang, and X. Li, “Enhanced wear-
rate leveling for pram lifetime improvement considering process
variation,” IEEE Trans. Very Large Scale Integr. (VLSI) Syst., vol. 24,
no. 1, pp. 92–102, Jan. 2016.
[45] X. Zhang and G. Sun, “Toss-up wear leveling: Protecting phase-
change memories from inconsistent write patterns,” in Proc. Des.
Autom. Conf., 2017, pp. 1–6.
Xianzhang Chen (Member, IEEE) received the BS
and MS degrees in computer science and engi-
neering from Southeast University, Nanjing, China,
and the PhD degree from the College of Computer
Science, Chongqing University, China in 2017. He
was a research fellow with the National University
of Singapore from 2019 to 2020. He is currently
an associate professor with Chongqing University.
He was a recipient of best paper awards in
NVMSA’2015 and ICCD’17, “the Editor’s Pick of
2016” of IEEE TC, and the Chongqing Best PhD
Dissertation Award in 2018.
Authorized licensed use limited to: R V College of Engineering. Downloaded on December 28,2021 at 16:04:01 UTC from IEEE Xplore. Restrictions apply.
1045
Edwin H.-M. Sha (Senior Member, IEEE) received
the PhD degree from the Department of Computer
Science, Princeton University, Princeton, NJ, in
1992. From 1992 to 2000, he was with the Depart-
ment of Computer Science and Engineering, Uni-
versity of Notre Dame, Dame, IN. Since 2000, he
has been a tenured full professor with the Univer-
sity of Texas at Dallas, Richardson, TX. From 2012
to 2017, he served as the dean of the College of
Computer Science, Chongqing University, China.
He is currently a tenured distinguished professor
with East China Normal University. He was a recipient of the Teaching
Award, the Microsoft Trustworthy Computing Curriculum Award, the NSF
CAREER Award, the NSFC Overseas Distinguished Young Scholar
Award, and the Chang-Jiang Honorary chair professorship.
Xinxin Wang is currently working toward the mas-
ter’s degree majoring in computer science with
Chongqing University, Chongqing, China. His rese-
arch interests include emerging non-volatile mem-
ory technology and in-memory file system.
Chaoshu Yang received the BS degree from the
College of Computer Science, South-Central Uni-
versity for Nationalities, in 2008. He is currently
working toward the PhD degree with the College of
Computer Science, Chongqing University. His cur-
rent research interests include non-volatile memo-
ries, distributed systems, and file systems.
Weiwen Jiang received the PhD degree from the
College of Computer Science, Chongqing Univer-
sity. He is currently a post-doctoral scholar with
the University of Notre Dame. He was a recipient
of best paper awards in ICCD’17, and best paper
nominations in DAC’19, and CODES+ISSS’19.
His current research interests include neural
architecture search, FPGAs, nonvolatile memo-
ries, and HW/SW co-optimization.
Qingfeng Zhuge (Member, IEEE) received the BS
and MS degrees in electronics engineering from
Fudan University, Shanghai, China, and the PhD
degree from the Department of Computer Science,
University of Texas at Dallas, Richardson, TX, in
2003. She is currently a professor with East China
Normal University, China. Her current research
interests include parallel architectures, embedded
systems, real-time systems, optimization algo-
rithms, and scheduling. She was the recipient of
the Best PhD Dissertation Award, in 2003.
" For more information on this or any other computing topic,
please visit our Digital Library at www.computer.org/csdl.