2018 10th International Conference on Intelligent Human-Machine Systems and Cybernetics
understanding cache policy to evaluate write performance with hierarchical
hardware
Lin Qian, Zhu Mei, Hengmao Pang, Jun Yu, Guangxin Zhu, Haiyang Chen, Min Bu
State Grid Electric Power Research Institute
(SGEPRI)
Nanjing, China
qianlin@sgepri.sgcc.com.cn, meizhu2016@aliyun.com, panghengmao@sgepri.sgcc.com.cn,yujun@sgepri.sgcc.com.cn,
zhuguangxin@sgepri.sgcc.com.cn, chenhaiyang@sgepri.sgcc.com.cn, bumin@sgepri.sgcc.com.cn
Abstract—First, it briefly introduces the status of write-
optimized distributed file system, focuses on the issues related
to write optimization, and then introduces write-optimized full-
stack cache acceleration and related technology of storage
tiering, followed by full-stack cache acceleration in SSD Cache.
Accelerated technology was introduced in detail, and the
performance advantages were further demonstrated through
comparative testing, from the above data, we can clearly see
that using the Flash cache, the flash cache of the four nodes is
64% lower than the SATA hard disk processing time. Finally,
the challenges that may be faced in the future research work
were prospected.
Keywords-write feature;cache promoting;storage tiering;flash
cache; memory acceleration
I. INTRODUCTION
In recent years, with the continuous development of
emerging technologies such as big data and cloud computing,
the data in the world has increased dramatically, posing
severe challenges to the storage capacity and performance.
Distributed file system has become a research hotspot as the
underlying mass data storage system supporting the big data
era [1].
Data reading and writing are two basic operations
provided by a distributed file system. The performance of
read operations of a distributed file system[2] is extremely
high due to the read performance of physical media such as
memory and SSD, and read operations do not need to
consider data security. Research heat has not received much
attention compared to write optimization.
In the process of writing data, the distributed file system
needs to solve the contradiction between the consistency of
write data brought by the distributed architecture [3] and the
overall system performance, and has become a key concern
in the field of distributed storage in recent years. FAST, the
most important academic conference in the field of storage,
reported on the topic of writing optimized file systems in the
form of topics in the year 15 and 16. Among them, the write
optimization of memory acceleration [4] has a supportive
role in significantly improving the system performance, and
the premise of how to ensure data storage security Next, the
use of write buffers to speed up the concurrent suspension of
file system problems caused by frequent IO is a hot topic.
978-1-5386-5836-9/18/$31.00 ©2018 IEEE 308
DOI 10.1109/IHMSC.2018.10176
This article focuses on the research work of writing
optimized distributed file system. Specifically speaking, the
memory is used as the physical disk acceleration media to
build a multi-level cache, so that the hot and cold data are
adaptively provided in the storage system. Section 1 of this
paper briefly introduces the current status of write-optimized
distributed file systems and focuses on the issues related to
write optimization. Section 2 describes related techniques for
full-stack cache acceleration and storage tiering; Section 3
describes full-stack cache acceleration. The SSD Cache
acceleration technology in the focus of the introduction;
Section 4 through the Flash cache and HDD, SATA
comparison test to prove the performance advantages of
cache acceleration, the final summary of the full text, and the
future of attention to the research direction of a preliminary
discussion.
II. FLASH ACCELERATION RELATED TECHNOLOGIES
A. Full stack cache acceleration
With the rapid development of hardware technology,
attention is paid to the local rules of the program. Memory,
SSD, NVM, and 3D-Xpoint are used as the first storage
medium, and the storage location of data is dynamically
adjusted to balance the conflict between data capacity and
system performance. . At the same time, we noticed that the
performance bottleneck of the flash memory storage medium
is the write operation, not only because the write operation is
more time-consuming than the read operation, but also
because the read operation is also affected by the lock
mechanism when the write operation is not completed.
In this research direction, as early as the second session
of the FAST International Conference in 2003, N. Megiddo
[5] and others proposed an adaptive cache algorithm to
accelerate the storage IO performance. In the following years,
FAST, MSST and other important international storage The
optimization of storage IO performance by the cache at the
meeting has become a hot topic and will continue until 2016
with new applications such as mobile.
S. Huang [6] and others proposed using flash media and
adaptive algorithms to optimize the underlying storage
performance. R. Koller et al. [7] provided high-performance
storage through the client's flash acceleration technology to
enhance the coverage of the client to the bottom layer. The
full-stack IO optimization solution for storage, C. Li, P.
Shilane [8], etc. achieves an organic balance of performance
and capacity by optimizing the capacity of the SSD
acceleration process. Dulcardo Arteaga [9] and others solved
two problems in the virtual environment. On the one hand,
Flashcache acceleration was provided on demand according
to the dynamic load of the VM. On the other hand, the
dynamic cache migration technology was used to break the
performance degradation caused by insufficient cache
capacity.
B. Storage tiering
Under normal circumstances, the application type and
data type are different, and the underlying storage IO
processing features also show different ways. In fact, the
storage tiering technology is used to dynamically adapt to
what type of data should be stored on which kind of
performance-adapted storage media. Because the capacity of
high-speed storage media such as SSD is limited, when the
data volume exceeds the storage capacity threshold, it needs
to be performed. The storage of hot and cold data is replaced,
and this process is called storage dynamic data layering.
From the figure below, it can be seen that the virtual
volume is stored at the operating system level, in which the
data tiering control algorithm for the volume is divided into
DRAM cache, SSD cache, SSD non-volatile storage layer,
HDD volume storage layer, control strategy with IO quantity,
Read-write ratios, IO positions, IO parameters, and adaptive
changes such as high-speed and slow-media ratios for each
subsystem. The emergence of technologies such as 3D-
xpoint and NVM[10] over the past few years has enabled
storage tiering technology to form a multi-tier, multi-tiered,
interactive storage trend, and dynamically integrate resource
information in real time, under the premise of ensuring data
security and consistency, and in physical storage. Maximize
performance with non-volatile high-speed storage.
Figure 1. Storage layered framework
The existing data tiered storage strategy [11] is mainly
divided into three types, which are divided into write-back,
read-only, and write-through:
ЬWrite-back mode: For read data, all cached to high-
speed device SSD (if the SSD space is not enough, according
to the replacement strategy, the data in the cache is
synchronized to the ordinary hard disk; for the write
operation, it will also be cached to the SSD; However, when
dirty data (inconsistent data in the HDD) in the SSD reaches
a certain threshold, dirty data is synchronized to the SSD.
309
Ь Read-only mode: Only read operations are cached,
and the cache process is the same as write-through mode;
write operations are not cached. This mode is very suitable
for reading more than write less scenes;
ЬWrite-through mode: For read data, all data is cached
on the high-speed device SSD device; for write operations, it
is also cached in the high-speed device SSD and also written
to the HDD. This mode is more suitable for reading more
and writing less, and the data written is the hot data scene
recently accessed. This mode is because writing is also
written to the HDD at the same time, so the performance of
writing is not optimal, but the data is the safest.
Data writeback refers to flushing dirty data from an SSD
to an HDD hard disk. This is only available in write-back
mode. General data write-back will have some system
parameters, such as the threshold (that is, dirty data reaches a
certain rate will start to perform data write-back); aging time
(data over a certain period of time without operation, you can
perform data write-back); data back The rate of writing
(because it is slower to write the SSD winning data into the
HDD, and the rate limit of writing back needs to be
considered). The timing and rate of writing back data will
greatly affect the performance of writing. If the threshold is
set very high, the speed of data writing will be much lower
than the speed of data writing to the SSD when the threshold
is reached. This will cause the SSD to have insufficient space
for newly written data, which will cause writes to be directly
written to the HDD ordinary hard disk, which will not
achieve write acceleration.
In the write-back mode, there are four data replacement
strategies [12]. In the actual use process, the user can select
an appropriate replacement strategy according to the specific
situation, and make the maximum possible degree to keep
the hotspot data in the SSD to the maximum extent. SSD's
performance advantages are maximized.
FIFO (First In First Out): First-in-first-out strategy,
which is the simplest replacement strategy. The essence of
this strategy is to always select the block replacement that
has spent the longest time in the SSD, that is, the block that
newly enters the SSD, and replace it first. The reason is that
the block that was first written to the SSD is the most likely
to be read/written no longer. This strategy is only ideal when
accessing the address space in linear order, otherwise it is
inefficient;
LRU (Least Recently Used): The longest unused strategy
recently. The essence of this strategy is that, when it is
necessary to replace the block in the SSD, the block that has
not been used for the longest period of time is selected to be
replaced. The LRU strategy is related to the last time each
block is used. Due to the complexity of this strategy, there is
an LRU-approximate strategy NUR (Not Recently Used) that
has not been used recently;
LFU (Least Frequently Used): Minimal use of
replacement strategy. The essence of this strategy is that,
when it is necessary to replace the block in the SSD, the
block replacement that uses the least time is selected. The
LFU strategy is related to the number of times each block is
used;
 RAND: random replacement strategy. The essence of this
strategy is that, when it is necessary to replace the block in
the SSD, the serial number of the SSD to be replaced with
the hardware and software random number is replaced.
III. SSD CACHE ACCELERATION TECHNOLOGY
Device mapper is a mapping framework mechanism from
logical device to physical device provided in the Linux 2.6
kernel. Under this mechanism, users can easily formulate
management strategies for implementing storage resources
according to their own needs.
Device mapper is registered as a block device driver in
the kernel. It contains three important object concepts:
mapped device, mapping table, and target device:
Ь Mapped device is a logical abstraction that can be
understood as a logical device provided by the kernel.
Ь The mapping table describes the mapping from the
Mapped device to the Target device.
Ь Target device represents the physical space segment
mapped by a mapped device. For a logical device
represented by a mapped device, it is a physical device to
which the logical device is mapped.
The three objects in the Device mapper and the target
driver plug-in together form an iterable device tree. As
shown below:
Figure 2. Device Mapper kernel object mapping hierarchy
As can be seen from the above figure, the top-level root
node in the tree structure is the mapped device that is
ultimately provided as a logical device, and the leaf node is
the underlying physical device represented by the target
device. The smallest device tree consists of a single mapped
device and target device. Each target device is exclusively
owned by the mapped device and can only be used by a
mapped device. A mapped device can be mapped to one or
more target devices, and a mapped device can be used as the
target device on its upper mapped device. This layer can
theoretically be infinitely iterated under the device mapper
architecture.
310
The essential function of Device mapper is to forward IO
requests from the mapped device of the logical device to the
corresponding target device according to the mapping
relationship and the IO processing rule described by the
target driver.
DM-Cache utilizes the Device Mapper principle. The
HDD common disks and SSD disks are regarded as Target
Devices in the DM, aggregated into one virtual device
(Cache device), and the HDD ordinary disks are mapped
with the SSD disk blocks. The Dm-Cache can be mapped to
the HDD ordinary disks according to the mapping. The
operations are converted to operations on the SSD device,
thereby realizing the function of storing the hot spot data in
the SSD. The mapping of the HDD common disk to the SSD
device block may be a simple linear mapping or a Hash
mapping.
IV. PERFORMANCE TESTING
A. Comparison test with HDD hard disk
Here we chose the Flash cache as the Cache software for
testing. Compared with the performance of Flash cache and
HDD hard disk, the test produced 4 concurrent requests to
ngnix to forward pressure to 6 clients to mix and
concurrently read and write data, and record the average
feedback time and processing per transaction per press. The
total number of transactions and each node and client nmon
data, using the Filebench tool for results analysis.
Flash cache, 5 nodes, 400 concurrent, 50% read, 50%
write, 8k files:
Figure 3. Flash cache Test
HDD hard disk, 5 nodes, 400 concurrent, 50% read, 50%
write, 8k file:
Figure 4. HDD Test
 From the above data, we can clearly see that with the analyzes that the future write-optimized distributed file
Flash cache, the read/write performance has been greatly system still has room for optimization in terms of network
improved, which is about 6 times higher than that of the latency, local file system load awareness, and further
ordinary HDD hard disk. optimization of the metadata structure.

TABLE I. RAN- READ AND WRITE TEST RESULTS OF FLASH CACHE ACKNOWLEDGMENT
AND HDD
This work was financially supported by the State Grid
Disk Type 5 Node Notes Corporation of Science and Technology Project(WBS
HDD hard 372ops/s 5 Nodes, 400 number:521104170019).
disk Concurrency, 50% REFERENCES
Read, 50% Write,
[1] Zhou Jiang,Wang Weiping,Meng Dan,Ma Can,Gu Xiaoyan,Jiang Jie,
8k File “Key Technology in Distributed File System Towards Big Data
Flash cache 61ops/s 5 Nodes, 400 Analysis,” Journal of Computer Research & Development, Feb. 2014 ,
Concurrency, 50% 51 (2) :pp.382-394.
Read, 50% Write, [2] Zhu, Yifeng. Improved read performance in a cost-effective, fault-
tolerant parallel virtual file system (CEFT-PVFS). In Proceedings 3rd
8k File IEEE/ACM International Symposium on Cluster Computing and the
Grid, pp 730-735, 2003.
[3] Chen, Jie; Tan, Zhihu; Wu, Fei; Xie, Changsheng. SJournal. A new
B. Comparison test with SATA hard disk design of journaling for file systems to provide crash consistency. In
Compared with the performance of Flash cache and Proceedings - 9th IEEE International Conference on Networking,
SATA hard disk, the scenario description: Architecture, and Storage, pp 53-62, 2014.
Ь 800 concurrent random reads and writes [4] Chen, Kerhong; Bunt, Richard B.; Eager, Derek L. Write caching in
distributed file systems. In Proceedings International Conference on
Ь File size: 128k (60%) 1M (40%) Distributed Computing Systems, pp 457-466, 1995.
Ь Number of initialization files per client: 5000 Folders: [5] N. Megiddo and D. S. Modha. ARC: A self-tuning,low overhead
50 replacement cache. In Proceedings ofthe 2nd USENIX Conference on
Ь Read & Write Strategy: Read 60% Write 40% File and Storage Technologies (FAST 03), pages 115–130,
Berkeley,CA, USA, 2003. USENIX Association.
Ь Network Type: 10Gb [6] S. Huang, Q. Wei, J. Chen, C. Chen, and D. Feng.Improving flash-
Ь Number of nodes: 4,5 based disk cache with lazy adaptive replacement. In Proceedings of
the 29th IEEE Symposium on Mass Storage Systems and
TABLE II. RAN- READ AND WRITE TEST RESULTS OF FLASH CACHE Technologies(MSST 13), pages 1–10. IEEE, 2013.
AND SATA [7] R. Koller, L. Marmol,R. Ranganswami, S. Sundararaman,N.
Talagala, and M. Zhao. Write policies for host-side flash caches. In
Disk Type 4 Node 5 Node Notes Proceedings of the 11th USENIX conference on File and Storage
Sata 7200 1T 635ms 432ms Sata line is Technologies (FAST 13), 2013.
connected to raid [8] C. Li, P. Shilane, F. Douglis, H. Shim, S. Smaldone,and G. Wallace.
Nitro: A capacity-optimized SSD cache for primary storage. In
card, raid card has Proceedings of the 2014 USENIX Annual Technical Conference
512MB memory, (ATC14), pp 501–512. USENIX Association, 2014.
faster than bare disk [9] Dulcardo Arteaga,Jorge Cabrera,Jing Xu,Swaminathan
PCI-e as 227ms 171ms Pci-e cache cache Sundararaman, and Ming Zhao. CloudCache: On-demand Flash
Cache Management for Cloud Computing. In Proceedings of the 11th
Flash Cache for sata, size 100GB USENIX conference on File and Storage Technologies (FAST 14),
From the above data, we can clearly see that using the pages 352–369. USENIX Association, 2016.
Flash cache, the flash cache of the four nodes is 64% lower [10] Islam, Nusrat Sharmin, Wasi-Ur-Rahman, Md., Lu, Xiaoyi, Panda,
than the SATA hard disk processing time. Dhabaleswar K. High performance design for HDFS with byte-
addressability of NVM and RDMA. In Proceedings of the
V. CONCLUSIONS International Conference on Supercomputing, pp1-14,2016.
[11] J Sim. Dynamically configuring regions of a main memory in a write-
For the rapid popularization of Internet applications, back mode or a write-through mode. Advanced Micro Devices Inc
designing and developing a write-optimized distributed file California, 2014.
system that can handle a large number of small files has [12] Tanwir,G Hendrantoro,A Affandi. Early result from adaptive
become a hot topic in the field of storage research. This combination of LRU, LFU and FIFO to improve cache server
article focuses on the key technology of flash acceleration performance in telecommunication network. International Seminar on
write optimization to improve database application Intelligent Technology & Its Applications,2015,pp 429-432.
performance, and demonstrates the performance advantages
of this technology through comparative testing. The author

311