The 7th International Conference for Internet Technology and Secured Transactions (ICITST-2012


The survey of cache management in the shared

storage environment
Junfeng Xu, 80 Sun National Computer Network Emergency Response Technical Team, Coordination Center of China (CNCERT/CC) Beijing, China Chengxiang Si National Computer Network Emergency Response Technical Team, Coordination Center of China (CNCERT/CC) Beijing, China

Abstract-With the development of cloud computing, it needs high performance, availability and scalability of storage service. In shared storage environment based on the opening storage architecture, due to the inadaptability of the cache management, cache located on access paths provides performance acceleration, but limits availability and scalability of cloud storage system. This paper studies previous cache management technology of cluster or distributed system, and puts emphasis on availability and consistency, which bring down availability and scalability of cloud storage system. Moreover, this paper also analyzes the adaptability of previous cache management technology for shared storage environment.

error about hardware or software causes system crash, cache data will be lost. It may result in data inaccessible and interruption of application service; therefore, the system availability brings down. (2) Effect on system scalability. In the environment of cloud storage, in order to improve handing ability, it usually concurrently accesses the ended storage through several 10 paths. As a result, there are several copies in different paths; however, every copy is managed independently, which will result in cache inconsistency for multiply paths. One modified copy is known by local cache, but it is invisible to other copies in other paths, which results in cache in-consistency. From the above, with the development of cloud computing, it needs high performance, availability and scalability of storage service. In cloud storage environment based on the opening storage architecture, due to the inadaptability of the cache management, cache located on access paths provides performance acceleration, but limits availability and scalability of cloud storage system. This paper studies previous cache management technology of cluster or distributed system, and puts emphasis on availability and consistency, which bring down availability and scalability of cloud storage system. Moreover, this paper also analyzes the adaptability of previous cache management technology for cloud storage environment. II.

Keywords-shared storage; computing;






With rapid growth of the information and data scale, computing ability of IT systems are facing challenges. The emergence of cloud computing [18][19][20][30] may solve the problem. Cloud Computing is a technology that uses the internet and central remote servers to maintain data and applications. It provides user hardware, platform, software and storage services. As is known, data analysis includes data computing and data storage, so storage system is very important. With the explosive growth of data scale, cloud computing system provides high demand for capacity, performance, reliability and scalability of storage system. Cloud storage system [21][23][24] based on the opening storage architecture is widely noted. It usually employs general hardware and open source software to construct storage system. Cache is an effective method to improve 10 performance of storage system. It employs data access locality to improve performance and brings down access latency of applications. In the environment of cloud storage, cache located on access paths not only improves storage performance, but makes an important effect on availability and scalability of cloud storage system. To be specific, (1) Effect on system availability. In order to improve performance, cache data usually is updated to ended storage asynchronously or periodically. Due to asynchronous update between cache and ended storage, if an

In this section, we will study previous cache management technology of cluster or distributed system, and pay attention to availability and consistency. In terms of different usage modes, we will divide different cache management into the following types to illustrate.

Cache management of cluster file system

According to the cooperative relation between cache nodes, we divide into two types-cooperative cache and non­ cooperative cache. (1) Non-cooperative cache NFS [5] is widely used in the industry, and has been widely transplanted into all operating systems. The property information of client files or directories is kept in client cache

97B-1-90B320-0B/7/$2S.00©2012 IEEE


the cooperative cache in zFS can only buffer read data. Clients of coda file system [4][11] adopt local disks as data cache. zFS [12] [13] is developed by the IBM Haifa lab. NFS adopts the timeout mechanism to distinguish whether client cache is effective. it employed ICP [14] protocol to finish queries for every cache levels. which can avoid duplication of cache objects in the entire cluster and improve cache utilization of the entire storage cluster system. it employed the read-write token mechanism. file managers. but it needs special hardware which results in weak generality. lease managers. while data staying in cache exceeds a certain time. and cache consistency is guaranteed by the lease and distributed transaction mechanisms. which assign an upper limit for one lease. Internet is a widely distributed system. employing cache of other clients is feasible for distributed file system. proxy cache cluster (2) Proxy cache cluster Web Cache is widely used. The cache reliability of NFS is weak. It employed all memory of nodes as cache of the entire file system to improve performance.The 7th International Conference for Internet Technology and Secured Transactions (ICITST-2012) for a certain time. such as browser. it followed cache consistency protocols of its host file system. a Figure 1. which can guarantee persistent data access at the case of shared storage failures. it adopted synchronous-update strategy and only kept read data in cache. which is distributed in the entire Internet. server memory and server storage construct the entire storage architecture. As shown in Fig. and cache technology has been extensively used in network services. switches and routers. When a cache block is modified. ( I) Cache cluster for the ended database The type is usually deployed at the gateway or inner of local network. due to unreliability of write operations. The front web nodes manage all memory of cluster nodes by a global hash table. NVRAM is very expensive and low cost performance for a long time. Similar to reference [6] . and client memory. Memcached made fully use of all memory of storage cluster to construct cache objects which are shared and managed centrally. Otherwise. For example. Sprite adopted the "concurrent write­ sharing" protocol. clients won't access the according objects. To be noted. When clients fail. NVRAM can guarantee performance and reliability of cache effectively. According to the location of one cache server. In the reference. remote client memory is regarded as one level of the storage architecture. data which hasn't been flushed into cache disks won't assure reliability. When multiple clients open a file concurrently. the client never keeps file data in its cache. it will return immediately. In the aspect of cache consistency. To be specific. Reference [10] added data persistency mechanism for memcached to guarantee cache reliability. Memcached[7] is a universal distributed cache system. transition managers and oriented-object storage devices. the server will notifY every client to invalidate its cache. The special manager is responsible for the global cache management and consistency. such as web. Moreover. Its basic principle as follows: if a query hit in some level cache. Reference [3] adopted NVRAM to improve write performance and reliability of the distributed file system. and one of them modifies the file. (2) Cooperative cache Reference [6] indicated that with the development of networks. Due to the limit of cache consistency. the proxy server of the ISP service providers. and the central manager is responsible for token allocation and reclamation. streaming media. The latter employs usual lease methods to make cache effective. it adopted the "write­ invalidate" strategy to notifY the according clients to invalidate the cache block. such as Sprite and AFS. the query request will be forwarded to cache nodes at the upper 97B-1-90B320-0B/7/$2S. In the aspect of cache consistency. there are the following two cases. 1. every cache object is unique in the system. The cache consistency of the former is relatively strong. and the timeout mechanism also doesn't guarantee cache consistency for multiple clients fundamentally. The implement of it is usually simple.00©2012 IEEE 140 . so cache consistency is no need to be considered. due to the non­ volatile property of disk medium. gateway server. For the aspect of cache consistency. For the cooperative cache. cooperative cache managers. B. remote client memory. There are kinds of cache distributed at access paths from user terminals to network data sources. NFS will refresh the cache data at the following access. data which has been written into cache disks will be reliable. this way is only appropriate for the distributed file system whose write data buffered in clients. The goal of memcached is to improve cache utilization of the entire system without consideration for cache reliability In addition. which prevents concurrent partition writes and limits concurrent read and write for a single partition to solve access conflicts. It added NVRAM into client and server sides respectively as temporary cache for write data. When one lease exceeds the limit.the positive and the passive. The entire file system is consist of clients. Cache management of networking applications This section will introduce cache management strategies of several popular networking applications. and its distribution in access paths is similar to the tree. there are two strategies ---. It employed memory of storage cluster to provide cache service for web application nodes. However. which is used to bring down the load of the ended database.

and the according data in cache is still old. Availability of storage system demands that when parts of the hardware and software failures. but semantics of the distributed lock is very complex. From the overall perspective. A.00©2012 IEEE 141 . the cached data is regarded as fresh. the complexity of data server recovery from failures will be increased. Its great feature is to customize and manage cache system based on session semantic strategy. it provides a shared read lock and an exclusive write lock. Weak consistency technologies include TTL (Time-To-Live). the data will be flushed into the large log disk. Strong consistency mechanism IS relative complicated. to invalidate the cached data. a cache node failure can still ensure the availability of caching proxy service. It employed two-level cache to realize the protocol conversion and effectively alleviated performance gap between the two. and will increase access latency. Similar to the timeout mechanism of NFS. if one copy has been modified. CD synchronized update. It employed the distributed lock to synchronize shared access between clients and SANBoost. When fault happens. Therefore. III. and the latter made up of two-level cache. The timeout-update mechanism can guarantee weak consistency. NVRAM or other persistent media can effectively protect the written data. employed local disk as device cache for network storage system. Adaptive TTL and user query. and achieve the continuity of applications. When the data is modified on the server. STICS included two parts--. there are two ways to guarantee cache reliability. the latest data which is inaccessible may lead to the interruption of business (such as critical data inaccessible). if each server has its own independent cache disk. the availability of the entire system brings down. cache inconsistency will appear. The distributed lock mechanism usually needs the lock manager which may be distributed or centralized. it will notifY clients. @ the media property. In order to guarantee cache exclusive from the underlying disk array. the server will record which client has accessed the data. Therefore. Previous studies mainly employ timeout-update or write-invalidate mechanism to maintain different levels of cache coherency. From the view of high availability. but it can't guarantee continuity of cache service. As stated above. (2) Access path of network storage system Cache located on 10 path has a significant impact on the front applications and the ended storage. Every session is associated with the deployment and execution of every application. which have cached the according data. Moreover.The 7th International Conference for Internet Technology and Secured Transactions (ICITST-2012) level until the content server. If data in cache isn't flushed into the ended storage before cache nodes failure. other copies will invalidate. From the view of cache reliability. we summary the related research. including a large log disk and a small NVRAM. How to keep cache consistency has been widely studied. Om-cache didn't consider cache consistency. the system can still ensure data availability and integrity. For the latter. it only buffer hot data whose access frequency reached a certain value. solid state disk is persistent. Perspective of consistency The reason of cache inconsistency is that there are multiple copies for a data block. the redundant technology is employed to eliminate single point of failure. For the former. and referred to the upper application to solve the problem. Or we regarded the cache copy is gray. Its goal is to solve performance interference between several logic storage volumes and guaranteed quality of service. and analyze the applicability of the previous technology for cloud storage environment. If the current time is less than TTL. Since there are multiple paths to reach the web server. When the data amount ofNVRAM reaches a certain value. There are two ways to maintain cache consistency: write-invalid and write-update. failure detection and failover mechanisms will enable work to continue. in fact. namely the copy is old. B. Om-cache [9] developed by IBM. which is suitable for applications with low consistent demand. cache consistency of STICS is guaranteed through the lock and the write-invalidated mechanism. The data is synchronized to the back-end storage to ensure data consistency. C. SANBoost [1][2] realized a cache system based solid state disk for a virtual server. which can ensure that the write cache data is not lost in the event of system failure. The centralized lock manager usually becomes the bottleneck of 97B-1-90B320-0B/7/$2S. can reduce the amount of communication with the external network and meet the users' requirements. STICS provide a lock manager for every shared device and maintain a lock status table which record lock status of shared data segment. The cooperative work finished by the tree-level proxy cache servers. which is designed to finish effective conversion from SCSI protocol to IP protocol. Therefore. CONC LUSIONS In this section. Write-invalidate mechanism employs the distributed lock to achieve different levels of cache consistency. For strong consistency mechanism. Previous consistency algorithms are divided into two categories: weak consistency mechanism and strong consistency mechanism. When data of the web server is updated. Perspective of availability Cache management of network storage system ( I) Clients of network storage system STICS [8] is a cache system for network storage devices. which well supported failover. When other node demands the according data. TTL is an expired time assigned for a cache copy. the modification for one copy will notifY others. and deletes the corresponding cache after the session end. The technology of cache consistency is to guarantee the consistent view for the same data. it needs to reacquire the latest version from the cache and write cache. The beginning of the session establishes the corresponding cache. SANBoost and disk array cache form the two-level cooperative cache. Modified data is written into the small NVRAM. The former consisted of a large DRAM.

the scalability is limited with the increase of system scale.00©2012 IEEE 142 . 10-22. IBM Research Division Austin Research Laboratory. USA. Sun Microsystems. Deprit E. Darrell D ELong. NFS Version 3 Protocol Specification. . Brad Fitzpatrick. Proceedings of the 1st USENIX conference on Operating Systems Design and Implementation. USA. Teperman. September 1997. IEEE Transactions on Computers. Melanie Gottwals. 20 12 [29] Doug Beaver. M. Coda: A Highly Available File System for a Distributed Workstation Environment. CA. 97B-1-90B320-0B/7/$2S. Maria E Okasai. 28-30. Thomas E Anderson. John D. 9. Dick Henze. Vancouver. USA. June-July 2003. (Access date: 10 Agust 20 12) [22] James A. Vol. Dick Henze. [ 15] Chen G. . Cooperative Caching: Using Remote Client Memory. 2 17-224. Siegel. Puneet Kumar. Steere. Thomas Walsh. Weit Improving Performance of a Distributed File System Using OSDs and Cooperative Cache. [28] Laura M. Seltzer M. Dynamic policy disk caching for storage networking. 110.Ianuary 2002. pp. therefore. pp. USENIX Association. Staubach. Storage-Aware Caching: Revisiting Caching for Heterogeneous Storage Systems.l. In Proceedings of IEEE International Parallel & Distributed Processing Symposium.The 7th International Conference for Internet Technology and Secured Transactions (ICITST-2012) system performance. Asami S. Quebec. [ 16] Brian C Forney.Lau F..CA. June 1995. SANBoost: automated SAN­ level caching in storage area network.Linux . et al. Gardner. Michael D Dahlin. Monterey. United States Patent: US7941694Bl.pdf [25] Chuanpeng Li and Kai Shen. New York.O.Iehan-francois Paris. October 2007. CA. San Diego. CA: USENIX Association. IS. management technology. X He. Distributed caching with memcached. Application of Internet Cache protocol(version2). Livio Soares. In Proceedings of the 4th USENIX Conference on File and Storage Technologies. Steven Swanson.p df [24] Cloud Data Management Interface http//snia.Wessels. STICS SCSI-to-IP cache for storage area networks . Ismail Ari.145/SP800. Darrell G. David Patterson. 164 . According to the summary. Peter J Braam. The management complexity of the distributed lock manager is usually high. Vol. Monterey. [27] Liton Chakraborty and Ajit Singh. the system scalability brings down.. B. EY Hensbergen and M. April 2003. Randolph Y Wang. Wang e. September (Access date 18 Agust 20 12) [20] The NIST Definition of Cloud Computing http//csrc. USA November 1994. therefore.. 20 10 [30] Gartner .. USA. On the basis of guaranteeing performance. e.Las Vegas. and summarized the related research from the perspectives of cache availability and consistency respectively. Canada. A Scalable Cluster-based Web Server with Cooperative Caching Support. [ 13] A. Ellen H.computerweekly. 64. Inc.wikipedia. we will study the efficient cache protection and consistency maintenance techniques for cloud storage environment. Michael Stumm. No. Theodore Haining. In Proceedings of the 23rd IEEE 1l4th NASA Goddard Conference on Mass Storage Systems and Technologies (MSST).com/trends/ (Access date: 10 Agust REFERENCES Ismail Ari. Yang. A Utility-based Approach to Cost­ Aware Caching in Heterogeneous Storage Systems. . March 2000. [ 17] .nistgov/publications/nistpubs/800. RFC2187. 2006. James 1. Kimberly. In future work. IBM Technical Report. Managing Prefetch Memory for Data­ Intensive Online Servers. cache consistency limiting the scalability of the system. No 4. March 2007. pp. 2005.. Nevada. Boston. pp. and Q. College pp. [ 14] Duane. zFS . We analyzed design features. In Proceedings of IEEE System Mass Storage System.17 1. In Proceedings of the 1st IEEE International Conference on Autonomic Computing. October [2] [3] [4] [5] [6] [7] [8] [9] [ 10] [ 1 1] Mahadev Satyanarayanan.. http//memcachedb. CA. May 20 1 1. RC24123. 3 1-42. Zhao. Vol 39. Uri schonfeld. M. Finding a needle in Haystack: Facebook's photo storage Proc of the 9th USENIX Symposium on Operating Systems Design and Implementation (OSDI ' 10) . Claffy. May 2006. Reliable File Systems. Asymmetric data mirroring. MD. CA. Montreal. Freeman. [23] Cloud Storage for Cloud Computing http://ogforglResources/documents/CloudStorageForCloudComputing. September 2005. Aimed at inapplicability of cache management in the environment of shared storage based on the opening storage architecture. August 2004.A Scalable Distributed File System Using OSD. this paper studied previous cache management technology. Performance Boosting and Workload Isolation in Storage Area Networks with SANCache. Path: Page Access Tacking to Improve Memory Management Proceedings of International Symposium on Memory Management. [ 18] A history of cloud computing http://wVvw. A Stack Model based Replacement Policy for a Non-volatile Cache. USA. [ 1] [ 12] Ohad Rodeh. No. The Bleak Future of NAND Flash Memory Proc of the 10th USENIX Conference on File and Storage Technologies (FAST ' 12 ) . B. San Jose. [26] Reza Azimi. http//en.pdf (Access date: 5 Agust 20 12) [2 1] Cloudstorage. Andrea Carol Arpaci Dusseau. Sanjeev Kumar.. weak support for system availability. 10691085. April 1990. Davis. pp. Ousterhout J. Zhang. L.Linux . Callaghan. Avi Tepennan. The Coda Distributed File System.Iournal of Parallel and Distributed Computing.USA. Non-volatile Memory for Fast. pp. Memcachedb. MA. we will try to enhance availability and scalability of the entire system. we conclude that previous technologies for cache protection have the shortcomings of low�cloud-computing/ (Access date: 15 Agust 20 12) [ 19] GoogleTrends. Long Beach. May 2004. NY. Be. In proceedings of the 1st USENIX Conference on File and Storage Technologies.Iournal. Melanie Gottwals.Iune 1998. cache protection and cache consistency of several typical cache systems. Computation and Currency: Practice and Experience.145. San Francisco.Li. A. IV. Pawlowski. USA. Personal Cloud Services Will Be Integrated in Most Connected Devices http//www. Remzi Hussein Arpaci Dusseau. and David e. ppA47-459. Angela Demke Brown. Baker M . Grupp. Harry e. P. Kistler. In Proceeding of the 18th International Conference on Parallel and Distributed Computing Systems.Iournal. 20 12) http//www. In proceeding of the 20th IEEE / 1 1th NASA Goddard Conference on Mass Storage Systems and Technologies. In Proceedings of the 5th International Conference on Architectural Support for Programming Languages and Operating Systems.

Sign up to vote on this title
UsefulNot useful