You are on page 1of 4

The 7th International Conference for Internet Technology and Secured Transactions (ICITST-2012


The survey of cache management in the shared

storage environment
Junfeng Xu, 80 Sun National Computer Network Emergency Response Technical Team, Coordination Center of China (CNCERT/CC) Beijing, China Chengxiang Si National Computer Network Emergency Response Technical Team, Coordination Center of China (CNCERT/CC) Beijing, China

Abstract-With the development of cloud computing, it needs high performance, availability and scalability of storage service. In shared storage environment based on the opening storage architecture, due to the inadaptability of the cache management, cache located on access paths provides performance acceleration, but limits availability and scalability of cloud storage system. This paper studies previous cache management technology of cluster or distributed system, and puts emphasis on availability and consistency, which bring down availability and scalability of cloud storage system. Moreover, this paper also analyzes the adaptability of previous cache management technology for shared storage environment.

error about hardware or software causes system crash, cache data will be lost. It may result in data inaccessible and interruption of application service; therefore, the system availability brings down. (2) Effect on system scalability. In the environment of cloud storage, in order to improve handing ability, it usually concurrently accesses the ended storage through several 10 paths. As a result, there are several copies in different paths; however, every copy is managed independently, which will result in cache inconsistency for multiply paths. One modified copy is known by local cache, but it is invisible to other copies in other paths, which results in cache in-consistency. From the above, with the development of cloud computing, it needs high performance, availability and scalability of storage service. In cloud storage environment based on the opening storage architecture, due to the inadaptability of the cache management, cache located on access paths provides performance acceleration, but limits availability and scalability of cloud storage system. This paper studies previous cache management technology of cluster or distributed system, and puts emphasis on availability and consistency, which bring down availability and scalability of cloud storage system. Moreover, this paper also analyzes the adaptability of previous cache management technology for cloud storage environment. II.

Keywords-shared storage; computing;






With rapid growth of the information and data scale, computing ability of IT systems are facing challenges. The emergence of cloud computing [18][19][20][30] may solve the problem. Cloud Computing is a technology that uses the internet and central remote servers to maintain data and applications. It provides user hardware, platform, software and storage services. As is known, data analysis includes data computing and data storage, so storage system is very important. With the explosive growth of data scale, cloud computing system provides high demand for capacity, performance, reliability and scalability of storage system. Cloud storage system [21][23][24] based on the opening storage architecture is widely noted. It usually employs general hardware and open source software to construct storage system. Cache is an effective method to improve 10 performance of storage system. It employs data access locality to improve performance and brings down access latency of applications. In the environment of cloud storage, cache located on access paths not only improves storage performance, but makes an important effect on availability and scalability of cloud storage system. To be specific, (1) Effect on system availability. In order to improve performance, cache data usually is updated to ended storage asynchronously or periodically. Due to asynchronous update between cache and ended storage, if an

In this section, we will study previous cache management technology of cluster or distributed system, and pay attention to availability and consistency. In terms of different usage modes, we will divide different cache management into the following types to illustrate.

Cache management of cluster file system

According to the cooperative relation between cache nodes, we divide into two types-cooperative cache and non­ cooperative cache. (1) Non-cooperative cache NFS [5] is widely used in the industry, and has been widely transplanted into all operating systems. The property information of client files or directories is kept in client cache

97B-1-90B320-0B/7/$2S.00©2012 IEEE


such as web. In the reference. In the aspect of cache consistency. gateway server. Internet is a widely distributed system. The front web nodes manage all memory of cluster nodes by a global hash table. the server will notifY every client to invalidate its cache. transition managers and oriented-object storage devices. such as Sprite and AFS. remote client memory is regarded as one level of the storage architecture. Moreover. Its basic principle as follows: if a query hit in some level cache. cooperative cache managers. It added NVRAM into client and server sides respectively as temporary cache for write data. switches and routers. which is distributed in the entire Internet. The implement of it is usually simple. it adopted the "write­ invalidate" strategy to notifY the according clients to invalidate the cache block. data which hasn't been flushed into cache disks won't assure reliability. The goal of memcached is to improve cache utilization of the entire system without consideration for cache reliability In addition. server memory and server storage construct the entire storage architecture. It employed memory of storage cluster to provide cache service for web application nodes. which prevents concurrent partition writes and limits concurrent read and write for a single partition to solve access conflicts. which is used to bring down the load of the ended database. this way is only appropriate for the distributed file system whose write data buffered in clients. When clients fail. streaming media. there are two strategies ---. it adopted synchronous-update strategy and only kept read data in cache. clients won't access the according objects. For example. There are kinds of cache distributed at access paths from user terminals to network data sources. NVRAM can guarantee performance and reliability of cache effectively. To be noted. every cache object is unique in the system. and the timeout mechanism also doesn't guarantee cache consistency for multiple clients fundamentally. which can avoid duplication of cache objects in the entire cluster and improve cache utilization of the entire storage cluster system. a Figure 1.00©2012 IEEE 140 . and cache consistency is guaranteed by the lease and distributed transaction mechanisms. which assign an upper limit for one lease. Cache management of networking applications This section will introduce cache management strategies of several popular networking applications. the client never keeps file data in its cache. the cooperative cache in zFS can only buffer read data. Reference [3] adopted NVRAM to improve write performance and reliability of the distributed file system. it employed ICP [14] protocol to finish queries for every cache levels. while data staying in cache exceeds a certain time. NFS will refresh the cache data at the following access. and client memory. The cache reliability of NFS is weak. due to the non­ volatile property of disk medium. The entire file system is consist of clients. It employed all memory of nodes as cache of the entire file system to improve performance. which can guarantee persistent data access at the case of shared storage failures. and the central manager is responsible for token allocation and reclamation. and cache technology has been extensively used in network services. it followed cache consistency protocols of its host file system. When multiple clients open a file concurrently. Due to the limit of cache consistency. Similar to reference [6] . To be specific. (2) Cooperative cache Reference [6] indicated that with the development of networks. 1. but it needs special hardware which results in weak generality. file managers. When one lease exceeds the limit. B. it employed the read-write token mechanism. the query request will be forwarded to cache nodes at the upper 97B-1-90B320-0B/7/$2S. According to the location of one cache server. As shown in Fig. and its distribution in access paths is similar to the tree. The cache consistency of the former is relatively strong. For the cooperative cache. Clients of coda file system [4][11] adopt local disks as data cache. In the aspect of cache consistency. However. the proxy server of the ISP service providers. due to unreliability of write operations. so cache consistency is no need to be considered. For the aspect of cache consistency.The 7th International Conference for Internet Technology and Secured Transactions (ICITST-2012) for a certain time. remote client memory. ( I) Cache cluster for the ended database The type is usually deployed at the gateway or inner of local network.the positive and the passive. zFS [12] [13] is developed by the IBM Haifa lab. The latter employs usual lease methods to make cache effective. it will return immediately. there are the following two cases. lease managers. When a cache block is modified. employing cache of other clients is feasible for distributed file system. Otherwise. Memcached made fully use of all memory of storage cluster to construct cache objects which are shared and managed centrally. NVRAM is very expensive and low cost performance for a long time. Reference [10] added data persistency mechanism for memcached to guarantee cache reliability. such as browser. data which has been written into cache disks will be reliable. Sprite adopted the "concurrent write­ sharing" protocol. and one of them modifies the file. Memcached[7] is a universal distributed cache system. proxy cache cluster (2) Proxy cache cluster Web Cache is widely used. The special manager is responsible for the global cache management and consistency. NFS adopts the timeout mechanism to distinguish whether client cache is effective.

NVRAM or other persistent media can effectively protect the written data. it will notifY clients. the system can still ensure data availability and integrity. When other node demands the according data. For the cache and write cache. and analyze the applicability of the previous technology for cloud storage environment. Previous consistency algorithms are divided into two categories: weak consistency mechanism and strong consistency mechanism. From the view of cache reliability. namely the copy is old. There are two ways to maintain cache consistency: write-invalid and write-update. From the view of high availability. in fact. When data of the web server is updated. III. CONC LUSIONS In this section. Om-cache didn't consider cache consistency. Previous studies mainly employ timeout-update or write-invalidate mechanism to maintain different levels of cache coherency. The technology of cache consistency is to guarantee the consistent view for the same data. It employed two-level cache to realize the protocol conversion and effectively alleviated performance gap between the two. the latest data which is inaccessible may lead to the interruption of business (such as critical data inaccessible). When the data is modified on the server. but semantics of the distributed lock is very complex. and the latter made up of two-level cache. there are two ways to guarantee cache reliability. B. Since there are multiple paths to reach the web server. The timeout-update mechanism can guarantee weak consistency. the availability of the entire system brings down. Availability of storage system demands that when parts of the hardware and software failures. we summary the related research. As stated above. It employed the distributed lock to synchronize shared access between clients and SANBoost. Strong consistency mechanism IS relative complicated. cache consistency of STICS is guaranteed through the lock and the write-invalidated mechanism. and achieve the continuity of applications. Perspective of consistency The reason of cache inconsistency is that there are multiple copies for a data block. and deletes the corresponding cache after the session end. Therefore. From the overall perspective. if one copy has been modified. A. Perspective of availability Cache management of network storage system ( I) Clients of network storage system STICS [8] is a cache system for network storage devices. and referred to the upper application to solve the problem. the data will be flushed into the large log disk. if each server has its own independent cache disk. Om-cache [9] developed by IBM. cache inconsistency will appear. SANBoost [1][2] realized a cache system based solid state disk for a virtual server. The distributed lock mechanism usually needs the lock manager which may be distributed or centralized. When the data amount ofNVRAM reaches a certain value. Every session is associated with the deployment and execution of every application. Modified data is written into the small NVRAM. the redundant technology is employed to eliminate single point of failure. When fault happens.00©2012 IEEE 141 . The data is synchronized to the back-end storage to ensure data consistency. Adaptive TTL and user query. Or we regarded the cache copy is gray. Similar to the timeout mechanism of NFS. Its great feature is to customize and manage cache system based on session semantic strategy. The centralized lock manager usually becomes the bottleneck of 97B-1-90B320-0B/7/$2S. which can ensure that the write cache data is not lost in the event of system failure. The cooperative work finished by the tree-level proxy cache servers. If data in cache isn't flushed into the ended storage before cache nodes failure.The 7th International Conference for Internet Technology and Secured Transactions (ICITST-2012) level until the content server. Therefore. the server will record which client has accessed the data. Moreover. the complexity of data server recovery from failures will be increased. it needs to reacquire the latest version from the server. How to keep cache consistency has been widely studied. it only buffer hot data whose access frequency reached a certain value. solid state disk is persistent. other copies will invalidate. Its goal is to solve performance interference between several logic storage volumes and guaranteed quality of service. STICS included two parts--. which have cached the according data. In order to guarantee cache exclusive from the underlying disk array. STICS provide a lock manager for every shared device and maintain a lock status table which record lock status of shared data segment. the cached data is regarded as fresh. C. failure detection and failover mechanisms will enable work to continue. which is designed to finish effective conversion from SCSI protocol to IP protocol. TTL is an expired time assigned for a cache copy. CD synchronized update. For strong consistency mechanism. and will increase access latency. the modification for one copy will notifY others. For the former. a cache node failure can still ensure the availability of caching proxy service. employed local disk as device cache for network storage system. Weak consistency technologies include TTL (Time-To-Live). SANBoost and disk array cache form the two-level cooperative cache. to invalidate the cached data. it provides a shared read lock and an exclusive write lock. @ the media property. which well supported failover. (2) Access path of network storage system Cache located on 10 path has a significant impact on the front applications and the ended storage. Write-invalidate mechanism employs the distributed lock to achieve different levels of cache consistency. If the current time is less than TTL. including a large log disk and a small NVRAM. The former consisted of a large DRAM. but it can't guarantee continuity of cache service. which is suitable for applications with low consistent demand. Therefore. The beginning of the session establishes the corresponding cache. and the according data in cache is still old. can reduce the amount of communication with the external network and meet the users' requirements.

The management complexity of the distributed lock manager is usually high. the system scalability brings down. Grupp.p df [24] Cloud Data Management Interface http//snia. Darrell G. [ 17] . 10691085. No 4.wikipedia. Sun Microsystems. Gardner. Brad Fitzpatrick. USA November 1994. September 2004. Canada. IBM Research Division Austin Research Laboratory. [ 18] A history of cloud computing http://wVvw. San Diego. College Park. 20 12 [29] Doug Beaver. Path: Page Access Tacking to Improve Memory Management Proceedings of International Symposium on Memory Management. USENIX Association. Wang e.l. [ 1] [ 12] Ohad Rodeh.Iournal of Parallel and Distributed Computing.00©2012 IEEE 142 . 20 12) http//www. San Francisco. USA. Livio Soares. New York. No. Distributed caching with memcached. EY Hensbergen and M. A. Remzi Hussein Arpaci Dusseau. Thomas Walsh. 3 1-42. Baker M . A Scalable Cluster-based Web Server with Cooperative Caching Support. Vol. RFC2187. Performance Boosting and Workload Isolation in Storage Area Networks with SANCache. Dick Henze. B.Iehan-francois Paris. No.Wessels. Yang. MD. ... Inc. 2006. Michael Stumm. we will study the efficient cache protection and consistency maintenance techniques for cloud storage environment. We analyzed design features.Iune 1998. NFS Version 3 Protocol Specification. pp.Iournal. Michael D Dahlin. http//memcachedb. United States Patent: US7941694Bl. Randolph Y Wang. In Proceedings of the 23rd IEEE 1l4th NASA Goddard Conference on Mass Storage Systems and Technologies (MSST).Iournal. et al.17 1. [27] Liton Chakraborty and Ajit Singh. April 1990. B. Boston. On the basis of guaranteeing performance. Proceedings of the 1st USENIX conference on Operating Systems Design and Implementation. Steere. NY. Monterey. pp. weak support for system availability. Freeman. Thomas E Anderson. Staubach. Vancouver. [ 14] Duane. 10-22. cache consistency limiting the scalability of the system. A Stack Model based Replacement Policy for a Non-volatile Cache. (Access date 18 Agust 20 12) [20] The NIST Definition of Cloud Computing http//csrc.Ianuary 2002. Andrea Carol Arpaci Dusseau. Nevada. March 2007. David Patterson. May 2004. pp. Seltzer M. Be. RC24123.USA. Aimed at inapplicability of cache management in the environment of shared storage based on the opening storage architecture. Dick Henze. [26] Reza Azimi. USA. Peter J Braam. MA. 64. Long Beach..The 7th International Conference for Internet Technology and Secured Transactions (ICITST-2012) system performance.. pp. Ousterhout J. IV. 28-30. management technology. The Bleak Future of NAND Flash Memory Proc of the 10th USENIX Conference on File and Storage Technologies (FAST ' 12 ) . cache protection and cache consistency of several typical cache (Access date: 10 Agust 20 12) [22] James A. [ 16] Brian C Forney. Memcachedb. SANBoost: automated SAN­ level caching in storage area (Access date: 10 Agust REFERENCES Ismail Ari. we will try to enhance availability and scalability of the entire system. Pawlowski. October 2007. Siegel. August 2004. Reliable File Systems. Avi Tepennan. USA. . 20 10 [30] Gartner . In Proceedings of IEEE International Parallel & Distributed Processing Symposium. Dynamic policy disk caching for storage networking. the scalability is limited with the increase of system scale. Theodore Haining. In proceedings of the 1st USENIX Conference on File and Storage Technologies. M. Kimberly. Callaghan. In Proceedings of IEEE System Mass Storage System. http//en. Ellen H. pp. Coda: A Highly Available File System for a Distributed Workstation Environment.computerweekly. Puneet Kumar. Maria E Okasai.pdf [25] Chuanpeng Li and Kai Shen.145. San Jose. Steven Swanson. ppA47-459. May 20 1 1. Zhao. [28] Laura M. . Uri schonfeld. P. 7-8.A Scalable Distributed File System Using OSD.nistgov/publications/nistpubs/800. In Proceedings of the 4th USENIX Conference on File and Storage . M.. CA. IS. Finding a needle in Haystack: Facebook's photo storage Proc of the 9th USENIX Symposium on Operating Systems Design and Implementation (OSDI ' 10) . USA. CA: USENIX Association..gartneLcomlitipagejsp?id=1942015 [2] [3] [4] [5] [6] [7] [8] [9] [ 10] [ 1 1] Mahadev Satyanarayanan. 2 17-224. John D. e. USA. and summarized the related research from the perspectives of cache availability and consistency respectively.. X He. CA.pdf (Access date: 5 Agust 20 12) [2 1] Cloudstorage. Teperman. In Proceeding of the 18th International Conference on Parallel and Distributed Computing Systems. 110. [ 13] A. [ 15] Chen G. Ismail Ari. STICS SCSI-to-IP cache for storage area networks . In proceeding of the 20th IEEE / 1 1th NASA Goddard Conference on Mass Storage Systems and Technologies.Li. In Proceedings of the 5th International Conference on Architectural Support for Programming Languages and Operating Systems.145/SP800. Harry e. Personal Cloud Services Will Be Integrated in Most Connected Devices http//www. June-July 2003. May 2006. L.Linux . 2005. According to the summary. CA. therefore. Asami S. In future work.Las Vegas. [23] Cloud Storage for Cloud Computing http://ogforglResources/documents/CloudStorageForCloudComputing. IBM Technical Report. IEEE Transactions on Computers.O. Non-volatile Memory for Fast. Quebec. The Coda Distributed File System. Montreal. Vol 39. 97B-1-90B320-0B/7/$2S. April 2003. March 2000. Claffy. Darrell D ELong. Sanjeev Kumar. Managing Prefetch Memory for Data­ Intensive Online Servers. Computation and Currency: Practice and Experience. October 1992. Vol. September 1997. Kistler. Angela Demke Brown. Asymmetric data mirroring. and Q. Melanie Gottwals.Lau F. USA. June 1995. Deprit E. 164 . September 2005. zFS . and David e. Cooperative Caching: Using Remote Client Memory. therefore. In Proceedings of the 1st IEEE International Conference on Autonomic�cloud-computing/ (Access date: 15 Agust 20 12) [ 19] GoogleTrends. Storage-Aware Caching: Revisiting Caching for Heterogeneous Storage Systems. Weit Improving Performance of a Distributed File System Using OSDs and Cooperative Cache. Monterey. Melanie Gottwals. Zhang. Application of Internet Cache protocol(version2). 9. A Utility-based Approach to Cost­ Aware Caching in Heterogeneous Storage Systems. pp. this paper studied previous cache management technology. Davis. we conclude that previous technologies for cache protection have the shortcomings of low performance.CA. James 1.