A Reconfigurable Cache Consistency Mechanism for Distributed Memory Pool

Duan Zhuohui; Liu Haikun; Zhao Jinwei; Liu Yihang; Liao Xiaofei; Jin Hai

doi:10.7544/issn1000-1239.202330448

Journal of Computer Research and Development > 2023 > 60(9): 1960-1972. > DOI: 10.7544/issn1000-1239.202330448

Duan Zhuohui, Liu Haikun, Zhao Jinwei, Liu Yihang, Liao Xiaofei, Jin Hai. A Reconfigurable Cache Consistency Mechanism for Distributed Memory Pool[J]. Journal of Computer Research and Development, 2023, 60(9): 1960-1972. DOI: 10.7544/issn1000-1239.202330448

Citation:

PDF (2009 KB)

A Reconfigurable Cache Consistency Mechanism for Distributed Memory Pool

National Engineering Research Center for Big Data Technology and System (Huazhong University of Science and Technology), Wuhan 430074
Services Computing Technology and System Lab (Huazhong University of Science and Technology), Ministry of Education, Wuhan 430074
Cluster and Grid Computing Lab (Huazhong University of Science and Technology), Wuhan 430074
School of Computer Science and Technology, Huazhong University of Science and Technology, Wuhan 430074

Funds: This work was supported by the National Key Research and Development Program of China（2022YFB4500303）and the National Natural Science Foundation of China (62072198, 61825202, 61929103).

More Information

Author Bio:
Duan Zhuohui: born in 1994. PhD, post-doc researcher. His main research interests include non-volatile memory systems, storage systems and techniques, and distributed systems

Liu Haikun: born in 1981. PhD, professor, PhD supervisor. Senior member of CCF. His main research interests include in-memory computing, cloud computing, and distributed systems

Zhao Jinwei: born in 1996. Master. His main research interests include remote direct memory access and distributed systems

Liu Yihang: born in 1994. Master candidate. His main research interests include non-volatile storage and distributed systems

Liao Xiaofei: born in 1978. PhD, professor, PhD supervisor. Senior member of CCF. His main research interests include computer architecture, system software, and big data processing

Jin Hai: born in 1966. PhD, professor, PhD supervisor. Fellow of CCF, fellow of IEEE. His main research interests include computer architecture, parallel and distributed computing, big data processing, data storage, and system security
Received Date: June 04, 2023
Revised Date: July 18, 2023
Available Online: July 23, 2023

Graphical Abstract

Abstract

Abstract

In distributed memory systems, caching is an effective way for reducing the latency of remote memory accesses. However, a single cache consistency mechanism often fails to efficiently adapt to the access behaviors of various workloads. We propose a hybrid and reconfigurable cache consistency mechanism for distributed heterogeneous memory pool systems, which has advantages of both directory-based and broadcast-based cache consistency mechanisms. We use the four-quadrant matrix analysis method to analyze the access pattern of each object, and then adopt the most efficient cache consistency mechanism. Moreover, the adopted cache consistency mechanism can be dynamically shifted to another mechanism based on the change of memory access pattern. Experimental results show that the reconfigurable hybrid cache consistency mechanism can improve the read and write performance of distributed heterogeneous memory pool systems by 32.31% and 31.20% on average, respectively, compared with a single cache consistency mechanism. Moreover, the hybrid cache consistency mechanism shows good scalability when the number of clients increases.

FullText(HTML)

References (33)

References

[1]	Bailleu M, Giantsidi D, Gavrielatos V, et al. Avocado: A secure in-memory distributed storage system[C]//Proc of the 2021 USENIX Annual Technical Conf (ATC 2021). Berkeley, CA: USENIX Association, 2021: 65−79
[2]	肖仁智,冯丹,胡燏翀,等. 面向非易失内存的数据一致性研究综述[J]. 计算机研究与发展,2020,57(1):85−101 Xiao Renzhi, Feng Dan, Hu Yuchong, et al. A survey of data consistency research for non-volatile memory[J]. Journal of Computer Research and Development, 2020, 57(1): 85−101 (in Chinese)
[3]	舒继武,陆游游,张佳程,等. 基于非易失性存储器的存储系统技术研究进展[J]. 科技导报,2016,34(14):86−94 Shu Jiwu, Lu Youyou, Zhang Jiacheng, et al. Research progress on non-volatile memory based storage system[J]. Science & Technology Review, 2016, 34(14): 86−94 (in Chinese)
[4]	Gu Juncheng, Lee Y, Zhang Yiwen, et al. Efficient memory disaggregation with infiniswap[C]//Proc of the 14th USENIX Symp on Networked Systems Design and Implementation (NSDI 2017). Berkeley, CA: USENIX Association, 2017: 649−667
[5]	Zheng Shengan, Hoseinzadeh M, Swanson S. Ziggurat: A tiered file system for non-volatile main memories and disks[C] //Proc of the 17th USENIX Conf on File and Storage Technologies (FAST 2019). Berkeley, CA: USENIX Association, 2019: 207−219
[6]	Duan Zhuohui, Liu Haikun, Lu Haodi, et al. Gengar: An RDMA-based distributed hybrid memory pool[C] //Proc of the 2021 IEEE 41st Int Conf on Distributed Computing Systems (ICDCS 2021). Piscataway, NJ: IEEE, 2021: 92−103
[7]	Liao Jianwei, Trahay F, Cai Zhigang, et al. Fine granularity and adaptive cache update mechanism for client caching[J]. IEEE Systems Journal, 2019, 13(2): 1587−1598 doi: 10.1109/JSYST.2018.2866905
[8]	Cai Qingchao, Guo Wentian, Zhang Hao, et al. Efficient distributed memory management with RDMA and caching[J]. Proceedings of the VLDB Endowment, 2018, 11(11): 1604−1617 doi: 10.14778/3236187.3236209
[9]	Fawaz K, Artail H. Dcim: Distributed cache invalidation method for maintaining cache consistency in wireless mobile networks[J]. IEEE Transactions on Mobile Computing, 2013, 12(4): 680−693 doi: 10.1109/TMC.2012.37
[10]	Gao Yixiao, Li Qiang, Tang Lingbo, et al. When cloud storage meets RDMA[C] //Proc of the 18th USENIX Symp on Networked Systems Design and Implementation (NSDI 2021). Berkeley, CA: USENIX Association, 2021: 519−533
[11]	Gray C G, Cheriton D R. Leases: An efficient fault-tolerant mechanism for distributed file cache consistency[C]//Proc of the ACM Symp on Operating System Principles (SOSP 1989). New York: ACM, 1989: 202−210
[12]	Li Xiaoqian, Zhou Huachun, Qiu Feng, et al. A lease algorithm to maintain strong mapping cache consistency[J]. Wireless Personal Communications, 2014, 76(4): 713−730 doi: 10.1007/s11277-013-1530-0
[13]	Yu Xiangyao, Xia Yu, Pavlo A, et al. Sundial: Harmonizing concurrency control and caching in a distributed OLTP database management system[C] // Proc of the 44th Int Conf on Very Large Data Bases (VLDB 2018). Rio de Janeiro, Brazil: VLDB Endowment Inc., 2018: 1289−1302
[14]	Caheny P, Alvarez L, Valero M, et al. Runtime-assisted cache coherence deactivation in task parallel programs[C] // Proc of the Int Conf for High Performance Computing, Networking, Storage and Analysis (SC 2018). Piscataway, NJ: IEEE, 2018: 454−465
[15]	He Ximing, Ma Sheng, Liu Wenjie, et al. VISU: A simple and efficient cache coherence protocol based on self-updating[C] // Proc of the Int Conf on Algorithms and Architectures for Parallel Processing (ICA3PP 2018). Cham, Switzerland: Springer, 2018: 341−357
[16]	Al-Waisi Z, O. Agyeman M. An overview of on-chip cache coherence protocols[C] // Proc of the Intelligent Systems Conf (IntelliSys 2017). Piscataway, NJ: IEEE, 2017: 304−309
[17]	Shukur H, Zeebaree S, Zebari R, et al. Cache coherence protocols in distributed systems[J]. Journal of Applied Science and Technology Trends, 2020, 1(3): 92−97 doi: 10.38094/jastt1329
[18]	Amory I A, Ahmed A H, Hasan Z. MESI protocol for multicore processors based on FPGA[J]. Periodicals of Engineering and Natural Sciences, 2021, 9(1): 80−89 doi: 10.21533/pen.v9i1.1772
[19]	Kaur D P, Sulochana V. Design and implementation of cache coherence protocol for high-speed multiprocessor system[C] //Proc of the 2018 2nd IEEE Int Conf on Power Electronics, Intelligent Control and Energy Systems (ICPEICES 2018). Piscataway, NJ: IEEE, 2018: 1097−1102
[20]	Nair A S, Pai A V, Raveendran B K, et al. MOESIL: A cache coherency protocol for locked mixed criticality l1 data cache[C] //Proc of the 2021 IEEE/ACM 25th Int Symp on Distributed Simulation and Real Time Applications (DS-RT 2021). Piscataway, NJ: IEEE, 2021: 1−8
[21]	Ibrahim R K, Jumma L F, Amory I A, et al. Design of MOESI protocol for multicore processors based on FPGA[J]. International Journal of Nonlinear Analysis and Applications, 2021, 12(Special Issue): 1229−1242
[22]	Duvvuri V, Shenoy P, Tewari R. Adaptive leases: A strong consistency mechanism for the world wide web[J]. IEEE Transactions on Knowledge and Data Engineering, 2003, 15(5): 1266−1276 doi: 10.1109/TKDE.2003.1232277
[23]	Gafni E, Lamport L. Disk Paxos[J]. Distributed Computing, 2003, 16(1): 1−20 doi: 10.1007/s00446-002-0070-8
[24]	Lamport L. Fast Paxos[J]. Distributed Computing, 2006, 19(2): 79−103 doi: 10.1007/s00446-006-0005-x
[25]	Kraska T, Pang G, Franklin M J, et al. MDCC: Multi-data center consistency[C] //Proc of the 8th ACM European Conf on Computer Systems (EuroSys 2013). New York: ACM, 2013: 113−126
[26]	Liu Shenling, Zhang Chunyuan, Chen Yujiao. DCC: Distributed cache consistency[C] //Proc of the Int Conf of Pioneering Computer Scientists, Engineers and Educators (ICPCSEE 2017). Berlin: Springer, 2017: 377−387
[27]	Jialin Li, Nelson J, Michael E, et al. Pegasus: Tolerating skewed workloads in distributed storage with in-network coherence directories[C] //Proc of the 14th USENIX Conf on Operating Systems Design and Implementation (OSDI 2020). Berkeley, CA: USENIX Association, 2020: 387−406
[28]	Attiya H, Gramoli V, Milani A. A provably starvation-free distributed directory protocol[C] //Proc of the Int Symp on Stabilizing, Safety, and Security of Distributed Systems (SSS 2010). Berlin: Springer, 2010: 405−419
[29]	陈志强,周宏伟,冯权友,等. 面向多核处理器的可配置缓存一致性协议设计与实现[J]. 计算机研究与发展,2021,58(6):1166−1175 Chen Zhiqiang, Zhou Hongwei, Feng Quanyou, et al. Design and implementation of configurable cache coherence protocol for multi-core processor[J]. Journal of Computer Research and Development, 2021, 58(6): 1166−1175 (in Chinese)
[30]	郑阳,陈海波,臧斌宇. 基于广播的分布式共享内存[J]. 上海交通大学学报,2018,52(10):1333−1338 doi: 10.16183/j.cnki.jsjtu.2018.10.023 Zheng Yang, Chen Haibo, Zang Binyu. Snooping-based distributed shared memory[J]. Journal of Shanghai Jiaotong University, 2018, 52(10): 1333−1338 (in Chinese) doi: 10.16183/j.cnki.jsjtu.2018.10.023
[31]	Liu Guoxin, Shen Haiying, Chandler H, et al. Measuring and evaluating live content consistency in a large-scale CDN[J]. IEEE Transactions on Parallel and Distributed Systems, 2016, 27(7): 2074−2090 doi: 10.1109/TPDS.2015.2479222
[32]	Bhat S, Vijayakumar V. Adaptive directory based cache coherency model to optimize the network bandwidth in distributed applications[J]. IOSR Journal of Engineering, 2018, 8(9): 41−46
[33]	Wang Qing, Lu Youyou, Xu E, et al. Concordia: Distributed shared memory with in-network cache coherence[C] //Proc of the 19th USENIX Conf on File and Storage Technologies (FAST 2021). Berkeley, CA: USENIX Association, 2021: 277−292

[1]	Wei Zheng, Dou Yu, Gao Yanzhen, Ma Jie, Sun Ninghui, Xing Jing. A Consistent Hash Data Placement Algorithm Based on Stripe[J]. Journal of Computer Research and Development, 2021, 58(4): 888-903. DOI: 10.7544/issn1000-1239.2021.20190732
[2]	Chen Maotang, Zheng Sheng’an, You Litong, Wang Jingyu, Yan Tian, Tu Yaofeng, Han Yinjun, Huang Linpeng. A Distributed Persistent Memory File System Based on RDMA Multicast[J]. Journal of Computer Research and Development, 2021, 58(2): 384-396. DOI: 10.7544/issn1000-1239.2021.20200369
[3]	Tian Junfeng, Wang Yanbiao. Causal-Pdh: Causal Consistency Model for NoSQL Distributed Data Storage Using HashGraph[J]. Journal of Computer Research and Development, 2020, 57(12): 2703-2716. DOI: 10.7544/issn1000-1239.2020.20190686
[4]	Chen Bo, Lu Youyou, Cai Tao, Chen Youmin, Tu Yaofeng, Shu Jiwu. A Consistency Mechanism for Distributed Persistent Memory File System[J]. Journal of Computer Research and Development, 2020, 57(3): 660-667. DOI: 10.7544/issn1000-1239.2020.20190074
[5]	He Ximing, Ma Sheng, Huang Libo, Chen Wei, Wang Zhiying. A Simple and Efficient Cache Coherence Protocol Based on Self-Updating[J]. Journal of Computer Research and Development, 2019, 56(4): 719-729. DOI: 10.7544/issn1000-1239.2019.20170898
[6]	Chen Jicheng, Li Yihan, Zhao Yaqian, Wang Endong, Shi Hongzhi, Tang Shibin. A Shared-Forwarding State Based Multiple-Tier Cache Coherency Protocol[J]. Journal of Computer Research and Development, 2017, 54(4): 764-774. DOI: 10.7544/issn1000-1239.2017.20160141
[7]	Wang Endong, Tang Shibin, Chen Jicheng, Wang Hongwei, Ni Fan, Zhao Yaqian. Directory Cache Design for Multi-Core Processor[J]. Journal of Computer Research and Development, 2015, 52(6): 1242-1253. DOI: 10.7544/issn1000-1239.2015.20150140
[8]	Zhao Xiaofei, Tian Dongping, Zhang Wenbo, Shi Zhongzhi. Properties and Distributed Tableaux Reasoning Algorithm for D3L(ccy)[J]. Journal of Computer Research and Development, 2014, 51(3): 570-579.
[9]	Dong Jian, Zuo Decheng, Liu Hongwei, Yang Xiaozong, and Ren Xiao. A Protocol of Fault Diagnosis Agreement Based on Invalid Link[J]. Journal of Computer Research and Development, 2007, 44(6): 914-923.
[10]	Xiong Jin, Fan Zhihua, Ma Jie, Tang Rongfeng, Li Hui, Meng Dan. Metadata Consistency in DCFS2[J]. Journal of Computer Research and Development, 2005, 42(6): 1019-1027.