ISSN 1000-1239 CN 11-1777/TP

Journal of Computer Research and Development ›› 2022, Vol. 59 ›› Issue (3): 597-616.doi: 10.7544/issn1000-1239.20210577

Special Issue: 2022存储系统与智能处理专题

Previous Articles     Next Articles

Near-Data Processing-Based Parallel Compaction Optimization for Key-Value Stores

Sun Hui1, Lou Bendong1, Huang Jianzhong2, Zhao Yuhong3, Fu Song4   

  1. 1(School of Computer Science and Technology, Anhui University, Hefei 230601);2(Wuhan National Laboratory for Optoelectronics, Huazhong University of Science and Technology, Wuhan 430074);3(Institute of Information Engineering, Chinese Academy of Sciences, Beijing 100093);4(Department of Computer Science and Engineering, University of North Texas, Denton, TX, USA 76203)
  • Online:2022-03-07
  • Supported by: 
    This work was supported by the University Synergy Innovation Program of Anhui Province (GXXT-2019-007), the State Key Laboratory of Computer Architecture (ICT, CAS) (CARCH201915), and the National Natural Science Foundation of China (62072001, 61702004, 61572209).

Abstract: Large-scale unstructured data management brings unprecedented challenges to existing relational databases. The log-structured merge tree (LSM-tree) based key-value store has been widely used and plays an essential role in data-intensive applications. The LSM-tree can convert random-write operations into sequential ones, thereby improving write performance. However, the LSM-tree key-value storage system also has some problems. First, the key-value storage system uses compaction operations to update data to balance system performance, but it impacts system performance and causes serious write amplification. Second, the traditional computing-centric data transmission also limits the overall system performance in compaction. This paper applied the data-centric near-data processing (NDP) model in the storage system. We propose a collaborative parallel compaction optimization for LSM-tree key-value stores named CoPro. The two parallel (i.e., data and pipeline parallelism) are fully utilized to improve compaction performance. When the compaction is triggered, the host-side CoPro determines the partitioning ratio of the compaction tasks according to the offloading strategy and divides tasks according to the ratio. Then, compaction subtasks are offloaded to the host and device sides, respectively, through the semantic management module. We design a decision component in the host-side and device-side CoPro, which is remarked as CoPro+. CoPro+ can dynamically adjust the parallelism according to changes in the resource of system and the value of key-value pairs in workloads. Extensive experimental results validate the benefits of CoPro compared with two popular NDP-based key-value stores.

Key words: log-structured merge tree (LSM-tree), key-value store, near-data processing (NDP), task offloading, data-pipeline parallelism

CLC Number: