ISSN 1000-1239 CN 11-1777/TP

计算机研究与发展 ›› 2019, Vol. 56 ›› Issue (2): 227-239.doi: 10.7544/issn1000-1239.2019.20170849

• 综述 •    下一篇



  1. 1(清华大学计算机科学与技术系 北京 100084); 2(中兴通讯股份有限公司 南京 210012) (
  • 出版日期: 2019-02-01
  • 基金资助: 

Survey on RDMA-Based Distributed Storage Systems

Chen Youmin1, Lu Youyou1, Luo Shengmei2, Shu Jiwu1   

  1. 1(Department of Computer Science and Technology, Tsinghua University, Beijing 100084); 2(ZTE Corporation, Nanjing 210012)
  • Online: 2019-02-01

摘要: 远程直接内存访问(remote direct memory access, RDMA)技术正在大数据领域被越来越广泛地应用,它支持在对方主机CPU不参与的情况下远程读写异地内存,并提供高带宽、高吞吐和低延迟的数据传输特性,从而大幅提升分布式存储系统的性能,因此基于RDMA的分布式存储系统将为满足大数据高时效处理和存储带来新的机遇.首先分析了基于RDMA的分布式存储系统简单替换网络传输模块并不能充分发挥RDMA在语义和性能上的优势的原因,并指出存储系统架构需要变革的因素.然后阐述了高效运用RDMA技术主要取决于2个方面:第1方面是硬件资源的高效管理,包括网卡缓存和CPU缓存的合理利用、多核CPU的并行加速以及内存资源管理等;第2方面是软硬件的紧耦合设计,借助RDMA在语义和性能上的特性,重构新型数据组织和索引方式、优化分布式协议等.同时,以分布式文件系统、分布式键值存储和分布式事务系统为典型应用场景,分别阐述了它们在硬件资源管理和软件重构这2个方面的相关研究.最后,给出了总结和展望.

关键词: 远程直接内存访问, 分布式存储, 键值存储, 文件系统, 分布式事务

Abstract: RDMA (remote direct memory access) is being widely used in big data area, which allows local host to access the remote memory without the involvements of remote CPUs, and provides extremely high bandwidth, high throughput and low latency, thus helping to boost the performance of distributed storage systems dramatically. As a whole, the RDMA-enabled distributed storage systems bring new opportunity to the big data processing. In this paper, we firstly point out that simply replacing the network module in distributed systems cannot fully exploit the advantages of RDMA in both semantics and efficiency, and revolutions of storage system design are urgently needed. Then, two key aspects of efficiently using RDMA are illustrated: One is the efficient management of hardware resources, including the careful utilization of NIC an CPU cache, parallel acceleration of multicore CPUs and memory management, and the other is the reformation of the software by closely coupling the software design and RDMA semantics, which uses the new features of RDMA to redesign the data placement schemes, data indexing and distributed protocols. Relative research works of distributed file systems, distributed key-value stores, and distributed transactional systems are introduced to illustrate the above two aspects. Summarizes of the paper, and suggestions for future research are also given at the end of this paper.

Key words: remote direct memory access, distributed storage, key-value store, file system, distri-buted transaction