• 中国精品科技期刊
  • CCF推荐A类中文期刊
  • 计算领域高质量科技期刊T1类
Advanced Search
Bao Han, Wang Yijie, Xu Fangliang. A Cross-Datacenter Erasure Code Writing Method Based on Generator Matrix Transformation[J]. Journal of Computer Research and Development, 2020, 57(2): 291-305. DOI: 10.7544/issn1000-1239.2020.20190542
Citation: Bao Han, Wang Yijie, Xu Fangliang. A Cross-Datacenter Erasure Code Writing Method Based on Generator Matrix Transformation[J]. Journal of Computer Research and Development, 2020, 57(2): 291-305. DOI: 10.7544/issn1000-1239.2020.20190542

A Cross-Datacenter Erasure Code Writing Method Based on Generator Matrix Transformation

Funds: This work was supported by the National Key Research and Development Program of China (2016YFB1000101), the National Natural Science Foundation of China (61379052), the Science Foundation of Ministry of Education of China (2018A02002), and the Natural Science Foundation for Distinguished Young Scholars of Hunan Province (14JJ1026).
More Information
  • Published Date: January 31, 2020
  • In cross-datacenter storage systems, existing writing methods of erasure code usually has low encoding efficiency, low transmission efficiency, and large network resource consumption. Therefore, cross-datacenters erasure code usually has a low writing rate. This paper proposes a cross-datacenter erasure code writing method based on generator matrix transformation called CREW. Specifically, we first propose a greedy strategy-based transmission topology construction algorithm called GBTC, which can construct a tree-structured transmission topology with incremental weights (the weights are set to the network distances between datacenters) from top to bottom to organize data transmission between datacenters. Then, we propose a generator matrix transformation algorithm called GMT. Without changing the linear relationship of coded blocks, GMT can transform the generator matrix so that the number of data blocks related to a coded block is negatively correlated with the network distance between the datacenter where the coded block is located and the root of the tree-structured topology. Therefore, CREW only needs to transfer a small number of data blocks through a long network distance to write data. Thus, the network resource consumption is reduced. Finally, we propose a distributed pipelined writing algorithm called DPW to distribute encoding operations to different nodes for parallel execution and limit the number of forwards of data blocks, thereby improving encoding efficiency and transmission efficiency. Experiments show that compared with writing methods of traditional erasure code, the write rate of CREW is increased by 36.3%~57.9%. And compared with the existing writing method of cross-datacenter erasure code (IncEncoding), the writing rate of CREW is increased by 32.4%.
  • Related Articles

    [1]Yang Fan, Zhang Peng, Wang Zhan, Yuan Guojun, An Xuejun. Accelerating Byzantine Fault Tolerance with In-Network Computing[J]. Journal of Computer Research and Development, 2021, 58(1): 164-177. DOI: 10.7544/issn1000-1239.2021.20190723
    [2]Yang Hongzhang, Yang Yahui, Tu Yaofeng, Sun Guangyu, Wu Zhonghai. Proactive Fault Tolerance Based on “Collection—Prediction—Migration—Feedback” Mechanism[J]. Journal of Computer Research and Development, 2020, 57(2): 306-317. DOI: 10.7544/issn1000-1239.2020.20190549
    [3]Ouyang Yiming, Sun Chenglong, Li Jianhua, Liang Huaguo, Huang Zhengfeng, Du Gaoming. Addressing Transient and Intermittent Link Faults in NoC with Fault-Tolerant Method[J]. Journal of Computer Research and Development, 2017, 54(5): 1109-1120. DOI: 10.7544/issn1000-1239.2017.20151017
    [4]Jia Jia, Yang Xuejun, Li Zhiling. A Redundancy-Multithread-Based Multiple GPU Copies Fault-Tolerance Technique[J]. Journal of Computer Research and Development, 2013, 50(7): 1551-1562.
    [5]Xie Min, Lu Yutong, Zhou Enqiang, Cao Hongjia, and Yang Xuejun. Implementation and Evaluation of MPI Checkpointing System over Lustre File System[J]. Journal of Computer Research and Development, 2007, 44(10): 1709-1716.
    [6]Liu Dong, Zhang Chunyuan, Li Rui, Huang Ying, and Li Yi. Fault-Tolerant Real-Time Scheduling Algorithm in Software Fault-Tolerant Module[J]. Journal of Computer Research and Development, 2007, 44(9): 1495-1500.
    [7]Zhang Sanfeng and Wu Guoxin. A Fault-Tolerant Asymmetric DHT Method Towards Dynamic and Heterogeneous Network[J]. Journal of Computer Research and Development, 2007, 44(6): 905-913.
    [8]Fu Zhongchuan, Chen Hongsong, Cui Gang, and Yang Xiaozong. Processor Fault-Tolerance Technology Research and Prospect[J]. Journal of Computer Research and Development, 2007, 44(1): 154-160.
    [9]Xiong Tinggang, Ma Zhong, Yuan Youguang. Research on Synchronization Technology of Fault-Tolerant Computer System Based on Operating System Calls[J]. Journal of Computer Research and Development, 2006, 43(11): 1985-1992.
    [10]Wang Di, Xue Wei, Shu Jiwu, and Shen Meiming. Fault Tolerance with Virtual Disk Replicas in the Mass Storage Network[J]. Journal of Computer Research and Development, 2006, 43(10): 1849-1854.
  • Cited by

    Periodical cited type(6)

    1. 张凯鑫 ,王意洁 ,包涵 ,阚浚晖 . 面向存算联调的跨云纠删码自适应数据访问方法. 计算机研究与发展. 2024(03): 571-588 . 本站查看
    2. 周杨,王春林,郭锐. 基于随机森林算法的数据中心运维异常告警方法. 现代电子技术. 2023(08): 143-148 .
    3. 包涵,王意洁. 低跨云数据中心修复流量的纠删码的快速构造方法. 计算机研究与发展. 2023(10): 2418-2439 . 本站查看
    4. 刘元莹. 基于目录哈希树的电力通信网络数据容灾备份. 电子设计工程. 2022(19): 102-105+110 .
    5. 张淑清. 基于哈希计算的大数据冗余消除算法设计. 微型电脑应用. 2021(12): 68-70 .
    6. 陈建兵,梁立,叶志霞. 有限拓扑的编码算法. 云南师范大学学报(自然科学版). 2020(05): 42-46 .

    Other cited types(5)

Catalog

    Article views (887) PDF downloads (244) Cited by(11)

    /

    DownLoad:  Full-Size Img  PowerPoint
    Return
    Return