• 中国精品科技期刊
  • CCF推荐A类中文期刊
  • 计算领域高质量科技期刊T1类
Advanced Search
Wang Chenxi, Lü Fang, Cui Huimin, Cao Ting, John Zigman, Zhuang Liangji, Feng Xiaobing. Heterogeneous Memory Programming Framework Based on Spark for Big Data Processing[J]. Journal of Computer Research and Development, 2018, 55(2): 246-264. DOI: 10.7544/issn1000-1239.2018.20170687
Citation: Wang Chenxi, Lü Fang, Cui Huimin, Cao Ting, John Zigman, Zhuang Liangji, Feng Xiaobing. Heterogeneous Memory Programming Framework Based on Spark for Big Data Processing[J]. Journal of Computer Research and Development, 2018, 55(2): 246-264. DOI: 10.7544/issn1000-1239.2018.20170687

Heterogeneous Memory Programming Framework Based on Spark for Big Data Processing

More Information
  • Published Date: January 31, 2018
  • Due to the boom of big data applications, the amount of data being processed by servers is increasing rapidly. In order to improve processing and response speed, industry is deploying in-memory big data computing systems, such as Apache Spark. However, traditional DRAM memory cannot satisfy the large memory request of these systems for the following reasons: firstly, the energy consumption of DRAM can be as high as 40% of the total; secondly, the scaling of DRAM manufacturing technology is hitting the limit. As a result, heterogeneous memory integrating DRAM and NVM (non-volatile memory) is a promising candidate for future memory systems. However, because of the longer latency and lower bandwidth of NVM compared with DRAM, it is necessary to place data in appropriate memory module to achieve ideal performance. This paper analyzes the memory access behavior of Spark applications and proposes a heterogeneous memory programming framework based on Spark. It is easy to apply this framework to existing Spark applications without rewriting the code. Experiments show that for Spark benchmarks, by utilizing our framework, only placing 20%~25% data on DRAM and the remaining on NVM can reach 90% of the performance when all the data is placed on DRAM. This leads to an improved performance-dollar ratio compared with DRAM-only servers and the potential support for larger scale in-memory computing applications.
  • Related Articles

    [1]Wang Chuang, Ding Yan, Huang Chenlin, Song Liantao. Bitsliced Optimization of SM4 Algorithm with the SIMD Instruction Set[J]. Journal of Computer Research and Development, 2024, 61(8): 2097-2109. DOI: 10.7544/issn1000-1239.202220531
    [2]Li Maowen, Qu Guoyuan, Wei Dazhou, Jia Haipeng. Performance Optimization of Neural Network Convolution Based on GPU Platform[J]. Journal of Computer Research and Development, 2022, 59(6): 1181-1191. DOI: 10.7544/issn1000-1239.20200985
    [3]Shen Jie, Long Biao, Jiang Hao, Huang Chun. Implementation and Optimization of Vector Trigonometric Functions on Phytium Processors[J]. Journal of Computer Research and Development, 2020, 57(12): 2610-2620. DOI: 10.7544/issn1000-1239.2020.20190721
    [4]Zhang Jun, Xie Jingcheng, Shen Fanfan, Tan Hai, Wang Lümeng, He Yanxiang. Performance Optimization of Cache Subsystem in General Purpose Graphics Processing Units: A Survey[J]. Journal of Computer Research and Development, 2020, 57(6): 1191-1207. DOI: 10.7544/issn1000-1239.2020.20200113
    [5]Sun Chang’ai, Wang Zhen, Pan Lin. Optimized Mutation Testing Techniques for WS-BPEL Programs[J]. Journal of Computer Research and Development, 2019, 56(4): 895-905. DOI: 10.7544/issn1000-1239.2019.20180037
    [6]Liu Song, Wu Weiguo, Zhao Bo, Jiang Qing. Loop Tiling for Optimization of Locality and Parallelism[J]. Journal of Computer Research and Development, 2015, 52(5): 1160-1176. DOI: 10.7544/issn1000-1239.2015.20131387
    [7]Wang Yongxian, Zhang Lilun, Che Yonggang, Xu Chuanfu, Liu Wei, Cheng Xinghua. Heterogeneous Computing and Optimization on Tianhe-2,Supercomputer System for High-Order Accurate CFD Applications[J]. Journal of Computer Research and Development, 2015, 52(4): 833-842. DOI: 10.7544/issn1000-1239.2015.20131922
    [8]Gu Rong, Yan Jinshuang, Yang Xiaoliang, Yuan Chunfeng, and Huang Yihua. Performance Optimization for Short Job Execution in Hadoop MapReduce[J]. Journal of Computer Research and Development, 2014, 51(6): 1270-1280.
    [9]Luo Hongbing, Zhang Xiaoxia, Wang Wei, and Wu Linping. Instruction Level Parallel Optimizing for Scientific Computing Application[J]. Journal of Computer Research and Development, 2014, 51(6): 1263-1269.
    [10]Li Lei, Niu Chunlei, Chen Ningjiang, Wei Jun. A High-Performance Strategy for Optimizing Web Services[J]. Journal of Computer Research and Development, 2007, 44(7): 1191-1198.
  • Cited by

    Periodical cited type(5)

    1. 郭炜杰,包晓安. 基于Ajax的智能终端一次性口令身份认证仿真. 计算机仿真. 2023(07): 176-179 .
    2. 罗娟,章翠君,王纯. 基于众包的多楼层定位方法. 计算机研究与发展. 2022(02): 452-462 . 本站查看
    3. 胡美慧,向志威. 基于离散余弦变换的电力营销系统客户权限自动识别方法. 自动化技术与应用. 2022(05): 125-129 .
    4. 赵鹏飞. 港口身份智能识别系统设计与实现. 舰船科学技术. 2021(14): 202-204 .
    5. 倪志文,马小虎,孙霄,边丽娜. 结合显式和隐式特征交互的深度融合模型. 计算机工程. 2020(03): 87-92+98 .

    Other cited types(9)

Catalog

    Article views (1365) PDF downloads (732) Cited by(14)

    /

    DownLoad:  Full-Size Img  PowerPoint
    Return
    Return