• 中国精品科技期刊
  • CCF推荐A类中文期刊
  • 计算领域高质量科技期刊T1类
Advanced Search
Wang Chenxi, Lü Fang, Cui Huimin, Cao Ting, John Zigman, Zhuang Liangji, Feng Xiaobing. Heterogeneous Memory Programming Framework Based on Spark for Big Data Processing[J]. Journal of Computer Research and Development, 2018, 55(2): 246-264. DOI: 10.7544/issn1000-1239.2018.20170687
Citation: Wang Chenxi, Lü Fang, Cui Huimin, Cao Ting, John Zigman, Zhuang Liangji, Feng Xiaobing. Heterogeneous Memory Programming Framework Based on Spark for Big Data Processing[J]. Journal of Computer Research and Development, 2018, 55(2): 246-264. DOI: 10.7544/issn1000-1239.2018.20170687

Heterogeneous Memory Programming Framework Based on Spark for Big Data Processing

More Information
  • Published Date: January 31, 2018
  • Due to the boom of big data applications, the amount of data being processed by servers is increasing rapidly. In order to improve processing and response speed, industry is deploying in-memory big data computing systems, such as Apache Spark. However, traditional DRAM memory cannot satisfy the large memory request of these systems for the following reasons: firstly, the energy consumption of DRAM can be as high as 40% of the total; secondly, the scaling of DRAM manufacturing technology is hitting the limit. As a result, heterogeneous memory integrating DRAM and NVM (non-volatile memory) is a promising candidate for future memory systems. However, because of the longer latency and lower bandwidth of NVM compared with DRAM, it is necessary to place data in appropriate memory module to achieve ideal performance. This paper analyzes the memory access behavior of Spark applications and proposes a heterogeneous memory programming framework based on Spark. It is easy to apply this framework to existing Spark applications without rewriting the code. Experiments show that for Spark benchmarks, by utilizing our framework, only placing 20%~25% data on DRAM and the remaining on NVM can reach 90% of the performance when all the data is placed on DRAM. This leads to an improved performance-dollar ratio compared with DRAM-only servers and the potential support for larger scale in-memory computing applications.
  • Related Articles

    [1]Zeng Weixin, Zhao Xiang, Tang Jiuyang, Tan Zhen, Wang Wei. Iterative Entity Alignment via Re-Ranking[J]. Journal of Computer Research and Development, 2020, 57(7): 1460-1471. DOI: 10.7544/issn1000-1239.2020.20190643
    [2]Dai Chenchao, Wang Hongyuan, Ni Tongguang, Chen Shoubing. Person Re-Identification Based on Deep Convolutional Generative Adversarial Network and Expanded Neighbor Reranking[J]. Journal of Computer Research and Development, 2019, 56(8): 1632-1641. DOI: 10.7544/issn1000-1239.2019.20190195
    [3]Du Ruizhong, Li Mingyue, Tian Junfeng. Multi-keyword Ranked Ciphertext Retrieval Scheme Based on Clustering Index[J]. Journal of Computer Research and Development, 2019, 56(3): 555-565. DOI: 10.7544/issn1000-1239.2019.20170830
    [4]Guo Jiafeng, Fan Yixing. Exploration on Neural Information Retrieval Framework[J]. Journal of Computer Research and Development, 2018, 55(9): 1987-1999. DOI: 10.7544/issn1000-1239.2018.20180133
    [5]Zhong Qi, Wang Jing, Guan Xuetao, Huang Tao, Wang Keyi. Data Object Scale Aware Rank-Level Memory Allocation[J]. Journal of Computer Research and Development, 2014, 51(3): 672-680.
    [6]Liu Xiping, Wan Changxuan, and Liu Dexi. Effective XML Vague Content and Structure Retrieval and Scoring[J]. Journal of Computer Research and Development, 2010, 47(6): 1070-1078.
    [7]Xu Cunlu, Chen Yanqiu, Lu Hanqing. Statistical Landscape Features for Texture Retrieval[J]. Journal of Computer Research and Development, 2006, 43(4): 702-707.
    [8]Xing Qiang, Yuan Baozong, and Tang Xiaofang. A Fast Image Retrieval Method Based on Weighted Chromaticity Histogram[J]. Journal of Computer Research and Development, 2005, 42(11): 1903-1910.
    [9]Ru Liyun, Ma Shaoping, and Lu Jing. Feature Fusion Based on the Average Precision in Image Retrieval[J]. Journal of Computer Research and Development, 2005, 42(9): 1640-1646.
    [10]Zhang Min, Lin Chuan, and Ma Shaoping. Dynamic Parameter Learning Approach for Information Retrieval with Genetic Algorithm[J]. Journal of Computer Research and Development, 2005, 42(3).

Catalog

    Article views PDF downloads Cited by()

    /

    DownLoad:  Full-Size Img  PowerPoint
    Return
    Return