ISSN 1000-1239 CN 11-1777/TP

计算机研究与发展 ›› 2017, Vol. 54 ›› Issue (12): 2858-2872.doi: 10.7544/issn1000-1239.2017.20160717

• 软件技术 • 上一篇    

基于RDD关键度的Spark检查点管理策略

英昌甜1,2,于炯2,卞琛2,王维庆1,3,鲁亮2,钱育蓉2   

  1. 1(新疆大学电气工程博士后科研流动站 乌鲁木齐 830046); 2(新疆大学软件学院 乌鲁木齐 830008); 3(新疆大学电气工程学院 乌鲁木齐 830046) (yingct@xju.edu.com)
  • 出版日期: 2017-12-01
  • 基金资助: 
    国家自然科学基金项目(61262088,61462079,61363083,61562086,51667020);新疆维吾尔自治区自然科学基金项目(2017D01A20);新疆维吾尔自治区高校科研计划(XJEDU2016S106)

Criticality Checkpoint Management Strategy Based on RDD Characteristics in Spark

Ying Changtian1,2, Yu Jiong2, Bian Chen2, Wang Weiqing1,3, Lu Liang2, Qian Yurong2   

  1. 1(Postdoctoral Research Station of Electrical Engineering, Xinjiang University, Urumqi 830046); 2(School of Software, Xinjiang University, Urumqi 830008); 3(School of Electrical Engineering, Xinjiang University, Urumqi 830046)
  • Online: 2017-12-01

摘要: Spark默认容错机制由程序员设置检查点,并利用弹性分布式数据集(resilient distributed dataset, RDD)的血统(lineage)进行计算.在应用程序复杂度高、迭代次数多以及数据量较大时,恢复过程需要耗费大量的计算开销.同时,在执行恢复任务时,仅考虑数据本地性选择节点,并未考虑节点的计算能力,这都会导致恢复时间增加,无法最大化发挥集群的性能.因此,在建立Spark执行模型、检查点模型和RDD关键度模型的基础上,提出一种基于关键度的检查点管理(criticality checkpoint management, CCM)策略,其中包括检查点设置算法、失效恢复算法和清理算法.其中检查点设置算法通过分析作业中RDD的属性以及对作业恢复时间的影响,选择关键度大的RDD作为检查点存储;恢复算法根据各节点的计算能力做出决策,选择合适的节点执行恢复任务;清理算法在磁盘空间不足时,清除关键度较低的检查点.实验结果表明:该策略在略增加执行时间的情况下,能够选择有备份价值的RDD作为检查点,在节点失效时能够有效地降低恢复开销,提高节点的磁盘有效利用率.

关键词: 内存计算, Spark, 检查点管理, 失效恢复, RDD属性

Abstract: The default fault tolerance mechanism of Spark is setting the checkpoint by programmer. When facing data loss, Spark recomputes the tasks based on the RDD lineage to recovery the data. Meanwhile, in the circumstance of complicated application with multiple iterations and large amount of input data, the recovery process may cost a lot of computation time. In addition, the recompute task only considers the data locality by default regardless the computing capabilities of nodes, which increases the length of recovery time. To reduce recovery cost, we establish and demonstrate the Spark execution model, the checkpoint model and the RDD critically model. Based on the theory, the criticality checkpoint management (CCM) strategy is proposed, which includes the checkpoint algorithm, the failure recovery algorithm and the cleaning algorithm. The checkpoint algorithm is used to analyze the RDD charactersitics and its influence on the recovery time, and selects valuable RDDs as checkpoints. The failure recovery algorithm is used to choose the appropriate nodes to recompute the lost RDDs, and cleaning algorithm cleans checkpoints when the disk space becomes insufficient. Experimental results show that: the strategy can reduce the recovery overhead efficiently, select valuable RDDs as checkpoints, and increase the efficiency of disk usage on the nodes with sacrificing the execution time slightly.

Key words: memory computing, Spark, checkpoint management, failure recovery, RDD characteristics

中图分类号: