ISSN 1000-1239 CN 11-1777/TP

计算机研究与发展 ›› 2016, Vol. 53 ›› Issue (9): 2107-2131.doi: 10.7544/issn1000-1239.2016.20148443

• 软件技术 • 上一篇    

MapReduce能耗建模及优化分析

廖彬1,4,张陶2,3,于炯2,4,尹路通4,郭刚4,国冰磊2,4   

  1. 1(新疆财经大学统计与信息学院 乌鲁木齐 830012); 2(新疆大学信息科学与工程学院 乌鲁木齐 830046); 3(新疆医科大学医学工程技术学院 乌鲁木齐 830011) ; 4(新疆大学软件学院 乌鲁木齐 830008) (liaobin665@163.com)
  • 出版日期: 2016-09-01
  • 基金资助: 
    国家自然科学基金项目(61562078,61262088,71261025);新疆财经大学博士科研启动基金项目(2015BS007)

Energy Consumption Modeling and Optimization Analysis for MapReduce

Liao Bin1,4, Zhang Tao2,3, Yu Jiong2,4, Yin Lutong4, Guo Gang4, Guo Binglei2,4   

  1. 1(School of Statistics and Information, Xinjiang University of Finance and Economics, Urumqi 830012);2(School of Information Science and Engineering, Xinjiang University, Urumqi 830046);3(College of Medical Engineering and Technology, Xinjiang Medical University, Urumqi 830011);4(School of Software, Xinjiang University, Urumqi 830008)
  • Online: 2016-09-01

摘要: 云计算中心规模的不断扩大以及设计时对能耗因素的忽略,使其日益暴露出高能耗低效率的问题.为提高MapReduce框架能耗利用率,首先对MapReduce任务进行了能耗建模,提出基于CPU利用率估算、主要部件能耗累加及平均功耗估算的任务能耗模型,并在此基础上建立了MapReduce作业能耗模型.其次,基于能耗模型对能耗优化进行了分析,提出从优化MapReduce作业执行能耗、减少MapReduce任务等待能耗与提高MapReduce集群能源利用效率3个方向对MapReduce进行能耗优化.再次,提出异构环境下的数据放置策略减小MapReduce任务等待能耗,提出截止时间约束下的最小资源分配方法提高MapReduce作业能耗利用效率.通过大量的实验及能耗数据分析,验证了能耗模型及能耗优化方法的有效性.

关键词: 绿色计算, 任务调度, 能耗建模, 节能分析, 数据布局

Abstract: The continuous expansion of the cloud computing centers scale and neglect of energy consumption factors exposed the problem of high energy consumption and low efficiency. To improve the MapReduce framework utilization of energy consumption, we build an energy consumption model for MapReduce framework. First, we propose a task energy consumption model which is based on CPU utilization estimation, energy consumption accumulation of main components and the average energy consumption estimation as well as the job energy consumption model of MapReduce. Specifically, after analyzing the energy optimization under energy consumption model, we come up with three directions to optimize energy consumption of MapReduce: optimize MapReduce energy consumption of job execution, reduce MapReduce energy consumption of task waiting and improve the energy utilization rate of MapReduce cluster. We further propose the data placement policy to decrease energy consumption of task waiting under heterogeneous environment and the minimum resource allocation algorithms to improve energy utilization rate of MapReduce jobs by the deadline constraints. A large number of experiments and data analysis of energy consumption demonstrate the effectiveness of energy consumption model and optimum policy of energy consumption.

Key words: green computing, task scheduling, energy consumption modeling, energy analysis, data layout

中图分类号: