高级检索
    王有为, 王伟平, 孟丹. 基于统计方法的Hive数据仓库查询优化实现[J]. 计算机研究与发展, 2015, 52(6): 1452-1462. DOI: 10.7544/issn1000-1239.2015.20140403
    引用本文: 王有为, 王伟平, 孟丹. 基于统计方法的Hive数据仓库查询优化实现[J]. 计算机研究与发展, 2015, 52(6): 1452-1462. DOI: 10.7544/issn1000-1239.2015.20140403
    Wang Youwei, Wang Weiping, Meng Dan. Query Optimization by Statistical Approach for Hive Data Warehouse[J]. Journal of Computer Research and Development, 2015, 52(6): 1452-1462. DOI: 10.7544/issn1000-1239.2015.20140403
    Citation: Wang Youwei, Wang Weiping, Meng Dan. Query Optimization by Statistical Approach for Hive Data Warehouse[J]. Journal of Computer Research and Development, 2015, 52(6): 1452-1462. DOI: 10.7544/issn1000-1239.2015.20140403

    基于统计方法的Hive数据仓库查询优化实现

    Query Optimization by Statistical Approach for Hive Data Warehouse

    • 摘要: Map/Reduce是海量离线数据分析中广泛应用的并行编程模型.Hive数据仓库基于Map/Reduce实现了查询处理引擎,然而Map/Reduce框架在处理偏斜数据时会出现工作负载分布不均的问题.均衡计算模型(computation balanced model, CBM),其核心思想是通过数据分布特征指导查询计划优化.相应研究贡献包括2部分,首先针对应用极广的GroupBy查询和Join查询建立了运行估价模型,确定了不同场景下查询计划的优化选择分支;其次基于Hive ETL机制设计了一种统计信息收集方法,解决了统计海量数据分布特征的问题.实验数据表明,通过CBM优化的 GroupBy查询耗时节省了8%~45%,Join查询耗时节省了12%~46%;集群CPU负载均衡指标优化了60%~80%,I/O负载均衡指标优化了60%~90%.实验结果证实了基于CBM模型优化的查询计划生成器能显著均衡化Hive查询运行时的集群负载,并优化了查询处理效率.

       

      Abstract: Map/Reduce is an efficient parallel programming model, which is now widely utilized to analyze massive data. Hive is an open source data warehouse which utilizes Map/Reduce to implement its query processing engine. However, the issue of unbalanced workload distribution in the whole cluster arises when processing skewed data using Map/Reduce. The possible results range from low runtime efficiency to task failures. To solve such problem, we propose an approach named the computation balanced model (CBM), which optimizes to queries by using distribution statistics. The main contributions of this paper include two parts correspondingly: firstly, the runtime cost evaluation model is established for two widely-used types of queries, i.e., the GroupBy and Join queries, especially under different situations; secondly, the highly-efficient statistics approach for massive data is designed and implemented adapting to the data access mechanism of Hive. Experiment results show the processing time of GroupBy query optimized by CBM is reduced by about 8%-45%, while the processing time of Join query is reduced by over 12%-46%. And the balance distribution of cluster payload is improved by about 60%-80% for CPU and 60%-90% for I/O. We believe the optimized query plan generator by CBM significantly balances the payload distribution during the execution of Map/Reduce tasks, as well as improves the query efficiency greatly.

       

    /

    返回文章
    返回