ISSN 1000-1239 CN 11-1777/TP

计算机研究与发展 ›› 2020, Vol. 57 ›› Issue (12): 2683-2693.doi: 10.7544/issn1000-1239.2020.20190683

• 网络技术 • 上一篇    下一篇

一种分布式异构带宽环境下的高效数据分区方法

马卿云1,季航旭1,赵宇海1,毛克明2,王国仁3   

  1. 1(东北大学计算机科学与工程学院 沈阳 110169);2(东北大学软件学院 沈阳 110169);3(北京理工大学计算机学院 北京 100081) (maqy1995@163.com)
  • 出版日期: 2020-12-01
  • 基金资助: 
    国家重点研发计划项目(2018YFB1004402);国家自然科学基金项目(61772124)

An Efficient Data Partitioning Method in Distributed Heterogeneous Bandwidth Environment

Ma Qingyun1, Ji Hangxu1, Zhao Yuhai1, Mao Keming2, Wang Guoren3   

  1. 1(School of Computer Science and Engineering, Northeastern University, Shenyang 110169);2(Software College, Northeastern University, Shenyang 110169);3(School of Computer Science and Technology, Beijing Institute of Technology, Beijing 100081)
  • Online: 2020-12-01
  • Supported by: 
    This work was supported by the National Key Research and Development Program of China (2018YFB1004402) and the National Natural Science Foundation of China (61772124).

摘要: 在分布式大数据处理框架的作业运行过程中,会有大量的数据通过网络传输,数据在各节点之间传输所需的时间已成为作业运行的主要开销之一.在节点异构带宽的情况下,因为带宽瓶颈节点的存在,传统的数据分区方法效率低下.针对这个问题,建立了节点间的数据传输模型,该模型以降低数据传输时间为目标,根据各节点的上下行带宽和初始数据量大小,计算出各节点的最优数据分发比例.以该模型为基础,设计了基于带宽的数据分区方法,该数据分区方法使得各节点按最优数据分发比例来分配数据.最后在Apache Flink框架中将基于带宽的数据分区方法进行了实现,并通过实验进行了验证.实验结果表明:异构带宽条件下,基于带宽的数据分区方法可以有效减少数据分区所需的时间.

关键词: 数据分区, Apache Flink, 负载均衡, 异构带宽, 分布式系统

Abstract: A large quantity of data is transmitted through the network during the process in distributed big data processing framework, resulting in the time consumption for data transmission between each node becomes one of the main costs of the operation. However, in the case of heterogeneous bandwidth of nodes, traditional data partitioning methods such as Hash partitioning or range partitioning will be inefficient, due to the existence of bandwidth bottleneck nodes. Data partitioning is necessary for big data processing and inefficient data partitioning methods would significantly increase the running time of jobs. We therefore propose a data transmission model between nodes to reduce time consumption in distributed heterogeneous bandwidth networks. The model calculates each node’s optimal data distribution ratio to minimize the data transfer time, according to its uplink and downlink bandwidth as well as the initial data size. Besides, a bandwidth-based data partitioning method is designed based on the proposed model, enabling each node to allocate data under the optimal data distribution ratio. We demonstrate the effectiveness of our bandwidth-based data partitioning method through the implementation in the Apache Flink framework and have significantly improved efficiency. Extensive experimental results show that the bandwidth-based data partitioning method can effectively reduce the time consumption of data partitioning in distributed heterogeneous bandwidth conditions.

Key words: data partitioning, Apache Flink, load balancing, heterogeneous bandwidth, distributed system

中图分类号: