ISSN 1000-1239 CN 11-1777/TP

Journal of Computer Research and Development ›› 2016, Vol. 53 ›› Issue (1): 53-67.doi: 10.7544/issn1000-1239.2016.20150656

Special Issue: 2016优青专题

Previous Articles     Next Articles

Aggregating Incast Transfers in Data Centers

Guo Deke, Luo Lailong, Li Yan, Hu Zhiyao, Ren Bangbang   

  1. (Key Laboratory on Information System and Engineering (National University of Defense Technology), Changsha 410073)
  • Online:2016-01-01

Abstract: Data transfers, such as the common shuffle and incast communication patterns, contribute most of the network traffic in MapReduce like working paradigms and thus have severe impacts on application performance in modern data centers. This motivates us to bring opportunities for performing the inter-flow data aggregation during the transmission phase as early as possible rather than just at the receiver side. In this paper, we first examine the gain and feasibility of the inter-flow data aggregation with novel data center network structures. To achieve such a gain, we model the incast minimal tree problem. We propose two approximate incast tree construction methods, RS-based and ARS-based incast trees. We are thus able to generate an efficient incast tree solely based on the labels of incast members and the data center topology. We further present incremental methods to tackle the dynamic and fault-tolerant issues of the incast tree. Based on a prototype implementation and large-scale simulations, we demonstrate that our approach can significantly decrease the amount of network traffic, save the data center resources, and reduce the delay of job processing. Our approach for BCube and FBFLY can be adapted to other data centers structures with minimal modifications.

Key words: in-network aggregation, data centers, incast transfer, shuffle transfer, network transfer

CLC Number: