ISSN 1000-1239 CN 11-1777/TP

Journal of Computer Research and Development ›› 2017, Vol. 54 ›› Issue (5): 1097-1108.doi: 10.7544/issn1000-1239.2017.20151062

Previous Articles     Next Articles

An Orthogonal Decomposition Based Design Method and Implementation for Big Data Processing System

Xiang Xiaojia1, Zhao Xiaofang1, Liu Yang1, Gong Guanjun1, Zhang Han2   

  1. 1(Institute of Computing Technology, Chinese Academy of Science, Beijing 100190); 2(School of Computer Science, North China University of Technology, Beijing 100144)
  • Online:2017-05-01

Abstract: Big data stimulates a revolution in data storage and processing field, resulting in the thriving of big data processing systems, such as Hadoop, Spark, etc, which build a brand new platform with platform independence, high throughput, and good scalability. On the other hand, substrate platform underpinning these systems are ignored because their designation and optimization mainly focus on the processing model and related frameworks & algorithms. We here present a new loose coupled, platform dependent big data processing system designation & optimization method which can exploit the power of underpinning platform, including OS and hardware, and get more benefit from these local infrastructures. Furthermore, based on local OS and hardware, two strategies, that is, lock-free based storage and super optimization based data processing execution engine, are proposed. Directed by the aforementioned methods and strategies, we present Arion, a modified version of vanilla Hadoop, which show us a new promising way for Hadoop optimization, meanwhile keeping its high scalability and upper layer platform independence. Our experiments prove that the prototype Arion can accelerate big data processing jobs up to 7.7%.

Key words: big data processing system, computing framework, localization, lock free, super optimization, excecution engine

CLC Number: