Advanced Search
    Xiang Xiaojia, Zhao Xiaofang, Liu Yang, Gong Guanjun, Zhang Han. An Orthogonal Decomposition Based Design Method and Implementation for Big Data Processing System[J]. Journal of Computer Research and Development, 2017, 54(5): 1097-1108. DOI: 10.7544/issn1000-1239.2017.20151062
    Citation: Xiang Xiaojia, Zhao Xiaofang, Liu Yang, Gong Guanjun, Zhang Han. An Orthogonal Decomposition Based Design Method and Implementation for Big Data Processing System[J]. Journal of Computer Research and Development, 2017, 54(5): 1097-1108. DOI: 10.7544/issn1000-1239.2017.20151062

    An Orthogonal Decomposition Based Design Method and Implementation for Big Data Processing System

    • Big data stimulates a revolution in data storage and processing field, resulting in the thriving of big data processing systems, such as Hadoop, Spark, etc, which build a brand new platform with platform independence, high throughput, and good scalability. On the other hand, substrate platform underpinning these systems are ignored because their designation and optimization mainly focus on the processing model and related frameworks & algorithms. We here present a new loose coupled, platform dependent big data processing system designation & optimization method which can exploit the power of underpinning platform, including OS and hardware, and get more benefit from these local infrastructures. Furthermore, based on local OS and hardware, two strategies, that is, lock-free based storage and super optimization based data processing execution engine, are proposed. Directed by the aforementioned methods and strategies, we present Arion, a modified version of vanilla Hadoop, which show us a new promising way for Hadoop optimization, meanwhile keeping its high scalability and upper layer platform independence. Our experiments prove that the prototype Arion can accelerate big data processing jobs up to 7.7%.
    • loading

    Catalog

      Turn off MathJax
      Article Contents

      /

      DownLoad:  Full-Size Img  PowerPoint
      Return
      Return