An Adaptive Tasks Scheduling Method Based on the Ability of Node in Hadoop Cluster

Scheduling algorithm has played an important role in improving Hadoop cluster performance. However, the phenomenon of uneven load distribution exists in current Hadoop inherent task-level scheduling methods. In order to maintain the load status of each node in the cluster, we focus on the analysis and research about the implementation capacity of cluster nodes. An adaptive tasks scheduling method based on the node capability is proposed in this paper. According to the node history and the current load status, the method takes node performance, task characteristics and node failure rate as the parameters to calculate node executive ability. On this basis, the different amount of tasks are assigned to cluster nodes. Thus, joined nodes adaptively adjust the amount of running tasks to make the node suited for different tasks better. Finally experimental results indicate that the proposed adaptive tasks scheduling method can make the total task completion time being reduced significantly, and moreover, the load on each node gets more balanced and the node resource utilization is more reasonable.