ISSN 1000-1239 CN 11-1777/TP

Journal of Computer Research and Development ›› 2019, Vol. 56 ›› Issue (7): 1578-1586.doi: 10.7544/issn1000-1239.2019.20180797

Previous Articles    

Performance Optimization of Lustre File System Based on Reinforcement Learning

Zhang Wentao1,2, Wang Lu1, Cheng Yaodong1   

  1. 1(Computing Center, Institute of High Energy Physics, Chinese Academy of Sciences, Beijing 100049);2(University of Chinese Academy of Sciences, Beijing 100049)
  • Online:2019-07-01

Abstract: Computing of high energy physics is a typical data-intensive application. The throughput and response time of distributed storage system are key performance indicators, and they are often the targets of performance optimization. There are a large number of parameters that can be adjusted in a distributed storage system. The setting of these parameters has great influence on the performance of the system. At present, these parameters are either set with static values or automatically tuned by some heuristic rules defined by experienced administrators. Neither of the method is optimistic taking into account the diversity of data access patterns and hardware capabilities, and the difficulty of finding heuristic rules for hundreds of interacted parameters based on human experience. In fact, if the tuning engine is regarded as an agent and the storage system is regarded as the environment, the parameter adjustment problem of the storage system can be treated as a typical sequential decision problem. Therefore, based on data access characteristics of high energy physics calculation, we propose an automated parameter tuning method using the reinforcement learning. Experiments show that in the same test environment, using the default parameters of the Lustre file system as a baseline, this method can increase the throughput by about 30%.

Key words: distributed storage, reinforcement learning, performance tuning, deep learning, parameter adjustment

CLC Number: