ISSN 1000-1239 CN 11-1777/TP

• 人工智能 •

### Heavy-Ball型动量方法的最优个体收敛速率

1. 1(中国人民解放军陆军炮兵防空兵学院信息工程系 合肥 230031);2(中国人民解放军陆军工程大学指挥控制工程学院 南京 210007) (m13776555321@163.com)
• 出版日期: 2019-08-01
• 基金资助:
国家自然科学基金项目(61673394)

### Optimal Individual Convergence Rate of the Heavy-Ball-Based Momentum Methods

Cheng Yujia1, Tao Wei2, Liu Yuxiang1, Tao Qing1

1. 1(Department of Information Engineering, Army Academy of Artillery and Air Defense of PLA, Hefei 230031);2(College of Command and Control Engineering, Army Engineering University of PLA, Nanjing 210007)
• Online: 2019-08-01

Abstract: The momentum method is widely used as an acceleration technique to improve the convergence of the first-order gradient algorithms. So far, the momentum methods discussed in most literatures are only limited to the accelerated method proposed by Nesterov, but the Heavy-ball momentum method proposed by Polyak is seldom studied. In particular, in the case of non-smooth objective functions, the individual optimal convergence of Nesterov accelerated methods has been derived, and it has high performance in solving sparse optimization problems. In contrast, while it has been proved that the Heavy-ball momentum method has an optimal convergence rate,it is only in terms of the averaged outputs. To our best knowledge, whether it has optimal individual convergence or not still remains unknown. In this paper, we focus on the non-smooth optimizations. We prove that the Heavy-ball momentum method achieves the optimal individual convergence by skillfully selecting the time-varying step-size, which indicates that Heavy-ball momentum is an efficient acceleration strategy for the individual convergence of the projected subgradient methods. As an application, the constrained hinge loss function optimization problems within an l\-1-norm ball are considered. In comparison with other optimization algorithms, the experiments demonstrate the correctness of our theoretical analysis and performance of the proposed algorithms in keeping the sparsity.