分布式深度学习训练网络综述

朱泓睿; 元国军; 姚成吉; 谭光明; 王展; 户忠哲; 张晓扬; 安学军

doi:10.7544/issn1000-1239.2021.20190881

分布式深度学习训练网络综述

朱泓睿^1,2,
元国军¹,
姚成吉³,
谭光明¹,
王展¹,
户忠哲^1,2,3,
张晓扬^1,2,3,
安学军¹

¹(中国科学院计算技术研究所北京 100190)
²(中国科学院大学北京 100049)
³(北京旷视科技有限公司北京 100080) (zhuhongrui@ncic.ac.cn)

基金项目: 中国科学院战略性先导科技专项(B类) (XDB24050200)；国家自然科学基金面上项目(61972380，61702484)；中国科学院计算技术研究所创新课题(20166060)

详细信息

中图分类号: TP393
计量
- 文章访问数: 2259
- HTML全文浏览量: 22
- PDF下载量: 1807
出版历程
- 发布日期: 2020-12-31

Survey on Network of Distributed Deep Learning Training

¹(Institute of Computing Technology, Chinese Academy of Sciences, Beijing 100190)
²(University of Chinese Academy of Sciences, Beijing 100049)
³(Megvii Inc., Beijing 100080)

Funds: This work was supported by the CAS Strategic Priority Program(B) (XDB24050200), the General Program of the National Natural Science Foundation of China (61972380, 61702484), and the Innovation Fund from the Institute of Computing Technology, Chinese Academy of Sciences (20166060).

摘要

摘要: 近年来深度学习在图像、语音、自然语言处理等诸多领域得到广泛应用，但随着人们对深度学习的训练速度和数据处理能力的需求不断提升，传统的基于单机的训练过程愈发难以满足要求，分布式的深度学习训练方法成为持续提升算力的有效途径．其中训练过程中节点间网络的通信性能至关重要，直接影响训练性能.分析了分布式深度学习中的性能瓶颈，在此基础上对目前常用的网络性能优化方案进行综述，详细阐述了目前最新的超大规模分布式训练的体系结构、优化方法、训练环境和最有效的优化方法，最后对分布式训练仍然存在的困难进行了总结，对其未来研究方向进行了展望.
- 分布式计算 /
- 深度学习 /
- 通信网络 /
- 性能优化 /
- 集合通信 /
- 集群网络
Abstract: In recent years, deep learning has achieved better results than traditional algorithms in many fields such as image, speech, and natural language processing. People are increasingly demanding training speed and data processing capabilities for deep learning. However, the calculating ability of a single server has a limit and cannot achieve human demands. Distributed deep learning training has become the most effective method to expand deep learning training computing ability. At present, distributed deep learning faces a training bottleneck due to communication problems in the network during the training process which leads the communication network to be the most influential factor. There are currently many network performance optimization researches for distributed deep learning. In this paper, the main performance bottlenecks and optimization schemes are firstly demonstrated. Then the current state-of-art ultra-large-scale distributed training architecture and methods for optimization performance are specifically analyzed. Finally, a comparative summary of each performance optimization scheme and the difficulties still existing in distributed deep learning training are given, and the future research directions are pointed out as well.
- distributed calculating /
- deep learning /
- communication network /
- performance optimization /
- collective communication /
- cluster network

HTML全文

参考文献(0)

施引文献(10)

期刊类型引用(4)

1.	王海晓，冯星霖，郭敏. 改进H-GASA算法求解网约车拼车服务问题. 内蒙古农业大学学报(自然科学版). 2025(02): 53-62 . 百度学术
2.	张宇，郭仁拥. 基于即时共享率和预测需求密度的一对多车辆共乘匹配. 系统工程理论与实践. 2024(12): 3979-3996 . 百度学术
3.	汪厚俊，郑明飞. 基于改进Dijkstra算法的医院智慧停车路径规划研究. 微型电脑应用. 2022(05): 120-124 . 百度学术
4.	郭羽含，刘永武. 动态车辆共乘问题的双模式协作匹配算法. 计算机研究与发展. 2022(07): 1533-1552 . 本站查看