Survey on Network of Distributed Deep Learning Training
-
摘要: 近年来深度学习在图像、语音、自然语言处理等诸多领域得到广泛应用,但随着人们对深度学习的训练速度和数据处理能力的需求不断提升,传统的基于单机的训练过程愈发难以满足要求,分布式的深度学习训练方法成为持续提升算力的有效途径.其中训练过程中节点间网络的通信性能至关重要,直接影响训练性能.分析了分布式深度学习中的性能瓶颈,在此基础上对目前常用的网络性能优化方案进行综述,详细阐述了目前最新的超大规模分布式训练的体系结构、优化方法、训练环境和最有效的优化方法,最后对分布式训练仍然存在的困难进行了总结,对其未来研究方向进行了展望.Abstract: In recent years, deep learning has achieved better results than traditional algorithms in many fields such as image, speech, and natural language processing. People are increasingly demanding training speed and data processing capabilities for deep learning. However, the calculating ability of a single server has a limit and cannot achieve human demands. Distributed deep learning training has become the most effective method to expand deep learning training computing ability. At present, distributed deep learning faces a training bottleneck due to communication problems in the network during the training process which leads the communication network to be the most influential factor. There are currently many network performance optimization researches for distributed deep learning. In this paper, the main performance bottlenecks and optimization schemes are firstly demonstrated. Then the current state-of-art ultra-large-scale distributed training architecture and methods for optimization performance are specifically analyzed. Finally, a comparative summary of each performance optimization scheme and the difficulties still existing in distributed deep learning training are given, and the future research directions are pointed out as well.
-
-
期刊类型引用(7)
1. 李曼文,张月琴,张晨威,张泽华. 异质图嵌入的地理不敏感时空兴趣点推荐方法. 计算机科学与探索. 2024(03): 755-767 . 百度学术
2. 金柯君,于洪涛,吴翼腾,李邵梅,张建朋,郑洪浩. 改进的基于奇异值分解的图卷积网络防御方法. 计算机应用. 2023(05): 1511-1517 . 百度学术
3. 王小红,刘琴. 基于深度迁移的有向加权网络节点重叠检测. 计算机仿真. 2023(09): 492-496 . 百度学术
4. 金柯君,于洪涛,李邵梅,张建朋. 基于注意力机制的图卷积网络防御方法. 信息工程大学学报. 2023(06): 718-724 . 百度学术
5. 杨旭华,王磊,叶蕾,张端,周艳波,龙海霞. 基于节点相似性和网络嵌入的复杂网络社区发现算法. 计算机科学. 2022(03): 121-128 . 百度学术
6. 刘志鑫,张泽华,张杰. 基于多层次多视角的图注意力Top-N推荐方法. 计算机科学. 2021(04): 104-110 . 百度学术
7. 陈晋音,黄国瀚,张敦杰,张旭鸿,纪守领. 一种面向图神经网络的图重构防御方法. 计算机研究与发展. 2021(05): 1075-1091 . 本站查看
其他类型引用(4)
计量
- 文章访问数: 2264
- HTML全文浏览量: 22
- PDF下载量: 1808
- 被引次数: 11