ISSN 1000-1239 CN 11-1777/TP

计算机研究与发展 ›› 2018, Vol. 55 ›› Issue (9): 1871-1888.doi: 10.7544/issn1000-1239.2018.20180129

所属专题: 2018优青专题

• 综述 • 上一篇    下一篇



  1. 1(厦门大学信息科学与技术学院 福建厦门 361005); 2(福建省智慧城市感知与计算重点实验室(厦门大学) 福建厦门 361005); 3(上海腾讯科技有限公司优图实验室 上海 200233) (
  • 出版日期: 2018-09-01
  • 基金资助: 
    国家重点研发计划项目(2017YFC0113000,2016YFB10015032);国家自然科学基金项目(U1705262,61772443,61402388,61572410);国家自然科学基金优秀青年科学基金项目(61422210);福建省自然科学基金项目(2017J01125) This work was supported by the National Key Research and Development Program (2017YFC0113000, 2016YFB1001503), the National Natural Science Foundation of China (U1705262, 61772443, 61402388, 61572410), the National Natural Science Foundation of China for Excellent Young Scientists (61422210),and the Natural Science Foundation of Fujian Province of China (2017J01125).

Deep Neural Network Compression and Acceleration: A Review

Ji Rongrong1,2,Lin Shaohui1,2,Chao Fei1,Wu Yongjian3,Huang Feiyue3   

  1. 1(School of Information Science and Engineering, Xiamen University, Xiamen, Fujian, 361005); 2(Fujian Key Laboratory of Sensing and Computing for Smart City (Xiamen University), Xiamen, Fujian, 361005); 3(BestImage Laboratory, Tencent Technology (Shanghai) Co.,Ltd, Shanghai, 200233)
  • Online: 2018-09-01

摘要: 深度神经网络在人工智能的应用中,包括计算机视觉、语音识别、自然语言处理方面,取得了巨大成功.但这些深度神经网络需要巨大的计算开销和内存存储,阻碍了在资源有限环境下的使用,如移动或嵌入式设备端.为解决此问题,在近年来产生大量关于深度神经网络压缩与加速的研究工作.对现有代表性的深度神经网络压缩与加速方法进行回顾与总结,这些方法包括了参数剪枝、参数共享、低秩分解、紧性滤波设计及知识蒸馏.具体地,将概述一些经典深度神经网络模型,详细描述深度神经网络压缩与加速方法,并强调这些方法的特性及优缺点.此外,总结了深度神经网络压缩与加速的评测方法及广泛使用的数据集,同时讨论分析一些代表性方法的性能表现.最后,根据不同任务的需要,讨论了如何选择不同的压缩与加速方法,并对压缩与加速方法未来发展趋势进行展望.

关键词: 深度神经网络压缩, 深度神经网络加速, 参数剪枝, 参数共享, 低秩分解, 知识蒸馏

Abstract: In recent years, deep neural networks (DNNs) have achieved remarkable success in many artificial intelligence (AI) applications, including computer vision, speech recognition and natural language processing. However, such DNNs have been accompanied by significant increase in computational costs and storage services, which prohibits the usages of DNNs on resource-limited environments such as mobile or embedded devices. To this end, the studies of DNN compression and acceleration have recently become more emerging. In this paper, we provide a review on the existing representative DNN compression and acceleration methods, including parameter pruning, parameter sharing, low-rank decomposition, compact filter designed, and knowledge distillation. Specifically, this paper provides an overview of DNNs, describes the details of different DNN compression and acceleration methods, and highlights the properties, advantages and drawbacks. Furthermore, we summarize the evaluation criteria and datasets widely used in DNN compression and acceleration, and also discuss the performance of the representative methods. In the end, we discuss how to choose different compression and acceleration methods to meet the needs of different tasks, and envision future directions on this topic.

Key words: DNN compression, DNN acceleration, parameter pruning, parameter sharing, low-rank decomposition, knowledge distillation