ISSN 1000-1239 CN 11-1777/TP

计算机研究与发展 ›› 2019, Vol. 56 ›› Issue (10): 2135-2150.doi: 10.7544/issn1000-1239.2019.20190415

所属专题: 2019密码学与智能安全研究专题

• 综述 • 上一篇    下一篇



  1. 1(智能网络与网络安全教育部重点实验室(西安交通大学) 西安 710049);2(西安交通大学电子与信息学部 西安 710049);3(武汉大学网络安全学院 武汉 430072);4(清华大学网络科学与网络空间研究院 北京 100084);5(香港城市大学计算机科学系 香港 999077);6(浙江大学网络空间安全研究中心 杭州 310027);7(浙江大学计算机科学与技术学院 杭州 310027);8(乔治亚大学计算机科学系 乔治亚州雅典市 30602) (
  • 出版日期: 2019-10-16
  • 基金资助: 

Security and Privacy Risks in Artificial Intelligence Systems

Chen Yufei1,2, Shen Chao1,2, Wang Qian3, Li Qi4, Wang Cong5, Ji Shouling6,7, Li Kang8, Guan Xiaohong1,2   

  1. 1(Key Laboratory for Intelligent Networks and Network Security(Xi’an Jiaotong University), Ministry of Education, Xi’an 710049);2(Faculty of Electronic and Information Engineering, Xi’an Jiaotong University, Xi’an 710049);3(School of Cyber Science and Engineering, Wuhan University, Wuhan 430072);4(Institute for Network Sciences and Cyberspace, Tsinghua University, Beijing 100084);5(Department of Computer Science, City University of Hong Kong, Hong Kong 999077);6(Institute of Cyberspace Research, Zhejiang University, Hangzhou 310027);7(College of Computer Science and Technology, Zhejiang University, Hangzhou 310027);8(Department of Computer Science, University of Georgia, Athens, Georgia, the United States 30602)
  • Online: 2019-10-16

摘要: 人类正在经历着由深度学习技术推动的人工智能浪潮,它为人类生产和生活带来了巨大的技术革新.在某些特定领域中,人工智能已经表现出达到甚至超越人类的工作能力.然而,以往的机器学习理论大多没有考虑开放甚至对抗的系统运行环境,人工智能系统的安全和隐私问题正逐渐暴露出来.通过回顾人工智能系统安全方面的相关研究工作,揭示人工智能系统中潜藏的安全与隐私风险.首先介绍了包含攻击面、攻击能力和攻击目标的安全威胁模型.从人工智能系统的4个关键环节——数据输入(传感器)、数据预处理、机器学习模型和输出,分析了相应的安全隐私风险及对策.讨论了未来在人工智能系统安全研究方面的发展趋势.

关键词: 智能系统安全, 系统安全, 数据处理, 人工智能, 深度学习

Abstract: Human society is witnessing a wave of artificial intelligence (AI) driven by deep learning techniques, bringing a technological revolution for human production and life. In some specific fields, AI has achieved or even surpassed human-level performance. However, most previous machine learning theories have not considered the open and even adversarial environments, and the security and privacy issues are gradually rising. Besides of insecure code implementations, biased models, adversarial examples, sensor spoofing can also lead to security risks which are hard to be discovered by traditional security analysis tools. This paper reviews previous works on AI system security and privacy, revealing potential security and privacy risks. Firstly, we introduce a threat model of AI systems, including attack surfaces, attack capabilities and attack goals. Secondly, we analyze security risks and counter measures in terms of four critical components in AI systems: data input (sensor), data preprocessing, machine learning model and output. Finally, we discuss future research trends on the security of AI systems. The aim of this paper is to arise the attention of the computer security society and the AI society on security and privacy of AI systems, and so that they can work together to unlock AI’s potential to build a bright future.

Key words: intelligent system security, system security, data processing, artificial intelligence (AI), deep learning