ISSN 1000-1239 CN 11-1777/TP

Journal of Computer Research and Development ›› 2021, Vol. 58 ›› Issue (12): 2585-2603.doi: 10.7544/issn1000-1239.2021.20211023

Special Issue: 2021可解释智能学习方法及其应用专题

Previous Articles     Next Articles

Spatio-Clock Synchronous Constraint Guided Safe Reinforcement Learning for Autonomous Driving

Wang Jinyong1,2, Huang Zhiqiu1,2, Yang Deyan3, Xiaowei Huang4, Zhu Yi3, Hua Gaoyang1,2   

  1. 1(College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics, Nanjing 211106);2(Key Laboratory of Safety-Critical Software (Nanjing University of Aeronautics and Astronautics), Ministry of Industry and Information Technology, Nanjing 211106);3(School of Computer Science and Technology, Jiangsu Normal University, Xuzhou, Jiangsu 221116);4(Department of Computer Science, University of Liverpool, Liverpool, UK L69 3BX)
  • Online:2021-12-01
  • Supported by: 
    This work was supported by the National Key Research and Development Program of China (2018YFB1003900) and the National Natural Science Foundation of China (61772270, 62077029).

Abstract: Autonomous driving systems integrate complex interactions between hardware and software. In order to ensure the safe and reliable operations, formal methods are used to provide rigorous guarantees to satisfy logical specifications and safety-critical requirements in the design stage. As a widely employed machine learning architecture, deep reinforcement learning (DRL) focuses on finding an optimal policy that maximizes a cumulative discounted reward by interacting with the environment, and has been applied to autonomous driving decision-making modules. However, black-box DRL-based autonomous driving systems cannot provide guarantees of safe operation and reward definition interpretability techniques for complex tasks, especially when they face unfamiliar situations and reason about a greater number of options. In order to address these problems, spatio-clock synchronous constraint is adopted to augment DRL safety and interpretability. Firstly, we propose a dedicated formal properties specification language for autonomous driving domain, i.e., spatio-clock synchronous constraint specification language, and present domain-specific knowledge requirements specification that is close to natural language to make the reward functions generation process more interpretable. Secondly, we present domain-specific spatio-clock synchronous automata to describe spatio-clock autonomous behaviors, i.e., controllers related to certain spatio- and clock-critical actions, and present safe state-action space transition systems to guarantee the safety of DRL optimal policy generation process. Thirdly, based on the formal specification and policy learning, we propose a formal spatio-clock synchronous constraint guided safe reinforcement learning method with the goal of easily understanding the safe reward function. Finally, we demonstrate the effectiveness of our proposed approach through an autonomous lane changing and overtaking case study in the highway scenario.

Key words: spatio-clock synchronous constraint, formal specification, safe reinforcement learning, temporal difference, intelligent traffic simulation, autonomous driving safety

CLC Number: