Exploration Approaches in Deep Reinforcement Learning Based on Intrinsic Motivation: A Review

In recent years, deep reinforcement learning has made many important achievements in game artificial intelligence, robotics and other fields. However, in the realistic application scenarios with sparse rewards and random noises, such methods are suffering much from exploring the large state-action space. Introducing the notion of intrinsic motivation from psychology into deep reinforcement learning is an important idea to solve the above problem. Firstly, the connotation of the difficulty of exploration in deep reinforcement learning is explained, and three classical exploration methods are introduced, and their limitations in high-dimensional or continuous scenarios are discussed. Secondly, the background of the introduction of intrinsic motivation into deep reinforcement learning and the common testing environments of algorithms and models are described. On this basis, the basic principles, advantages and disadvantages of various exploration methods are analyzed in detail, including count-based, knowledge-based and competency-based approaches. Then, the applications of deep reinforcement learning based on intrinsic motivation in different fields are introduced. Finally, this paper throws light on the key problems that need to be solved for more advanced algorithms, such as the difficulty in constructing effective state representation, and also pinpoints some prospective research directions such as representation learning and knowledge accumulation. Hopefully, this review can provide readers with guidance of designing suitable intrinsic rewards for problems in hand and devising more effective exploration algorithms.