• 中国精品科技期刊
  • CCF推荐A类中文期刊
  • 计算领域高质量科技期刊T1类
Advanced Search
Zhong Shan, Liu Quan, Fu Qiming, Zhang Zongzhang, Zhu Fei, Gong Shengrong. A Heuristic Dyna Optimizing Algorithm Using Approximate Model Representation[J]. Journal of Computer Research and Development, 2015, 52(12): 2764-2775. DOI: 10.7544/issn1000-1239.2015.20148160
Citation: Zhong Shan, Liu Quan, Fu Qiming, Zhang Zongzhang, Zhu Fei, Gong Shengrong. A Heuristic Dyna Optimizing Algorithm Using Approximate Model Representation[J]. Journal of Computer Research and Development, 2015, 52(12): 2764-2775. DOI: 10.7544/issn1000-1239.2015.20148160

A Heuristic Dyna Optimizing Algorithm Using Approximate Model Representation

More Information
  • Published Date: November 30, 2015
  • In allusion to the problems of reinforcement learning with Dyna-framework, such as slow convergence and inappropriate representation of the environment model, delayed learning of the changed environment and so on, this paper proposes a novel heuristic Dyna optimization algorithm based on approximate model—HDyna-AMR, which approximates Q value function via linear function, and solves the optimal value function by using gradient descent method. HDyna-AMR can be divided into two phases, such as the learning phase and the planning phase. In the former one, the algorithm approximately models the environment by interacting with the environment and records the feature appearing frequency, while in the latter one, the approximated environment model can be used to do the planning with some extra rewards according to the feature appearing frequency. Additionally, the paper proves the convergence of the proposed algorithm theoretically. Experimentally, we apply HDyna-AMR to the extended Boyan Chain problem and Mountain Car problem, and the results show that HDyna-AMR can get the approximately optimal policy in both discrete and continuous state space. Furthermore, compared with Dyna-LAPS (Dyna-style planning with linear approximation and prioritized sweeping) and Sarsa(λ), HDyna-AMR outperforms Dyna-LAPS and Sarsa(λ) in terms of convergence rate, and the robustness to the changed environment.
  • Related Articles

    [1]Zou Shichen, Wang Huiqiang, Lü Hongwu, Feng Guangsheng, Lin Junyu. Service Substitution Method in Distributed Virtualized Environment Based on Transaction[J]. Journal of Computer Research and Development, 2018, 55(2): 377-390. DOI: 10.7544/issn1000-1239.2018.20160925
    [2]Shi Yuan, Zhang Huanguo, Wu Fusheng. A Method of Constructing the Model of Trusted Virtual Machine Migration[J]. Journal of Computer Research and Development, 2017, 54(10): 2284-2295. DOI: 10.7544/issn1000-1239.2017.20170465
    [3]Zhu Changpeng, Zhao Yinliang, Han Bo, Zeng Qinghua, and Liu Songjia. A VM-centric Approach for Dynamic Layer Binding[J]. Journal of Computer Research and Development, 2014, 51(2): 429-444.
    [4]Chen Hao, Peng Cuifen, Sun Jianhua, and Shi Lin. XenRPC:Design and Implementation of Security VM Remote Procedure Call[J]. Journal of Computer Research and Development, 2012, 49(5): 996-1004.
    [5]Zhang Xiang, Huo Zhigang, Ma Jie, Meng Dan. Fast and Live Whole-System Migration of Virtual Machines[J]. Journal of Computer Research and Development, 2012, 49(3): 661-668.
    [6]Wang Kai, Hou Zifeng. A Relaxed Co-Scheduling Method of Virtual CPUs on Xen Virtual Machines[J]. Journal of Computer Research and Development, 2012, 49(1): 118-127.
    [7]Wang Kai, Hou Zifeng. An Adaptive Scheduling Method of Weight Parameter Adjustment on Virtual Machines[J]. Journal of Computer Research and Development, 2011, 48(11): 2094-2102.
    [8]Wang Lina, Gao Hanjun, Liu Wei, Peng Yang. Detecting and Managing Hidden Process via Hypervisor[J]. Journal of Computer Research and Development, 2011, 48(8): 1534-1541.
    [9]Jin Hai, Zhong Alin, Wu Song, and Shi Xuanhua. Virtual Machine VCPU Scheduling in the Multi-core Environment:Issues and Challenges[J]. Journal of Computer Research and Development, 2011, 48(7): 1216-1224.
    [10]Xu Mingwei, Hu Chunming, Liu Xudong, and Ma Dianfu. Research and Implementation of Web Service Differentiated QoS[J]. Journal of Computer Research and Development, 2005, 42(4): 669-675.

Catalog

    Article views (1576) PDF downloads (554) Cited by()

    /

    DownLoad:  Full-Size Img  PowerPoint
    Return
    Return