Optimization of Multi-Agent Handover Based on Team Model in C-V2X Environments

Liu Bingyi; Wang Dongdong; Shi Haiyong; Wang Enshu; Wu Libing; Wang Jianping

doi:10.7544/issn1000-1239.202440404

Journal of Computer Research and Development > 2024 > 61(11): 2806-2820. > DOI: 10.7544/issn1000-1239.202440404 CSTR: 32373.14.issn1000-1239.202440404

Liu Bingyi, Wang Dongdong, Shi Haiyong, Wang Enshu, Wu Libing, Wang Jianping. Optimization of Multi-Agent Handover Based on Team Model in C-V2X Environments[J]. Journal of Computer Research and Development, 2024, 61(11): 2806-2820. DOI: 10.7544/issn1000-1239.202440404

Citation:

PDF (1847 KB)

Optimization of Multi-Agent Handover Based on Team Model in C-V2X Environments

1.
School of Computer Science and Artificial Intelligence, Wuhan University of Technology, Wuhan 430070
2.
Sanya Science and Education Innovation Park, Wuhan University of Technology, Sanya, Hainan 572000
3.
School of Cyber Science and Engineering, Wuhan University, Wuhan 430070
4.
Department of Computer Science, City University of Hong Kong, Kowloon, Hong Kong

Funds: The work was supported by the National Natural Science Foundation of China (62272357, 62302326, 62202348, U20A20177), the Key Research and Development Program of Hubei province (2022BAA052), and the Project of Hong Kong Research Grant Council under NSFC/RGC (N_CityU140/20).

More Information

Author Bio:
Liu Bingyi: born in 1990. PhD, professor and PhD supervisor. His main research interests include wireless networking, vehicular networking, autonomous driving, and Internet of things

Wang Dongdong: born in 2000. Master candidate. His main research interests include vehicular networking, autonomous driving, and reinforcement learning

Shi Haiyong: born in 1998. PhD candidate. His main research interests include vehicular networking, autonomous driving, and reinforcement learning

Wang Enshu: born in 1990. PhD, associate professor. His main research interests include vehicular networking, autonomous driving, and reinforcement learning

Wu Libing: born in 1970. PhD, professor, PhD supervisor. Distinguished member of CCF. His main research interests include autonomous driving, distributed computing, network security, and wireless sensor networks

Wang Jianping: born in 1975. PhD, professor. IEEE Fellow and AAIA Fellow. Her main research interests include autonomous driving, reliable Internet, cloud computing, optical networks, and service-oriented networks and data center networking
Received Date: May 30, 2024
Revised Date: July 30, 2024
Available Online: August 13, 2024

Graphical Abstract

Abstract

Abstract

Cellular vehicle-to-everything (C-V2X) communication technology is a crucial component of future intelligent transportation systems (ITS). Millimeter wave (mmWave), as one of the primary carriers for C-V2X technology, offers high bandwidth to users. However, due to limited propagation distance and sensitivity to obstructions, mmWave base stations must be densely deployed to maintain reliable communication. This requirement causes intelligent connected vehicle (ICV) to frequently switch communications during travel, easily leading to local resource shortages, thus degrading service quality and user experience. To address these challenges, we treat each ICV as an agent and model the ICV communication switching issue as a cooperative multi-agent game problem. To solve this problem, we propose a cooperative reinforcement learning framework based on a teammate model. Specifically, we design a teammate model to quantify the interdependencies among agents in complex dynamic environments. Furthermore, we propose a dynamic weight allocation scheme that generates weighted mutual information among teammates for the input of the mixing network, aiming to assist teammates in switching to base stations that provide satisfactory QoS and QoE, thereby achieving high throughput and low communication switching frequency. During the algorithm training process, we design an incentive-compatible training algorithm aimed at aligning the individual goals of the agents with collective goals, enhancing communication throughput. Experimental results demonstrate that this algorithm achieves a 13.8% to 38.2% increase in throughput compared with existing communication benchmark algorithms.
- C-V2X,
- resource allocation,
- handover,
- multi-agent reinforcement learning,
- cooperative multi-agent reinforcement learning

FullText(HTML)

References (33)

References

[1]	Olwal T O, Djouani K, Kurien A M. A survey of resource management toward 5G radio access networks[J]. IEEE Communications Surveys & Tutorials, 2016, 18(3): 1656−1686
[2]	齐彦丽,周一青,刘玲,等. 融合移动边缘计算的未来5G移动通信网络[J]. 计算机研究与发展,2018,55(3):478−486 Qi Yanli, Zhou Yiqing, Liu Ling, et al. MEC coordinated future 5G mobile wireless networks[J]. Journal of Computer Research and Development, 2018, 55(3): 478−486 (in Chinese)
[3]	周玉轩,杨絮,秦传义,等. HDM网络架构与混合式数据分发策略[J]. 计算机研究与发展,2020,57(9):1911−1927 Zhou Yuxuan, Yang Xu, Qin Chuanyi, et al. HYBRID-D2D-MIMO (HDM) network architecture and hybrid data distributing strategy (HDDS)[J]. Journal of Computer Research and Development, 2020, 57(9): 1911−1927 (in Chinese)
[4]	Liu Ling, Zhou Yiqing, Zhuang Weihua, et al. Tractable coverage analysis for hexagonal macrocell-based heterogeneous UDNs with adaptive interference-aware CoMP[J]. IEEE Transactions on Wireless Communications, 2018, 18(1): 503−517
[5]	Liu Bingyi, Han Weizhen, Wang Enshu, et al. Multi-agent attention double actor-critic framework for intelligent traffic light control in urban scenarios with hybrid traffic[J]. IEEE Transactions on Mobile Computing, 2023, 23(1): 660−672
[6]	Wang Enshu, Liu Bingyi, Lin Songrong, et al. Double graph attention actor-critic framework for urban bus-pooling system[J]. IEEE Transactions on Intelligent Transportation Systems, 2023, 24(5): 5313−5325 doi: 10.1109/TITS.2023.3238055
[7]	Abuajwa O, Roslee M B, Yusoff Z B. Simulated annealing for resource allocation in downlink NOMA systems in 5G networks[J]. Applied Sciences, 2021, 11(10): 4592 doi: 10.3390/app11104592
[8]	Ibrahim L F, Salman H A, Taha Z F, et al. A survey on heterogeneous mobile networks planning in indoor dense areas[J]. Personal and Ubiquitous Computing, 2020, 24: 487−498 doi: 10.1007/s00779-019-01243-y
[9]	吕华章,陈丹,范斌,等. 边缘计算标准化进展与案例分析[J]. 计算机研究与发展,2018,55(3):487−511 Lü Huazhang, Chen Dan, Fan Bin, et al. Standardization progress and case analysis of edge computing[J]. Journal of Computer Research and Development, 2018, 55(3): 487−511 (in Chinese)
[10]	Rappaport T S, Xing Y, MacCartney G R, et al. Overview of millimeter wave communications for fifth-generation (5G) wireless networks—With a focus on propagation models[J]. IEEE Transactions on Antennas and Propagation, 2017, 65(12): 6213−6230 doi: 10.1109/TAP.2017.2734243
[11]	Jameel F, Haider M A A, Butt A A. Massive MIMO: A survey of recent advances, research issues and future directions[C]//Proc of 2017 Int Symp on Recent Advances in Electrical Engineering (RAEE). Piscataway, NJ: IEEE, 2017: 1−6
[12]	Stojmenovic I. Handbook of Wireless Networks and Mobile Computing[M]. Hoboken, NJ: John Wiley & Sons, 2002
[13]	Kim Y, Kim Y, Oh J, et al. New radio (NR) and its evolution toward 5G-advanced[J]. IEEE Wireless Communications, 2019, 26(3): 2−7 doi: 10.1109/MWC.2019.8752473
[14]	Marquez-Barja J M, Ahmadi H, Tornell S M, et al. Breaking the vehicular wireless communications barriers: Vertical handover techniques for heterogeneous networks[J]. IEEE Transactions on vehicular Technology, 2014, 64(12): 5878−5890
[15]	Isa I N M, Baba M D, Yusof A L, et al. Handover parameter optimization for self-organizing LTE networks[C]//Proc of 2015 IEEE Symp on Computer Applications & Industrial Electronics (ISCAIE). Piscataway, NJ: IEEE, 2015: 1−6
[16]	Alhabo M, Zhang Li, Nawaz N. GRA-based handover for dense small cells heterogeneous networks[J]. IET Communications, 2019, 13((13): ): 1928−1935 doi: 10.1049/iet-com.2018.5938
[17]	Hasan M M, Kwon S, Oh S. Frequent-handover mitigation in ultra-dense heterogeneous networks[J]. IEEE Transactions on Vehicular Technology, 2018, 68(1): 1035−1040
[18]	Yang Bing, Yang Xuan, Ge Xiaohu, et al. Coverage and handover analysis of ultra-dense millimeter-wave networks with control and user plane separation architecture[J]. IEEE Access, 2018, 6: 54739−54750 doi: 10.1109/ACCESS.2018.2871363
[19]	Simsek M, Bennis M, Güvenc I. Context-aware mobility management in HetNets: A reinforcement learning approach[C]//Proc of 2015 IEEE Wireless Communications and Networking Conf (WCNC). Piscataway, NJ: IEEE, 2015: 1536−1541
[20]	Qi Weijing, Song Qingyang, Wang Shupeng, et al. Social prediction-based handover in collaborative-edge-computing-enabled vehicular networks[J]. IEEE Transactions on Computational Social Systems, 2021, 9(1): 207−217
[21]	Luong N C, Hoang D T, Gong S, et al. Applications of deep reinforcement learning in communications and networking: A survey[J]. IEEE Communications Surveys & Tutorials, 2019, 21(4): 3133−3174
[22]	Wang Zhi, Li Lihua, Xu Yue, et al. Handover control in wireless systems via asynchronous multiuser deep reinforcement learning[J]. IEEE Internet of Things Journal, 2018, 5(6): 4296−4307 doi: 10.1109/JIOT.2018.2848295
[23]	Nasim I, Skrimponis P, Ibrahim A S, et al. Reinforcement learning of millimeter wave beamforming tracking over COSMOS platform[C]//Proc of the 16th ACM Workshop on Wireless Network Testbeds, Experimental Evaluation & Characterization. New York: ACM, 2022: 40−44
[24]	Wei Yao, Lung C H, Ajila S, et al. Deep Q-networks assisted pre-connect handover management for 5G networks[C]//Proc of 2023 IEEE 97th Vehicular Technology Conf (VTC2023-Spring). Piscataway, NJ: IEEE, 2023: 1−6
[25]	Lin Yan, Zhang Zhengming, Huang Yongming, et al. Heterogeneous user-centric cluster migration improves the connectivity-handover trade-off in vehicular networks[J]. IEEE Transactions on Vehicular Technology, 2020, 69(12): 16027−16043 doi: 10.1109/TVT.2020.3041521
[26]	Lee C, Jung J, Chung J M. Intelligent dual active protocol stack handover based on double DQN deep reinforcement learning for 5G mmWave networks[J]. IEEE Transactions on Vehicular Technology, 2022, 71(7): 7572−7584 doi: 10.1109/TVT.2022.3170420
[27]	Wang Ruiyu, Sun Yao, Zhang Chao, et al. A novel handover scheme for millimeter wave network: An approach of integrating reinforcement learning and optimization[J/OL]. Digital Communications and Networks, 2023[2024-03-05]. https://www.sciencedirect.com/science/article/pii/S2352864823001360
[28]	Guo Delin, Tang Lan, Zhang Xinggan, et al. Joint optimization of handover control and power allocation based on multi-agent deep reinforcement learning[J]. IEEE Transactions on Vehicular Technology, 2020, 69(11): 13124−13138 doi: 10.1109/TVT.2020.3020400
[29]	Rashid T, Samvelyan M, de Witt C S, et al. Monotonic value function factorisation for deep multi-agent reinforcement learning[J]. Journal of Machine Learning Research, 2020, 21(178): 1−51
[30]	Wang Jianhao, Ren Zhizhou, Liu Terry, et al. QPLEX: Duplex dueling multi-agent Q-learning[J]. arXiv preprint, arXiv: 2008. 01062, 2020
[31]	Wang Tonghan, Wang Jianhao, Zheng Chongyi, et al. Learning nearly decomposable value functions via communication minimization[J]. arXiv preprint, arXiv: 1910. 05366, 2019
[32]	Yu Chao, Velu A, Vinitsky E, et al. The surprising effectiveness of PPO in cooperative multi-agent games[J]. Advances in Neural Information Processing Systems, 2022, 35: 24611−24624
[33]	Yuan Lei, Wang Jianhao, Zhang Fuxiang, et al. Multi-agent incentive communication via decentralized teammate modeling[C]//Proc of the AAAI Conf on Artificial Intelligence. Menlo Park, CA: AAAI, 2022, 36(9): 9466−9474