Wireless Resource Allocation Algorithm Based on Multi-Objective Deep Reinforcement Learning for Vehicle-to-Vehicle Communications

Li Ke; Ma Sai; Dai Penglin; Ren Jing; Fan Pingzhi

doi:10.7544/issn1000-1239.202330895

Journal of Computer Research and Development > 2024 > 61(9): 2229-2245. > DOI: 10.7544/issn1000-1239.202330895 CSTR: 32373.14.issn1000-1239.202330895

Li Ke, Ma Sai, Dai Penglin, Ren Jing, Fan Pingzhi. Wireless Resource Allocation Algorithm Based on Multi-Objective Deep Reinforcement Learning for Vehicle-to-Vehicle Communications[J]. Journal of Computer Research and Development, 2024, 61(9): 2229-2245. DOI: 10.7544/issn1000-1239.202330895

Citation:

PDF (2498 KB)

Wireless Resource Allocation Algorithm Based on Multi-Objective Deep Reinforcement Learning for Vehicle-to-Vehicle Communications

1.
School of Computing and Artificial Intelligence, Southwest Jiaotong University, Chengdu 611756
2.
School of Information and Communication Engineering, University of Electronic Science and Technology of China, Chengdu 611731
3.
School of Information Science and Technology, Southwest Jiaotong University, Chengdu 611756

Funds: This work was supported by the National Key Research and Development Program of China (2020YFB1807800), the National Natural Science Foundation of China (62202392, 62172342, U20A20156), the Project of Network and Data Security Key Laboratory in Sichuan Province (NDS2022-1), the Natural Science Foundation of Sichuan Province (2023NSFSC0459, 2022NSFSC0944), and the Excellent Youth Fund of the Natural Science Foundation of Hebei Province (F2022105003).

More Information

Author Bio:
Li Ke: born in 1983. PhD, master supervisor. Member of CCF. Her main research interests include machine learning, distributed system, and Internet of vehicles

Ma Sai: born in 2000. Master candidate. His main research interests include machine learning and Internet of vehicles

Dai Penglin: born in 1990. PhD, associate professor. Member of CCF. His main research interests include Internet of vehicles, edge computing, and intelligent transportation systems

Ren Jing: born in 1982. PhD, assistant researcher. Her main research interests include network architecture, protocol design, network modeling and optimization, and network security

Fan Pingzhi: born in 1955. PhD, professor. IEEE fellow. His main research interests include high mobility wireless communications, massive random-access techniques, and signal design and coding
Received Date: October 31, 2023
Revised Date: May 15, 2024
Available Online: June 03, 2024

Graphical Abstract

Abstract

Abstract

Due to the dynamic uncertainty, diversified service types and scarcity of wireless communication resources in the context of vehicle-to-everything, we explore the challenge of ensuring the requirement for multiple quality of service and the effective utilization of wireless resources in the scenario where V2N (vehicle-to-network) and V2V (vehicle-to-vehicle) links coexist and share spectrum in C-V2X (cellular vehicle-to-everything) networks. First, a multi-objective optimization problem is presented to model the decision-making process of channel selection and power control in C-V2X. The problem considers the impact of dynamic changes in the network environment, aiming to make a balance between the performance of the V2V link (i.e., age of information, delay, and capacity) and the capacity of the V2N link. On this basis, V2V wireless resource allocation algorithm based on multi-objective deep reinforcement learning is also proposed for training neural networks to solve the problem. Through the trained neural network model, the Pareto frontier of the multi-objective optimization problem can be obtained. Simulation results demonstrate that the proposed algorithm can achieve the near-optimal performance for different communication links. Compared with four representative algorithms, the age of information for V2V link is reduced by 12.0% to 17.2%, the V2N link capacity is increased by 11.4% to 21.6%, the V2V link transmission success rate is increased by 4.6% to 13.9%, and the decision delay time is reduced by 10.6% to 20.3%.

FullText(HTML)

References (36)

References

[1]	Noor-A-Rahim M, Liu Zilong, Lee H, et al. A survey on resource allocation in vehicular networks[J]. IEEE Transactions on Intelligent Transportation Systems, 2020, 23(2): 701−721
[2]	况博裕,李雨泽,顾芳铭,等. 车联网安全研究综述:威胁、对策与未来展望[J]. 计算机研究与发展,2023,60(10):2304−2321 doi: 10.7544/issn1000-1239.202330464 Kuang Boyu, Li Yuze, Gu Fangming, et al. A review of security research in telematics: Threats, countermeasures and future outlook[J]. Journal of Computer Research and Development, 2023, 60(10): 2304−2321(in Chinese) doi: 10.7544/issn1000-1239.202330464
[3]	Yan Guozhi, Liu Kai, Liu Chunhui, et al. Edge intelligence for Internet of vehicles: A survey[J/OL]. IEEE Transactions on Consumer Electronics, 2024[2024-03-25].https://ieeexplore.ieee.org/abstract/document/10474509
[4]	Liu Bingyi, Jia Dongyao, Lu Kejie, et al. Infrastructure-assisted message dissemination for supporting heterogeneous driving patterns[J]. IEEE Transactions on Intelligent Transportation Systems, 2017, 18(10): 2865−2876 doi: 10.1109/TITS.2017.2661962
[5]	Guo Chongtao, Wang Xijun, Liang Le, et al. Age of information, latency, and reliability in intelligent vehicular networks[J]. IEEE Network, 2023, 37(6): 109−116 doi: 10.1109/MNET.124.2200132
[6]	Gyawali S, Xu Shengjie, Qian Yi, et al. Challenges and solutions for cellular based V2X communications[J]. IEEE Communications Surveys & Tutorials, 2020, 23(1): 222−255
[7]	李方伟,张海波,王子心. 车联网中基于MEC的V2X协同缓存和资源分配[J]. 通信学报,2021,42(2):26−36 doi: 10.11959/j.issn.1000-436x.2021007 Li Fangwei, Zhang Haibo, Wang Zixin. MEC-based V2X cooperative caching and resource allocation in vehicular networking[J]. Journal on Communications, 2021, 42(2): 26−36 (in Chinese) doi: 10.11959/j.issn.1000-436x.2021007
[8]	Liu Zongkai, Dai Penglin, Xing Huanlai, et al. A distributed algorithm for task offloading in vehicular networks with hybrid fog/cloud computing[J]. IEEE Transactions on Systems, Man, and Cybernetics: Systems, 2022, 52(7): 4388−4401 doi: 10.1109/TSMC.2021.3097005
[9]	Kaul S, Yates R, Gruteser M. Real-time status: How often should one update[C]//Proc of IEEE INFOCOM. Piscataway, NJ: IEEE, 2012: 2731−2735
[10]	Cho J H, Wang Yating, Chen I R, et al. A survey on modeling and optimizing multi-objective systems[J]. IEEE Communications Surveys and Tutorials, 2017, 19(3): 1867−1901 doi: 10.1109/COMST.2017.2698366
[11]	Guo Chongtao, Liang Le, Li G Y. Resource allocation for vehicular communications with low latency and high reliability[J]. IEEE Transactions on Wireless Communications, 2019, 18(8): 3887−3902 doi: 10.1109/TWC.2019.2919280
[12]	Nguyen B L, Ngo D T, Dao M N, et al. Scheduling and power control for connectivity enhancement in multi-hop I2V/V2V networks[J]. IEEE Transactions on Intelligent Transporation Systems, 2022, 23(8): 10322−10332 doi: 10.1109/TITS.2021.3091130
[13]	Yang Haojun, Zheng Kan, Zhao Long, et al. Twin-timescale radio resource management for ultra-reliable and low-latency vehicular networks[J]. IEEE Transactions on Vehicular Technology, 2020, 69(1): 1023−1036 doi: 10.1109/TVT.2019.2954462
[14]	Zhou Jianshan, Tian Daxin, Wang Yunpeng, et al. Reliability-optimal cooperative communication and computing in connected vehicle systems[J]. IEEE Transactions on Mobile Computing, 2020, 19(5): 1216−1232 doi: 10.1109/TMC.2019.2907491
[15]	Qin Xiaoqi, Xia Yangyang, Li Hang, et al. Distributed data collection in age-aware vehicular participatory sensing networks[J]. IEEE Internet Things Journal, 2021, 8(19): 14501−14513 doi: 10.1109/JIOT.2021.3049999
[16]	Li Zipeng, Xiang Lin, Ge Xiaohu, et al. Age of information modeling and optimization for fast information dissemination in vehicular social networks[J]. IEEE Transactions on Vehicular Technology, 2022, 71(5): 5445−5459 doi: 10.1109/TVT.2022.3154766
[17]	Abdel-Aziz M K, Samarakoon S, Liu Chenfeng, et al. Optimized age of information tail for ultra-reliable low-latency communications in vehicular networks[J]. IEEE Transactions on Communications, 2020, 68(3): 1911−1924 doi: 10.1109/TCOMM.2019.2961083
[18]	Khan W U, Jamshed M A, Lagunas E, et al. Energy efficiency optimization for backscatter enhanced NOMA cooperative V2X communications under imperfect CSI[J]. IEEE Transactions on Intelligent Transportation Systems, 2023, 24(11): 12961−12972 doi: 10.1109/TITS.2022.3187567
[19]	Wu Weihua, Liu Runzi, Yang Qinghai, et al. Learning-based robust resource allocation for ultra-reliable V2X communications[J]. Transactions on Wireless Communications, 2021, 20(8): 5199−5211 doi: 10.1109/TWC.2021.3065996
[20]	Wu Weihua, Liu Runzi, Yang Qinghai, et al. Robust resource allocation for vehicular communications with imperfect CSI[J]. IEEE Transactions on Wireless Communications, 2021, 20(9): 5883−5897 doi: 10.1109/TWC.2021.3070894
[21]	Peng Nuoheng, Lin Yan, Zhang Yijin, et al. AoI-aware joint spectrum and power allocation for Internet of vehicles: A trust region policy optimization-based approach[J]. IEEE Internet Things Journal, 2022, 9(20): 19916−19927 doi: 10.1109/JIOT.2022.3172472
[22]	Parvini M, Javan M R, Mokari N, et al. AoI aware radio resource management of autonomous platoons via multi agent reinforcement learning[C/OL]//Proc of the 17th Int Symp on Wireless Communication Systems. Piscataway, NJ: IEEE, 2021[2023-09-16].https://ieeexplore.ieee.org/document/9562190
[23]	Zhang Xinran, Peng Mugen, Yan Shi, et al. Deep reinforcement learning based mode selection and resource allocation for cellular V2X communications[J]. IEEE Internet Things Journal, 2020, 7(7): 6380−6391 doi: 10.1109/JIOT.2019.2962715
[24]	Ye Hao, Li G Y, Juang B H F. Deep reinforcement learning based resource allocation for V2V communications[J]. IEEE Transactions on Vehicular Technology, 2019, 68(4): 3163−3173 doi: 10.1109/TVT.2019.2897134
[25]	Han Yan, Tao Xiaofeng, Zhang Xuefei, et al. Delay-aware resource management for multi-service coexisting LTE-D2D networks with wireless network virtualization[J]. IEEE Transactions on Vehicular Technology, 2020, 69(7): 7339−7353 doi: 10.1109/TVT.2020.2990402
[26]	Song Fuhong, Xing Huanlai, Luo Shouxi, et al. A multiobjective computation offloading algorithm for mobile-edge computing[J]. IEEE Internet of Things Journal, 2020, 7(9): 8780−8799 doi: 10.1109/JIOT.2020.2996762
[27]	Ma Mengyu, Wang Chao, Li Zuxing, et al. A multi-objective optimization-based transmission design for V2V communication networks[J]. IEEE Transactions on Vehicular Technology, 2023, 72(10): 13081−13093 doi: 10.1109/TVT.2023.3274780
[28]	Schulman J, Wolski F, Dhariwal P, et al. Proximal policy optimization algorithms[J]. arXiv preprint, arXiv: 1707.06347, 2017
[29]	Woo S, Park J, Lee J Y, et al. CBAM: Convolutional block attention module[C]//Proc of the European Conf on Computer Vision. Berlin: Springer, 2018: 3−19
[30]	Liu Bingyi, Han Weizhen, Wang Enshu, et al. Multi-agent attention double actor-critic framework for intelligent traffic light control in urban scenarios with hybrid traffic[J]. IEEE Transactions on Mobile Computing, 2024, 23(1): 660−672 doi: 10.1109/TMC.2022.3233879
[31]	Bodnar C, Day B, Lió P. Proximal distilled evolutionary reinforcement learning[C]//Proc of the 34th AAAI Conf on Artificial Intelligence. Palo Alt, CA: AAAI, 2020: 3283−3290
[32]	Deb K, Pratap A, Agarwal S, et al. A fast and elitist multiobjective genetic algorithm: NSGA-Ⅱ[J]. IEEE Transactions on Evolutionary Computation, 2002, 6(2): 182−197 doi: 10.1109/4235.996017
[33]	Lehman J, Chen J, Clune J, et al. Safe mutations for deep and recurrent neural networks through output gradients[C]//Proc of the Genetic and Evolutionary Computation Conf. New York: ACM, 2018: 117−124
[34]	Merias P. Study on LTE-based V2X services (release 14), TR 36.885 V14.0. 0 [R]. Sophia-Antipolis, France: 3GPP, 2016
[35]	Ashraf M I, Bennis M, Perfecto C, et al. Dynamic proximity-aware resource allocation in vehicle-to-vehicle (V2V) communications[C/OL]//Proc of 2016 IEEE Globecom Workshops (GC Wkshps). Piscataway, NJ: IEEE, 2016[2023-09-21].https://ieeexplore.ieee.org/abstract/document/7848885
[36]	Lin Kai, Li Chensi, Li Yihui, et al. Distributed learning for vehicle routing decision in software defined Internet of vehicles[J]. IEEE Transactions on Intelligent Transportation Systems, 2021, 22(6): 3730−3741

[1]	Li Gengsong, Liu Yi, Zheng Qibin, Li Xiang, Liu Kun, Qin Wei, Wang Qiang, Yang Changhong. Algorithm Selection Method Based on Multi-Objective Hybrid Ant Lion Optimizer[J]. Journal of Computer Research and Development, 2023, 60(7): 1533-1550. DOI: 10.7544/issn1000-1239.202220769
[2]	Sun Penghao, Lan Julong, Shen Juan, Hu Yuxiang. Pinning Control-Based Routing Policy Generation Using Deep Reinforcement Learning[J]. Journal of Computer Research and Development, 2021, 58(7): 1563-1572. DOI: 10.7544/issn1000-1239.2021.20200018
[3]	Hu Haiyang, Liu Runhua, Hu Hua. Multi-Objective Optimization for Task Scheduling in Mobile Cloud Computing[J]. Journal of Computer Research and Development, 2017, 54(9): 1909-1919. DOI: 10.7544/issn1000-1239.2017.20160757
[4]	Li Li, Wang Wanliang, Xu Xinli, Li Weikun. Multi-Objective Particle Swarm Optimization Based on Grid Ranking[J]. Journal of Computer Research and Development, 2017, 54(5): 1012-1023. DOI: 10.7544/issn1000-1239.2017.20160074
[5]	Bi Xiaojun, Zhang Lei, Xiao Jing. Constrained Multi-Objective Optimization Algorithm Based on Dual Populations[J]. Journal of Computer Research and Development, 2015, 52(12): 2813-2823. DOI: 10.7544/issn1000-1239.2015.20148025
[6]	Zhang Shiwen, Li Zhiyong, Chen Shaomiao, and Li Renfa. Dynamic Multi-Objective Optimization Algorithm Based on Ecological Strategy[J]. Journal of Computer Research and Development, 2014, 51(6): 1313-1330.
[7]	Wen Renqiang, Zhong Shaobo, Yuan Hongyong, Huang Quanyi. Emergency Resource Multi-Objective Optimization Scheduling Model and Multi-Colony Ant Optimization Algorithm[J]. Journal of Computer Research and Development, 2013, 50(7): 1464-1472.
[8]	Liu Chun'an, Wang Yuping. Dynamic Multi-Objective Optimization Evolutionary Algorithm Based on New Model[J]. Journal of Computer Research and Development, 2008, 45(4): 603-611.
[9]	Chang Yan, Liu Xu, Cheng Wenyuan, Xie Xianghui, Cui Degang. Research and Application of Multi-Objective Aircraft Optimization System Based on Grid[J]. Journal of Computer Research and Development, 2007, 44(1): 44-60.
[10]	Ma Ming, Zhou Chunguang, Zhang Libiao, Ma Jie. Fuzzy Neural Network Optimization by a Multi-Objective Particle Swarm Optimization Algorithm[J]. Journal of Computer Research and Development, 2006, 43(12): 2104-2109.

Cited By

Cited by

Periodical cited type(16)

1.	戎珂，施新伟，吕若明. “i7算”赋能AI产业生态可持续发展. 科学学研究. 2025(01): 197-204 .
2.	张浩严，吕文涛，余润泽，邓志江. 大语言模型研究现状. 无线电工程. 2025(01): 163-174 .
3.	李东闻，钟震宇，孙羽菲，申峻宇，马子智，于川越，张玉志. 玲珑：一个小规模的高质量中文预训练语言模型. 计算机研究与发展. 2025(03): 682-693 . 本站查看
4.	陶江垚，奚雪峰，盛胜利，崔志明，左严. 结构化思维提示增强大语言模型推理能力综述. 计算机工程与应用. 2025(06): 64-83 .
5.	魏楚元，王昕，周小平，赵光哲，黄明. 大型语言模型及其在建筑行业应用研究综述. 北京建筑大学学报. 2024(02): 1-14+121 .
6.	庞进喜. 大模型在汽车国际化多语言处理中的应用. 中国汽车. 2024(05): 14-20 .
7.	王晓璐，杨云轩，谢阳斌. 创造人机对话式学习新形态——大语言模型的教育应用现状与展望. 中小学信息技术教育. 2024(05): 15-17 .
8.	马伟民. 自然语言大模型技术在政务服务智能客服系统建设中的应用. 信息与电脑(理论版). 2024(08): 86-88 .
9.	曾白凌. “被中介的真理”：Sora对媒介相合性的追问. 现代传播(中国传媒大学学报). 2024(05): 1-10 .
10.	童俊杰，申佳，赫罡，张奎. 运营商智算中心建设思路及方案. 邮电设计技术. 2024(09): 68-73 .
11.	刘同军. 生成式人工智能革新数学教学:场景与案例. 中学数学杂志. 2024(10): 1-4 .
12.	尹为民. 一种基于预训练模型的类增量学习近似重放方法分析. 电子技术. 2024(10): 144-145 .
13.	崔金满，李冬梅，田萱，孟湘皓，杨宇，崔晓晖. 提示学习研究综述. 计算机工程与应用. 2024(23): 1-27 .
14.	王珍珍，向巴卓玛，赵岩松，马星光. 以ChatGPT为代表的大型语言模型在医学教学中的应用. 医学教育管理. 2024(06): 692-697 .
15.	王琳. 大语言模型技术背景下重塑研究生论文评价与指导. 学位与研究生教育. 2024(12): 30-37 .
16.	朱俊仪，朱尚明. 利用检索增强生成技术开发本地知识库应用. 通信学报. 2024(S2): 242-247 .