Abstract: Cellular vehicle-to-everything (C-V2X) communication technology is a crucial component of future intelligent transportation systems (ITS). Millimeter wave (mmWave), as one of the primary carriers for C-V2X technology, offers high bandwidth to users. However, due to limited propagation distance and sensitivity to obstructions, mmWave base stations must be densely deployed to maintain reliable communication. This requirement causes intelligent connected vehicle (ICV) to frequently switch communications during travel, easily leading to local resource shortages, thus degrading service quality and user experience. To address these challenges, we treat each ICV as an agent and model the ICV communication switching issue as a cooperative multi-agent game problem. To solve this problem, we propose a cooperative reinforcement learning framework based on a teammate model. Specifically, we design a teammate model to quantify the interdependencies among agents in complex dynamic environments. Furthermore, we propose a dynamic weight allocation scheme that generates weighted mutual information among teammates for the input of the mixing network, aiming to assist teammates in switching to base stations that provide satisfactory QoS and QoE, thereby achieving high throughput and low communication switching frequency. During the algorithm training process, we design an incentive-compatible training algorithm aimed at aligning the individual goals of the agents with collective goals, enhancing communication throughput. Experimental results demonstrate that this algorithm achieves a 13.8% to 38.2% increase in throughput compared with existing communication benchmark algorithms.