Abstract: Cellular Vehicle-to-Everything (C-V2X) communication technology is a crucial component of future Intelligent Transportation Systems (ITS). Millimeter Wave (mmWave), as one of the primary carriers for C-V2X technology, offers high bandwidth to users. However, due to limited propagation distance and sensitivity to obstructions, mmWave base stations must be densely deployed to maintain reliable communication. This requirement causes Intelligent Connected Vehicles (ICV) to frequently switch communications during travel, easily leading to local resource shortages, thus degrading service quality and user experience. To address these challenges, we treats each ICV as an agent and models the ICV communication switching issue as a cooperative multi-agent game problem. To solve this problem, the paper proposes a cooperative reinforcement learning framework based on a teammate model. Specifically, we design a teammate model to quantify the interdependencies among agents in complex dynamic environments. Furthermore, this paper propose a dynamic weight allocation scheme that generates weighted mutual information among teammates for input into the mixing network, aiming to assist teammates in switching to base stations that provide satisfactory QoS and QoE, thereby achieving high throughput and low communication switching frequency. During the algorithm training process, we designs an incentive-compatible training algorithm aimed at aligning the individual goals of the agents with collective goals, enhancing communication throughput. Experimental results demonstrate that this approach achieves a 13.8% to 38.2% increase in throughput compared to existing communication benchmark methods.