As an integrated network supporting various functions (e.g., traffic management, information services, and vehicle control), Internet of Vehicle (IoV) provides information services to different entities such as vehicles, roads, humans, and clouds. IoV brings a significant contribution to road safety and traffic efficiency. However, the large and complex IoV network architecture also exposes a vast number of attack surfaces, which in turn may cause vehicle theft, information leakage, driving failure, and other security incidents. In this paper, based on an in-depth summary of the existing literature on IoV security, we divide IoV framework into two parts: the intra-vehicle network and the extra-vehicle network. Then, we further granularly divide the intra-vehicle network into three layers: the firmware layer, the intra-vehicle communication layer, and the application decision layer, while dividing the extra-vehicle network into three layers: the device layer, the extra-vehicle communication layer, and the application & data layer. Based on the framework, this paper then systematically analyses and summarizes the existing attack threats at each layer, as well as outlining and comparing the security countermeasures that can be taken at each layer. Next, we introduce the current key technologies applied in IoV security. Finally, the paper forecasts several future research directions for IoV security.