ISSN 1000-1239 CN 11-1777/TP

Journal of Computer Research and Development ›› 2021, Vol. 58 ›› Issue (2): 436-443.doi: 10.7544/issn1000-1239.2021.20200244

Previous Articles    

Infrared and Visible Image Fusion Based on Semantic Segmentation

Zhou Huabing1,2, Hou Jilei1,2, Wu Wei1,2, Zhang Yanduo1,2, Wu Yuntao1,2, Ma Jiayi3   

  1. 1(College of Computer Science and Engineering, Wuhan Institute of Technology, Wuhan 430205);2(Hubei Key Laboratory of Intelligent Robot (Wuhan Institute of Technology), Wuhan 430205);3(Electronic Information School, Wuhan University, Wuhan 430072)
  • Online:2021-02-01
  • Supported by: 
    This work was supported by the National Natural Science Foundation of China (61771353, 61773295, 62072350, 41501505) and the Hubei Technology Innovation Project (2019AAA045).

Abstract: Infrared images can distinguish targets from their backgrounds due to the difference in thermal radiation even in poor lighting conditions. By contrast, visible images can represent texture details with high spatial resolution. Meanwhile, both of infrared and visible images preserve corresponding semantic information. Therefore, infrared and visible image fusion should keep both radiation information of the infrared image and texture details of the visible image; additionally, it needs to reserve the semantic information of both. Semantic segmentation can transform the source images into the masks with semantic information. In this paper, an infrared and visible image fusion method is proposed based on semantic segmentation. It can overcome the shortcomings that the existing fusion methods are not specific to different regions. Considering the specific information for different regions of infrared and visible images, we design two loss functions for different regions to improve the quality of fused image under the framework of generative adversarial network. Firstly, we gain the masks of the infrared images with semantic information by semantic segmentation; then we use the masks to divide the infrared and visible images into infrared target area, infrared background area, visible target area, and visible background area. Secondly, we employ different methods to fuse the target and background area, respectively. Finally, we combine the two regions to obtain the final fused image. The experiment shows that the proposed method outperforms state-of-the-art, where our results have higher contrast in the target area and richer texture details in the background area.

Key words: infrared image, visible image, image fusion, semantic segmentation, mask

CLC Number: