Infrared and Visible Image Fusion Based on Multiclassification Adversarial Mechanism in Feature Space

Zhang Hao; Ma Jiayi; Fan Fan; Huang Jun; Ma Yong

doi:10.7544/issn1000-1239.202110639

Journal of Computer Research and Development > 2023 > 60(3): 690-704. > DOI: 10.7544/issn1000-1239.202110639 CSTR: 32373.14.issn1000-1239.202110639

Zhang Hao, Ma Jiayi, Fan Fan, Huang Jun, Ma Yong. Infrared and Visible Image Fusion Based on Multiclassification Adversarial Mechanism in Feature Space[J]. Journal of Computer Research and Development, 2023, 60(3): 690-704. DOI: 10.7544/issn1000-1239.202110639

Citation:

PDF (5633 KB)

Infrared and Visible Image Fusion Based on Multiclassification Adversarial Mechanism in Feature Space

Electronic Information School, Wuhan University, Wuhan 430072

Funds: This work was supported by the National Natural Science Foundation of China (62075169, 62003247, 62061160370) and the Natural Science Foundation of Hubei Province (2019CFA037, 2020BAB113).

More Information

Received Date: June 10, 2021
Revised Date: June 06, 2022
Available Online: February 26, 2023

Graphical Abstract

Abstract

Abstract

To break the performance bottleneck caused by traditional fusion rules, an infrared and visible image fusion network based on multiclassification adversarial mechanism in the feature space is proposed. Compared with existing methods, the proposed method has more reasonable fusion rule and better performance. First, an autoencoder introducing attention mechanism is trained to perform the feature extraction and image reconstruction. Then, the generative adversarial network (GAN) is adopted to learn the fusion rule in the feature space extracted by the trained encoder. Specifically, we design a fusion network as the generator to fuse the features extracted from source images, and then design a multi-classifier as the discriminator. The multiclassification adversarial learning can make the fused features approximate both infrared and visible probability distribution at the same time, so as to preserve the most salient characteristics in source images. Finally, the fused image is reconstructed from the fused features by the trained decoder. Qualitative experiments show that the proposed method is in subjective evaluation better than all state-of-the-art infrared and visible image fusion methods, such as GTF, MDLatLRR, DenseFuse, FusionGAN and U2Fusion. In addition, the objective evaluation shows that the number of best quantitative metrics of our method is 2 times that of U2Fusion, and the fusion speed is more than 5 times that of other comparative methods.
- image fusion,
- fusion rule,
- deep learning,
- autoencoder,
- generative adversarial network

FullText(HTML)

References (44)

References

[1]	Ma Jiayi, Ma Yong, Li Chang. Infrared and visible image fusion methods and applications: A survey[J]. Information Fusion, 2019, 45: 153−178 doi: 10.1016/j.inffus.2018.02.004
[2]	王相海,魏婷婷,周志光,等. Contourlet四叉树系数方向相关性的遥感图像融合算法[J]. 计算机研究与发展,2013,50(8):1778−1786 doi: 10.7544/issn1000-1239.2013.20110464 Wang Xianghai, Wei Tingting, Zhou Zhiguang, et al. Research of remote sensing image fusion method based on the Contourlet coefficients' correlativity[J]. Journal of Computer Research and Development, 2013, 50(8): 1778−1786 (in Chinese) doi: 10.7544/issn1000-1239.2013.20110464
[3]	陈涛,易沫,刘忠轩,等. 相似尺度图像融合算法[J]. 计算机研究与发展,2005,42(2):2126−2131 Chen Tao, Yi Mo, Liu Zhongxuan, et al. Image fusion at similar scale[J]. Journal of Computer Research and Development, 2005, 42(2): 2126−2131 (in Chinese)
[4]	Muller A C, Narayanan S. Cognitively-engineered multisensor image fusion for military applications[J]. Information Fusion, 2009, 10(2): 137−149 doi: 10.1016/j.inffus.2008.08.008
[5]	Ma Jiayi, Zhang Hao, Shao Zhenfeng, et al. GANMcC: A generative adversarial network with multiclassification constraints for infrared and visible image fusion[J]. IEEE Transactions on Instrumentation, 2021, 70: 5005014
[6]	高雪琴,刘刚,肖刚,等. 基于FPDE的红外与可见光图像融合算法[J]. 自动化学报,2020,46(4):796−804 doi: 10.16383/j.aas.2018.c180188 Gao Xueqin, Liu Gang, Xiao Gang, et al. Fusion algorithm of infrared and visible images based on FPDE[J]. Acta Automatica Sinica, 2020, 46(4): 796−804 (in Chinese) doi: 10.16383/j.aas.2018.c180188
[7]	Li Shutao, Yang Bin, Hu Jianwen. Performance comparison of different multi-resolution transforms for image fusion[J]. Information Fusion, 2011, 12(2): 74−84 doi: 10.1016/j.inffus.2010.03.002
[8]	蔺素珍,朱小红,王栋娟,等. 基于嵌人式多尺度变换的多波段图像融合[J]. 计算机研究与发展,2015,52(4):952−959 doi: 10.7544/issn1000-1239.2015.20131736 Lin Suzhen, Zhu Xiaohong, Wang Dongjuan, et al. Multi-band image fusion based on embedded multi-scale transform[J]. Journal of Computer Research and Development, 2015, 52(4): 952−959 (in Chinese) doi: 10.7544/issn1000-1239.2015.20131736
[9]	Li Shutao, Yin Haitao, Fang Leyuan. Group-sparse representation with dictionary learning for medical image denoising and fusion[J]. IEEE Transactions on Biomedical Engineering, 2012, 59(12): 3450−3459 doi: 10.1109/TBME.2012.2217493
[10]	Kong Weiwei, Lei Yang, Zhao Huaixun. Adaptive fusion method of visible light and infrared images based on non-subsampled shearlet transform and fast non-negative matrix factorization[J]. Infrared Physics Technology, 2014, 67: 161−172 doi: 10.1016/j.infrared.2014.07.019
[11]	Zhang Xiaoye, Ma Yong, Fan Fan, et al. Infrared and visible image fusion via saliency analysis and local edge-preserving multi-scale decomposition[J]. Journal of the Optical Society of America, 2017, 34(8): 1400−1410 doi: 10.1364/JOSAA.34.001400
[12]	Ma Jiayi, Chen Chen, Li Chang, et al. Infrared and visible image fusion via gradient transfer and total variation minimization[J]. Information Fusion, 2016, 31: 100−109 doi: 10.1016/j.inffus.2016.02.001
[13]	周华兵,侯积磊,吴伟,等. 基于语义分割的红外和可见光图像融合[J]. 计算机研究与发展,2021,58(2):436−443 doi: 10.7544/issn1000-1239.2021.20200244 Zhou Huabing, Hou Jilei, Wu Wei, et al. Infrared and visible image fusion based on semantic segmentation[J]. Journal of Computer Research and Development, 2021, 58(2): 436−443 (in Chinese) doi: 10.7544/issn1000-1239.2021.20200244
[14]	Zhang Hao, Xu Han, Xiao Yang, et al. Rethinking the image fusion: A fast unified image fusion network based on proportional maintenance of gradient and intensity[C] //Proc of the 34th AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2020: 12797 − 12804
[15]	Xu Han, Ma Jiayi, Jiang Junjun, et al. U2Fusion: A unified unsupervised image fusion network[J]. IEEE Transactions on Pattern Analysis Machine Intelligence, 2020, 44(1): 502−518
[16]	Ma Jiayi, Yu Wei, Liang Pengwei, et al. FusionGAN: A generative adversarial network for infrared and visible image fusion[J]. Information Fusion, 2019, 48: 11−26 doi: 10.1016/j.inffus.2018.09.004
[17]	Ma Jiayi, Xu Han, Jiang Junjun, et al. DDcGAN: A dual-discriminator conditional generative adversarial network for multi-resolution image fusion[J]. IEEE Transactions on Image Processing, 2020, 29: 4980−4995 doi: 10.1109/TIP.2020.2977573
[18]	Li Hui, Wu Xiaojun. DenseFuse: A fusion approach to infrared and visible images[J]. IEEE Transactions on Image Processing, 2018, 28(5): 2614−2623
[19]	Goodfellow I, Pouget-Abadie J, Mirza M, et al. Generative adversarial nets[C] //Proc of the 28th Conf on Neural Information Processing Systems. Cambridge, MA: MIT Press, 2014: 2672−2680
[20]	鲍复民,李爱国,覃征. 基于SGNN的图像融合[J]. 计算机研究与发展,2005,42(3):417−423 doi: 10.1360/crad20050309 Bao Fumin, Li Aiguo, Qin Zheng. Image fusion using SGNN[J]. Journal of Computer Research and Development, 2005, 42(3): 417−423 (in Chinese) doi: 10.1360/crad20050309
[21]	Long Yongzhi, Jia Haitao, Zhong Yida, et al. RXDNFuse: A aggregated residual dense network for infrared and visible image fusion[J]. Information Fusion, 2021, 69: 128−141 doi: 10.1016/j.inffus.2020.11.009
[22]	Hou Ruichao, Zhou Dongming, Nie Rencan, et al. VIF-Net: An unsupervised framework for infrared and visible image fusion[J]. IEEE Transactions on Computational Imaging, 2020, 6: 640−651 doi: 10.1109/TCI.2020.2965304
[23]	Li Jing, Huo Hongtao, Li Chang, et al. Multigrained attention network for infrared and visible image fusion[J]. IEEE Transactions on Instrumentation Measurement, 2021, 70: 5002412
[24]	Li Jing, Huo Hongtao, Li Chang, et al. AttentionFGAN: Infrared and visible image fusion using attention-based generative adversarial networks[J]. IEEE Transactions on Multimedia, 2020, 23: 1383−1396
[25]	Ma Jiayi, Liang Pengwei, Yu Wei, et al. Infrared and visible image fusion via detail preserving adversarial learning[J]. Information Fusion, 2020, 54: 85−98 doi: 10.1016/j.inffus.2019.07.005
[26]	Wang Zhou, Bovik Alan C A. Universal image quality index[J]. IEEE Signal Processing Letters, 2002, 9(3): 81−84 doi: 10.1109/97.995823
[27]	Li Hui, Wu Xiaojun, Durrani T. Nestfuse: An infrared and visible image fusion architecture based on nest connection and spatial/channel attention models[J]. IEEE Transactions on Instrumentation Measurement, 2020, 69(12): 9645−9656 doi: 10.1109/TIM.2020.3005230
[28]	Jian Lihua, Yang Xiaomin, Liu Zheng, et al. SEDRFuse: A symmetric encoder–decoder with residual block network for infrared and visible image fusion[J]. IEEE Transactions on Instrumentation Measurement, 2021, 70: 5002215
[29]	Mao Xudong, Li Qing, Xie Haoran, et al. Least squares generative adversarial networks[C] //Proc of the 16th Int Conf on Computer Vision. Piscataway, NJ: IEEE, 2017: 2794−2802
[30]	Woo S H, Park J C, Lee J Y, et al. CBAM: Convolutional block attention module[C] //Proc of the 15th European Conf on Computer Vision. Berlin: Springer, 2018: 3−19
[31]	Huang Gao, Liu Zhuang, Laurens V D M, et al. Densely connected convolutional networks[C] //Proc of the 35th IEEE Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2017: 4700−4708
[32]	He Kaiming, Zhang Xiangyu, Ren Shaoqing, et al. Deep residual learning for image recognition[C] //Proc of the 34th IEEE Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2016: 770−778
[33]	Li Hui, Wu Xiaojun, Josef K. MDLatLRR: A novel decomposition method for infrared and visible image fusion[J]. IEEE Transactions on Image Processing, 2020, 29: 4733−4746 doi: 10.1109/TIP.2020.2975984
[34]	Alexander T, Hogervorst M A. Progress in color night vision[J]. Optical Engineering, 2012, 51(1): 010901
[35]	Ha Q, Kohei W, Karasawa T, et al. MFNet: Towards real-time semantic segmentation for autonomous vehicles with multi-spectral scenes [C] //Proc of the 30th Int Conf on Intelligent Robots and Systems. Piscataway, NJ: IEEE, 2017: 5108−5115
[36]	Xu Han, Ma Jiayi, Le Zhuliang, et al. FusionDN: A unified densely connected network for image fusion[C] //Proc of the 34th AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2020: 12484−12491
[37]	Jiang Xingyu, Ma Jiayi, Xiao Guobao, et al. A review of multimodal image matching: Methods and applications[J]. Information Fusion, 2021, 73: 22−71 doi: 10.1016/j.inffus.2021.02.012
[38]	Hamid R S, Bovik A C. Image information and visual quality[J]. IEEE Transactions on Image Processing, 2006, 15(2): 430−444 doi: 10.1109/TIP.2005.859378
[39]	Roberts J W, Van Aardt J A, Ahmed F B. Assessment of image fusion procedures using entropy, image quality, and multispectral classification[J]. Journal of Applied Remote Sensing, 2008, 2(1): 023522
[40]	Aslantas V, Bendes E. A new image quality metric for image fusion: The sum of the correlations of differences[J]. AEU-International Journal of Electronics and Communications, 2015, 69(12): 1890−1896
[41]	Qu Guihong, Zhang Dali, Yan Pingfan. Information measure for performance of image fusion[J]. Electronics Letters, 2002, 38(7): 313−315 doi: 10.1049/el:20020212
[42]	Piella G, Heijmans H. A new quality metric for image fusion[C] //Proc of the 10th IEEE Int Conf on Image Processing. Piscataway, NJ: IEEE, 2003: 173−176
[43]	Rao Yunjiang. In-fibre Bragg grating sensors[J]. Measurement Science Technology, 1997, 8(4): 355−375 doi: 10.1088/0957-0233/8/4/002
[44]	Eskicioglu A M, Fisher P S. Image quality measures and their performance[J]. IEEE Transactions on Communications, 1995, 43(12): 2959−2965 doi: 10.1109/26.477498

[1]	Qu Zhiguo, Chen Weilong, Sun Le, Liu Wenjie, Zhang Yanchun. ECG-QGAN: A ECG Generative Information System Based on Quantum Generative Adversarial Networks[J]. Journal of Computer Research and Development. DOI: 10.7544/issn1000-1239.202440527
[2]	Xue Zhihang, Xu Zheming, Lang Congyan, Feng Songhe, Wang Tao, Li Yidong. Text-to-Image Generation Method Based on Image-Text Semantic Consistency[J]. Journal of Computer Research and Development, 2023, 60(9): 2180-2190. DOI: 10.7544/issn1000-1239.202220416
[3]	Zhang Jing, Ju Jialiang, Ren Yonggong. Double-Generators Network for Data-Free Knowledge Distillation[J]. Journal of Computer Research and Development, 2023, 60(7): 1615-1627. DOI: 10.7544/issn1000-1239.202220024
[4]	Liu Guangrui, Zhang Weizhe, Li Xinjie. Data Contamination Defense Method for Intelligent Network Intrusion Detection Systems Based on Edge Examples[J]. Journal of Computer Research and Development, 2022, 59(10): 2348-2361. DOI: 10.7544/issn1000-1239.20220509
[5]	Guo Zhengshan, Zuo Jie, Duan Lei, Li Renhao, He Chengxin, Xiao Yingjie, Wang Peiyan. A Generative Adversarial Negative Sampling Method for Knowledge Hypergraph Link Prediction[J]. Journal of Computer Research and Development, 2022, 59(8): 1742-1756. DOI: 10.7544/issn1000-1239.20220074
[6]	Kong Leyi, Zhang Jinyi, Lou Liangliang. Tolerance Feature Extension of Substandard Sign Language Recognition with Finite Samples[J]. Journal of Computer Research and Development, 2022, 59(3): 683-693. DOI: 10.7544/issn1000-1239.20200789
[7]	Chen Dawei, Fu Anmin, Zhou Chunyi, Chen Zhenzhu. Federated Learning Backdoor Attack Scheme Based on Generative Adversarial Network[J]. Journal of Computer Research and Development, 2021, 58(11): 2364-2373. DOI: 10.7544/issn1000-1239.2021.20210659
[8]	Qian Yaguan, He Niannian, Guo Yankai, Wang Bin, Li Hui, Gu Zhaoquan, Zhang Xuhong, Wu Chunming. An Evasion Algorithm to Fool Fingerprint Detector for Deep Neural Networks[J]. Journal of Computer Research and Development, 2021, 58(5): 1106-1117. DOI: 10.7544/issn1000-1239.2021.20200903
[9]	Tian Jiwei, Wang Jinsong, Shi Kai. Positive and Unlabeled Generative Adversarial Network on POI Positioning[J]. Journal of Computer Research and Development, 2019, 56(9): 1843-1850. DOI: 10.7544/issn1000-1239.2019.20180847
[10]	Song Kehui, Zhang Ying, Zhang Jiangwei, Yuan Xiaojie. A Generative Model for Synthesizing Structured Datasets Based on GAN[J]. Journal of Computer Research and Development, 2019, 56(9): 1832-1842. DOI: 10.7544/issn1000-1239.2019.20180353

Cited By

Cited by

Periodical cited type(4)

1.	徐孟艳，屈炎伟. 远程医疗患者信息访问的多重身份认证仿真. 计算机仿真. 2025(01): 447-451 .
2.	蒲阳. 电网高压输电无人机巡检关键点图像融合研究. 电子设计工程. 2025(08): 150-154 .
3.	付启银. 基于视觉传达技术的可见光与红外图像融合方法. 激光杂志. 2025(04): 128-133 .
4.	蒋顺生. 基于视觉传达的大型船舶航行图像融合研究. 舰船科学技术. 2023(18): 178-181 .