• 中国精品科技期刊
  • CCF推荐A类中文期刊
  • 计算领域高质量科技期刊T1类
Advanced Search
Zhao Lei, Ji Boyan, Xing Wei, Lin Huaizhong, Lin Zhijie. Ancient Painting Inpainting Algorithm Based on Multi-Channel Encoder and Dual Attention[J]. Journal of Computer Research and Development, 2023, 60(12): 2814-2831. DOI: 10.7544/issn1000-1239.202220648
Citation: Zhao Lei, Ji Boyan, Xing Wei, Lin Huaizhong, Lin Zhijie. Ancient Painting Inpainting Algorithm Based on Multi-Channel Encoder and Dual Attention[J]. Journal of Computer Research and Development, 2023, 60(12): 2814-2831. DOI: 10.7544/issn1000-1239.202220648

Ancient Painting Inpainting Algorithm Based on Multi-Channel Encoder and Dual Attention

Funds: This work was supported by the Zhejiang Elite Program(2022C01222), the National Key Research and Development Program of China (2020YFC1522704), the National Natural Science Foundation of China (62172365), the Natural Science Foundation of Zhejiang Province (LY21F020005, LY19F020049), the Key Program of the National Social Science Foundation of China (19ZDA197), the Zhejiang Cultural Relics Protection Science and Technology Project (2019011), the Key Scientific Research Base for Digital Conservation of Cave Temples of State Administration for Cultural Heritage and the Project of MOE Frontier Science Center for Brain Science & Brain-Machine Integration (Zhejiang University) (2021008).
More Information
  • Author Bio:

    Zhao Lei: born in 1975. PhD, associate professor. Member of CCF. His main research interests include image restoration, deep learning, and intelligent image generation

    Ji Boyan: born in 2000. Master candidate. His main research interest includes image cross domain translation. (qinglanwuji@zju.edu.cn

    Xing Wei: born in 1967. PhD, associate professor. His main research interest includes image intelligent processing

    Lin Huaizhong: born in 1970. PhD, associate professor. His main research interest includes image intelligent processing

    Lin Zhijie: born in 1980. PhD, associate professor. His main research interest includes image intelligent processing.(bytelin@qq.com

  • Received Date: July 23, 2022
  • Revised Date: January 10, 2023
  • Available Online: September 19, 2023
  • Painting is an important form of culture and art. For thousands of years, a large number of paintings have been produced in ancient China. And paintings contain rich cultural, artistic, scientific, and historical values. But due to natural disasters (earthquake) and various reasons such as natural weathering and more and more human economic activities, some paintings are more or less damaged or missing in large pieces, which seriously affects the appreciation, cultural creativity, cultural communication and other activities based on these paintings. Compared with natural images, ancient painting images usually have high self-similarity, obvious style characteristics, rich cultural connotation and delicate texture. Although impressive progress has been made in the inpainting technology of natural images, these methods cannot be directly applied to the inpainting of ancient Chinese paintings. Combined with the characteristics of ancient Chinese paintings, we design the algorithm and model structure and proposes a Chinese ancient painting inpainting algorithm based on a multi-channel encoder and dual attention module. The goal is to automatically repair the damaged ancient paintings. In order to better repair the ancient paintings from multiple scales, we use a multi-channel encoder to learn the semantic features of ancient paintings at different scales and repair ancient paintings through the learned macro, meso, and micro semantic features, which solves the difficult problem of rich and delicate texture repair of ancient paintings. In order to better learn the global semantic features of ancient paintings and conduct the harmony and consistency of the repaired ancient paintings, we use the dual attention module to learn the global semantic features of ancient paintings from two aspects: style and content. In order to verify the advanced nature of the algorithm, an ancient painting data set is produced. Experiments on this dataset prove that the algorithm proposed in this paper has better repair quality than the SOTA algorithms.

  • [1]
    Nguyen A, Dosovitskiy A, Yosinski J, et al. Synthesizing the preferred inputs for neurons in neural networks via deep generator networks [C] // Proc of the 30th Advances in Neural Information Processing Systems (NIPS). La Jolla, CA: NIPS, 2017: 11−19
    [2]
    Fukushima K. Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position[J]. Biological Cybernetics, 1980, 36(1): 189−196
    [3]
    LeCun Y, Boser B, Denker J, et al. Backpropagation applied to handwritten zip code recognition[J]. Neural Computation, 1989, 1(1): 146−156
    [4]
    Pathak D, Krahenbuhl P, Donahue J, et al. Context encoders: Feature learning by inpainting [C] //Proc of the 31st IEEE Conf on Computer Vision and Pattern Recognition. Los Alamitos, CA: IEEE Computer Society, 2016: 1033−1038
    [5]
    Goodfellow I, Pouget-Abadie J, Mirza M, et al. Generative adversarial networks [C] //Proc of the 29th IEEE Conf on Computer Vision and Pattern Recognition (CVPR). Los Alamitos, CA: IEEE Computer Society, 2014: 2672−2680
    [6]
    Efros A, Leung T. Texture synthesis by non-parametric sampling [C] //Proc of the 7th IEEE Int Conf on Computer Vision (ICCV). Piscataway, NJ: IEEE, 1999: 1033–1038
    [7]
    Bertalmio M, Sapiro G, Caselleset V et al. Image inpainting [C] //Proc of the 8th ACM Int Conf on Multimedia (ACM MM). New York: ACM, 2000: 417−424
    [8]
    Telea A. An image inpainting technique based on the fast marching method[J]. Journals of Graphics Tools, 2004, 9(1): 25−36
    [9]
    Chan T, Shen Jiahong. Non texture inpainting by curvature-driven diffusion[J]. Journal of Visual Communication and Image Representation, 2001, 12(4): 436−449 doi: 10.1006/jvci.2001.0487
    [10]
    Chan T, Kang S, Shen Jiahong. Euler’s Elastica and curvature-based inpainting[J]. SIAM Journal on Applied Mathematics, 2003, 63(2): 564−592 doi: 10.1137/S0036139901390088
    [11]
    Tsai A, Yezzi A, Willsky A. Curve evolution implementation of the Mumford-Shah functional for image segmentation, denoising, interpolation, and magnification[J]. IEEE Transactions Image Processing, 2001, 10(8): 1169−1186 doi: 10.1109/83.935033
    [12]
    Shen Jiahong, Kang S, Chan T. Euler's Elastica and curvature based inpainting[J]. SIAM Journal on Applied Mathematics, 2002, 63(2): 564−592
    [13]
    Esedoglu S, Shen Jiahong. Digital inpainting based on the Mumford–Shah–Euler image model[J]. European Journal of Applied Mathematics, 2002, 13(4): 353−370
    [14]
    Efros A, Leung T. Texture synthesis by nonparametric sampling [C] //Proc of the 7th IEEE Int Conf on Computer Vision (ICCV). Piscataway, NJ: IEEE, 1999: 1033–1038
    [15]
    Efros A, Freeman T. Image quilting for texture synthesis and transfer [C] //Proc of the 8th Int Conf of ACM SIGGRAPH. New York: ACM, 2001: 341–346
    [16]
    Bertalmio M, Vese L, Sapiro G. Simultaneous structure and texture image inpainting[J]. IEEE Transactions on Image Processing, 2003, 12(8): 882−889 doi: 10.1109/TIP.2003.815261
    [17]
    Bertalmio M, Vese L, Sapiro G et a1. Image filling in a decomposition space [C] //Proc of the 10th Int Conf on Image Proceeding. Los Alamitos, CA: IEEE Computer Society, 2003: 852−855
    [18]
    Drori I, Cohen D, Yeshurun H. Fragment-based image completion[J]. ACM Transaction on Graphics, 2003, 22(3): 303−312 doi: 10.1145/882262.882267
    [19]
    Criminisi A, Perez P, Toyama K. Region filling and object removal by exemplar-based image inpainting[J]. IEEE Transacions on Image Processing, 2004, 13(9): 1200−1212 doi: 10.1109/TIP.2004.833105
    [20]
    Barnes C, Shechtman E, Goldman D. PatchMatch: A randomized correspondence algorithm for structural image editing[J]. ACM Transaction on Graphics, 2009, 28(3): 2401−24011
    [21]
    Zeng Y, Yi G, Jin Z. Feature learning and patch matching for diverse image inpainting[J]. Pattern Recognition, 2021, 119(2021): 108036 doi: 10.1109/TIP.2007.911828
    [22]
    Stark J, Elad M, Donoho D. Image decomposition via the combination of sparse representations and variational approach[J]. IEEE Transactions Image Processing, 2005, 14(10): 1570−1582 doi: 10.1109/TIP.2005.852206
    [23]
    Elad M, Stark J, Querre P, et al. Simultaneous cartoon and texture image inpaintingusing morphological component analysis[J]. Applied & Computational Harmonic Analysis, 2005, 19(3): 340−358
    [24]
    Mairal J, Elad M, Sapiro G. Sparse representation for color image restoration[J]. IEEE Transactions on Image Processing, 2008, 17(1): 53−69 doi: 10.1109/TIP.2007.911828
    [25]
    Mairal J, Bach F, Ponce J, et al. Online learning for matrix factorization and sparse coding[J]. Journal of Machine Learning Research, 2010, 11(1): 19−60
    [26]
    Shen Bin, Hu Wei, Zhang Yimin, et al. Image inpainting via sparse representation [C] //Proc of the 7th IEEE Int Conf on Acoustics, Speech and Signal Processing (ICASSP). Piscataway, NJ: IEEE, 2009: 697−700
    [27]
    Li Xin. Image recovery via hybrid sparse representations: A deterministic annealing approach[J]. IEEE Journal of Selected Topics Signal Processing, 2011, 5(5): 953−962 doi: 10.1109/JSTSP.2011.2138676
    [28]
    Gabriel P. Sparse modeling of textures[J]. Journal of Mathematical Imaging & Vision, 2009, 34(1): 17−31
    [29]
    Zhang Jian, Zhao Debin, Xiong Ruiqin, et al. Image restoration using joint statistical modeling in space-transform domain[J]. IEEE Transactions on Circuits System and Video Technology, 2014, 24(6): 915−928 doi: 10.1109/TCSVT.2014.2302380
    [30]
    Zhang Jian, Zhao Debin, Gao Wen. Group-based sparse representation for image restoration[J]. IEEE Transactions on Image Processing, 2014, 23(8): 3336−3351 doi: 10.1109/TIP.2014.2323127
    [31]
    Zhang Lin, Kang Bin, Liu Bin, et al. A new inpainting method for object removal based on patch local feature and sparse representation[J]. International Journal of Innovative Computing Information and Control, 2016, 12(1): 113−124
    [32]
    Goodfellow I, Pouget J, Mirza M, et al. Generative adversarial nets [C] //Proc of the 27th Advances in Neural Information Processing Systems (NIPS). La Jolla, CA: NIPS, 2014: 2672–2680
    [33]
    Yang Chao, Lu Xin, Lin Zhe, et al. High-resolution image inpainting using multi-scale neural patch synthesis [C] //Proc of the 11th IEEE Int Conf on Computer Vision (ICCV). Piscataway, NJ: IEEE, 2017: 1033–1038
    [34]
    Zeng Yanhong, Fu, Jialong, Chao Hongyang, et al. Learning pyramid context encoder network for high-quality image inpainting [C] // Proc of the 32nd IEEE Conf on Computer Vision and Pattern Recognition (CVPR). Los Alamitos, CA: IEEE Computer Society, 2019: 1486–1494
    [35]
    Ren Yurui, Yu Xiaoming, Zhang Ruonan, et al. Structureflow: Image inpainting via structure-aware appearance flow [C] //Proc of the 13th IEEE Int Conf on Computer Vision (ICCV). Piscataway, NJ: IEEE, 2019 : 181–190
    [36]
    Xiong Wei, Yu Jiahui, Lin Zhe, et al. Foreground-aware image inpainting[C] //Proc of the 32nd IEEE Conf on Computer Vision and Pattern Recognition (CVPR). Los Alamitos, CA: IEEE Computer Society, 2019: 5840–5848
    [37]
    Zheng Chuanxia, Cham T, Cai Jiafei. Pluralistic image completion[C] // Proc of the 32nd IEEE Conf on Computer Vision and Pattern Recognition (CVPR). Los Alamitos, CA: IEEE Computer Society, 2019: 1438–1447
    [38]
    Zhao Lei, Mo Qihang, Lin Sihuan, et al. UCTGAN: Diverse image inpainting based on unsupervised cross-space translation[C] //Proc of the 33rd IEEE Conf on Computer Vision and Pattern Recognition (CVPR). Los Alamitos, CA: IEEE Computer Society, 2020: 5741–5750
    [39]
    Liao Liang, Xiao Jing, Wang Zheng, et al. Image inpainting guided by coherence priors of semantics and textures [C] //Proc of the 34th IEEE Conf on Computer Vision and Pattern Recognition (CVPR). Los Alamitos, CA: IEEE Computer Society, 2021: 6539−6548
    [40]
    Liu Hongyu, Jiang Bin, Song Yibing, et al. Rethinking image inpainting via a mutual encoder decoder with feature equalizations [C] //Proc of the 18th European Conf on Computer Vision (ECCV). Berlin: Springer, 2020: 725−741
    [41]
    Nazeri K, Ng E, Joseph T, et al. EdgeConnect: Generative image inpainting with adversarial edge learning [C] //Proc of the 11th IEEE Int Conf on Computer Vision (ICCV). Piscataway, NJ: IEEE, 2019: 472−480
    [42]
    Yu Jiahui, Lin Zhe, Yang Jimei, et al. Free-form image inpainting with gated convolution [C] //Proc of the 11th IEEE Int Conf on Computer Vision (ICCV). Piscataway, NJ: IEEE, 2019: 4471−4480
    [43]
    Yu Fang, Koltun V, Funkhouser T. Dilated residual networks[C] //Proc of the 31st IEEE Conf on Computer Vision and Pattern Recognition (CVPR). Los Alamitos, CA: IEEE Computer Society, 2017: 472−480
    [44]
    Clevert D, Unterthiner T, Hochreiter S. Fast and accurate deep network learning by exponential linear units (Elus)[J]. arXiv preprint, arXiv: 1511.07289, 2015
    [45]
    Zhang Han, Goodfellow I, Metaxas D, et al. Self-attention generative adversarial networks [C] //Proc of the 36th Int Conf on Machine Learning (ICML). New York: ACM, 2019: 7354−7363
    [46]
    Gulrajani I, Ahmed F, Arjovsky M, et al. Improved training of wasserstein GANs[C] // Proc of the 30th Advances in Neural Information Processing Systems (NIPs). La Jolla, CA: NIPS, 2017: 5767−5777
    [47]
    Arjovsky M, Chintala S, Bottou L. Wasserstein generative adversarial networks [C] //Proc of the 30th Advances in Neural Information Processing Systems (NIPS). La Jolla, CA: NIPS, 2017: 214−223
    [48]
    Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition[J]. arXiv preprint, arXiv: 1409.1556, 2014
    [49]
    封顺. 论中国画中的“留白”艺术[J]. 大舞台,2013(6):90−91

    Feng Shun. On the art of "leaving blank" in Chinese painting[J]. Grand Stage, 2013(6): 90−91 (in Chinese)
    [50]
    Arjovsky M, Chintala S, Bottou L. Wasserstein generative adversarial networks[C] //Proc of the 34th Int Conf on Machine Learning (ICML). New York: ACM, 2017: 214−223
    [51]
    Yu Jiahui, Lin Zhe, Yang Jimeiecc, et al. Generative image inpainting with contextual attention [C] // Proc of the 31st IEEE Conf on Computer Vision and Pattern Recognition (CVPR). Los Alamitos, CA: IEEE Computer Society, 2018: 5505−5514
    [52]
    Liu Guilin, Reda F A, Shih K J, et al. Image inpainting for irregular holes using partial convolutions [C] //Proc of the 16th European Conf on Computer Vision (ECCV). Berlin: Springer, 2018: 85−100
  • Related Articles

    [1]Lai Baoqiang, Li Zheng, Zhao Ruilian, Guo Junxia. Context-Aware Based API Recommendation with Diversity[J]. Journal of Computer Research and Development, 2023, 60(10): 2335-2347. DOI: 10.7544/issn1000-1239.202220317
    [2]Tang Dan, Cai Hongliang, Geng Wei. Decoding Method of Reed-Solomon Erasure Codes[J]. Journal of Computer Research and Development, 2022, 59(3): 582-596. DOI: 10.7544/issn1000-1239.20210575
    [3]Zhang Bing, Wen Zheng, Wei Xiaoyu, Ren Jiadong. InterDroid: An Interpretable Android Malware Detection Method for Conceptual Drift[J]. Journal of Computer Research and Development, 2021, 58(11): 2456-2474. DOI: 10.7544/issn1000-1239.2021.20210560
    [4]Yang Wang, Gao Mingzhe, Jiang Ting. A Malicious Code Static Detection Framework Based on Multi-Feature Ensemble Learning[J]. Journal of Computer Research and Development, 2021, 58(5): 1021-1034. DOI: 10.7544/issn1000-1239.2021.20200912
    [5]Guo Jinyang, Shao Chuanming, Wang Jing, Li Chao, Zhu Haojin, Guo Minyi. Programming and Developing Environment for FPGA Graph Processing: Survey and Exploration[J]. Journal of Computer Research and Development, 2020, 57(6): 1164-1178. DOI: 10.7544/issn1000-1239.2020.20200106
    [6]Zheng Zhen, Zhai Jidong, Li Yan, Chen Wenguang. Workload Analysis for Typical GPU Programs Using CUPTI Interface[J]. Journal of Computer Research and Development, 2016, 53(6): 1249-1262. DOI: 10.7544/issn1000-1239.2016.20148354
    [7]Jiao Sibei, Ying Lingyun, Yang Yi, Cheng Yao, Su Purui, and Feng Dengguo. An Anti-Obfuscation Method for Detecting Similarity Among Android Applications in Large Scale[J]. Journal of Computer Research and Development, 2014, 51(7): 1446-1457.
    [8]Dong Longming, Wang Ji, Chen Liqian, Dong Wei. Memory Leak Detection for Heap-Manipulating Programs Based on Local Heap Abstraction[J]. Journal of Computer Research and Development, 2012, 49(9): 1832-1842.
    [9]Ma Peijun, Wang Tiantian, and Su Xiaohong. Automatic Grading of Student Programs Based on Program Understanding[J]. Journal of Computer Research and Development, 2009, 46(7): 1136-1142.
    [10]Wang Zhaofei and Huang Chun. Static Detection of Deadlocks in OpenMP Fortran Programs[J]. Journal of Computer Research and Development, 2007, 44(3).
  • Cited by

    Periodical cited type(1)

    1. 郭龙,梁灿,李彦丽. 知识库中标注词句序列命名实体识别方法. 计算机仿真. 2024(11): 512-516 .

    Other cited types(2)

Catalog

    Article views (187) PDF downloads (97) Cited by(3)

    /

    DownLoad:  Full-Size Img  PowerPoint
    Return
    Return