Ancient Painting Inpainting Algorithm Based on Multi-Channel Encoder and Dual Attention

Zhao Lei; Ji Boyan; Xing Wei; Lin Huaizhong; Lin Zhijie

doi:10.7544/issn1000-1239.202220648

Journal of Computer Research and Development > 2023 > 60(12): 2814-2831. > DOI: 10.7544/issn1000-1239.202220648

Zhao Lei, Ji Boyan, Xing Wei, Lin Huaizhong, Lin Zhijie. Ancient Painting Inpainting Algorithm Based on Multi-Channel Encoder and Dual Attention[J]. Journal of Computer Research and Development, 2023, 60(12): 2814-2831. DOI: 10.7544/issn1000-1239.202220648

Citation:

PDF (6458 KB)

Ancient Painting Inpainting Algorithm Based on Multi-Channel Encoder and Dual Attention

1.
School of Computer Science and Technology, Zhejiang University, Hangzhou 310027
2.
School of Information and Electronic Engineering, Zhejiang University of Science and Technology, Hangzhou 310023

Funds: This work was supported by the Zhejiang Elite Program（2022C01222）, the National Key Research and Development Program of China (2020YFC1522704), the National Natural Science Foundation of China (62172365), the Natural Science Foundation of Zhejiang Province (LY21F020005, LY19F020049), the Key Program of the National Social Science Foundation of China (19ZDA197), the Zhejiang Cultural Relics Protection Science and Technology Project (2019011), the Key Scientific Research Base for Digital Conservation of Cave Temples of State Administration for Cultural Heritage and the Project of MOE Frontier Science Center for Brain Science & Brain-Machine Integration (Zhejiang University) (2021008).

More Information

Author Bio:
Zhao Lei: born in 1975. PhD, associate professor. Member of CCF. His main research interests include image restoration, deep learning, and intelligent image generation

Ji Boyan: born in 2000. Master candidate. His main research interest includes image cross domain translation. （qinglanwuji@zju.edu.cn）

Xing Wei: born in 1967. PhD, associate professor. His main research interest includes image intelligent processing

Lin Huaizhong: born in 1970. PhD, associate professor. His main research interest includes image intelligent processing

Lin Zhijie: born in 1980. PhD, associate professor. His main research interest includes image intelligent processing.（bytelin@qq.com）
Received Date: July 23, 2022
Revised Date: January 10, 2023
Available Online: September 19, 2023

Graphical Abstract

Abstract

Abstract

Painting is an important form of culture and art. For thousands of years, a large number of paintings have been produced in ancient China. And paintings contain rich cultural, artistic, scientific, and historical values. But due to natural disasters (earthquake) and various reasons such as natural weathering and more and more human economic activities, some paintings are more or less damaged or missing in large pieces, which seriously affects the appreciation, cultural creativity, cultural communication and other activities based on these paintings. Compared with natural images, ancient painting images usually have high self-similarity, obvious style characteristics, rich cultural connotation and delicate texture. Although impressive progress has been made in the inpainting technology of natural images, these methods cannot be directly applied to the inpainting of ancient Chinese paintings. Combined with the characteristics of ancient Chinese paintings, we design the algorithm and model structure and proposes a Chinese ancient painting inpainting algorithm based on a multi-channel encoder and dual attention module. The goal is to automatically repair the damaged ancient paintings. In order to better repair the ancient paintings from multiple scales, we use a multi-channel encoder to learn the semantic features of ancient paintings at different scales and repair ancient paintings through the learned macro, meso, and micro semantic features, which solves the difficult problem of rich and delicate texture repair of ancient paintings. In order to better learn the global semantic features of ancient paintings and conduct the harmony and consistency of the repaired ancient paintings, we use the dual attention module to learn the global semantic features of ancient paintings from two aspects: style and content. In order to verify the advanced nature of the algorithm, an ancient painting data set is produced. Experiments on this dataset prove that the algorithm proposed in this paper has better repair quality than the SOTA algorithms.
- painting image inpainting,
- multi-channel encoder,
- dual attention mechanism,
- style expression,
- multi-scale reatoration

FullText(HTML)

References (52)

References

[1]	Nguyen A, Dosovitskiy A, Yosinski J, et al. Synthesizing the preferred inputs for neurons in neural networks via deep generator networks [C] // Proc of the 30th Advances in Neural Information Processing Systems (NIPS). La Jolla, CA: NIPS, 2017: 11−19
[2]	Fukushima K. Neocognitron: A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position[J]. Biological Cybernetics, 1980, 36(1): 189−196
[3]	LeCun Y, Boser B, Denker J, et al. Backpropagation applied to handwritten zip code recognition[J]. Neural Computation, 1989, 1(1): 146−156
[4]	Pathak D, Krahenbuhl P, Donahue J, et al. Context encoders: Feature learning by inpainting [C] //Proc of the 31st IEEE Conf on Computer Vision and Pattern Recognition. Los Alamitos, CA: IEEE Computer Society, 2016: 1033−1038
[5]	Goodfellow I, Pouget-Abadie J, Mirza M, et al. Generative adversarial networks [C] //Proc of the 29th IEEE Conf on Computer Vision and Pattern Recognition (CVPR). Los Alamitos, CA: IEEE Computer Society, 2014: 2672−2680
[6]	Efros A, Leung T. Texture synthesis by non-parametric sampling [C] //Proc of the 7th IEEE Int Conf on Computer Vision (ICCV). Piscataway, NJ: IEEE, 1999: 1033–1038
[7]	Bertalmio M, Sapiro G, Caselleset V et al. Image inpainting [C] //Proc of the 8th ACM Int Conf on Multimedia (ACM MM). New York: ACM, 2000: 417−424
[8]	Telea A. An image inpainting technique based on the fast marching method[J]. Journals of Graphics Tools, 2004, 9(1): 25−36
[9]	Chan T, Shen Jiahong. Non texture inpainting by curvature-driven diffusion[J]. Journal of Visual Communication and Image Representation, 2001, 12(4): 436−449 doi: 10.1006/jvci.2001.0487
[10]	Chan T, Kang S, Shen Jiahong. Euler’s Elastica and curvature-based inpainting[J]. SIAM Journal on Applied Mathematics, 2003, 63(2): 564−592 doi: 10.1137/S0036139901390088
[11]	Tsai A, Yezzi A, Willsky A. Curve evolution implementation of the Mumford-Shah functional for image segmentation, denoising, interpolation, and magnification[J]. IEEE Transactions Image Processing, 2001, 10(8): 1169−1186 doi: 10.1109/83.935033
[12]	Shen Jiahong, Kang S, Chan T. Euler's Elastica and curvature based inpainting[J]. SIAM Journal on Applied Mathematics, 2002, 63(2): 564−592
[13]	Esedoglu S, Shen Jiahong. Digital inpainting based on the Mumford–Shah–Euler image model[J]. European Journal of Applied Mathematics, 2002, 13(4): 353−370
[14]	Efros A, Leung T. Texture synthesis by nonparametric sampling [C] //Proc of the 7th IEEE Int Conf on Computer Vision (ICCV). Piscataway, NJ: IEEE, 1999: 1033–1038
[15]	Efros A, Freeman T. Image quilting for texture synthesis and transfer [C] //Proc of the 8th Int Conf of ACM SIGGRAPH. New York: ACM, 2001: 341–346
[16]	Bertalmio M, Vese L, Sapiro G. Simultaneous structure and texture image inpainting[J]. IEEE Transactions on Image Processing, 2003, 12(8): 882−889 doi: 10.1109/TIP.2003.815261
[17]	Bertalmio M, Vese L, Sapiro G et a1. Image filling in a decomposition space [C] //Proc of the 10th Int Conf on Image Proceeding. Los Alamitos, CA: IEEE Computer Society, 2003: 852−855
[18]	Drori I, Cohen D, Yeshurun H. Fragment-based image completion[J]. ACM Transaction on Graphics, 2003, 22(3): 303−312 doi: 10.1145/882262.882267
[19]	Criminisi A, Perez P, Toyama K. Region filling and object removal by exemplar-based image inpainting[J]. IEEE Transacions on Image Processing, 2004, 13(9): 1200−1212 doi: 10.1109/TIP.2004.833105
[20]	Barnes C, Shechtman E, Goldman D. PatchMatch: A randomized correspondence algorithm for structural image editing[J]. ACM Transaction on Graphics, 2009, 28(3): 2401−24011
[21]	Zeng Y, Yi G, Jin Z. Feature learning and patch matching for diverse image inpainting[J]. Pattern Recognition, 2021, 119(2021): 108036 doi: 10.1109/TIP.2007.911828
[22]	Stark J, Elad M, Donoho D. Image decomposition via the combination of sparse representations and variational approach[J]. IEEE Transactions Image Processing, 2005, 14(10): 1570−1582 doi: 10.1109/TIP.2005.852206
[23]	Elad M, Stark J, Querre P, et al. Simultaneous cartoon and texture image inpaintingusing morphological component analysis[J]. Applied & Computational Harmonic Analysis, 2005, 19(3): 340−358
[24]	Mairal J, Elad M, Sapiro G. Sparse representation for color image restoration[J]. IEEE Transactions on Image Processing, 2008, 17(1): 53−69 doi: 10.1109/TIP.2007.911828
[25]	Mairal J, Bach F, Ponce J, et al. Online learning for matrix factorization and sparse coding[J]. Journal of Machine Learning Research, 2010, 11(1): 19−60
[26]	Shen Bin, Hu Wei, Zhang Yimin, et al. Image inpainting via sparse representation [C] //Proc of the 7th IEEE Int Conf on Acoustics, Speech and Signal Processing (ICASSP). Piscataway, NJ: IEEE, 2009: 697−700
[27]	Li Xin. Image recovery via hybrid sparse representations: A deterministic annealing approach[J]. IEEE Journal of Selected Topics Signal Processing, 2011, 5(5): 953−962 doi: 10.1109/JSTSP.2011.2138676
[28]	Gabriel P. Sparse modeling of textures[J]. Journal of Mathematical Imaging & Vision, 2009, 34(1): 17−31
[29]	Zhang Jian, Zhao Debin, Xiong Ruiqin, et al. Image restoration using joint statistical modeling in space-transform domain[J]. IEEE Transactions on Circuits System and Video Technology, 2014, 24(6): 915−928 doi: 10.1109/TCSVT.2014.2302380
[30]	Zhang Jian, Zhao Debin, Gao Wen. Group-based sparse representation for image restoration[J]. IEEE Transactions on Image Processing, 2014, 23(8): 3336−3351 doi: 10.1109/TIP.2014.2323127
[31]	Zhang Lin, Kang Bin, Liu Bin, et al. A new inpainting method for object removal based on patch local feature and sparse representation[J]. International Journal of Innovative Computing Information and Control, 2016, 12(1): 113−124
[32]	Goodfellow I, Pouget J, Mirza M, et al. Generative adversarial nets [C] //Proc of the 27th Advances in Neural Information Processing Systems (NIPS). La Jolla, CA: NIPS, 2014: 2672–2680
[33]	Yang Chao, Lu Xin, Lin Zhe, et al. High-resolution image inpainting using multi-scale neural patch synthesis [C] //Proc of the 11th IEEE Int Conf on Computer Vision (ICCV). Piscataway, NJ: IEEE, 2017: 1033–1038
[34]	Zeng Yanhong, Fu, Jialong, Chao Hongyang, et al. Learning pyramid context encoder network for high-quality image inpainting [C] // Proc of the 32nd IEEE Conf on Computer Vision and Pattern Recognition (CVPR). Los Alamitos, CA: IEEE Computer Society, 2019: 1486–1494
[35]	Ren Yurui, Yu Xiaoming, Zhang Ruonan, et al. Structureflow: Image inpainting via structure-aware appearance flow [C] //Proc of the 13th IEEE Int Conf on Computer Vision (ICCV). Piscataway, NJ: IEEE, 2019 : 181–190
[36]	Xiong Wei, Yu Jiahui, Lin Zhe, et al. Foreground-aware image inpainting[C] //Proc of the 32nd IEEE Conf on Computer Vision and Pattern Recognition (CVPR). Los Alamitos, CA: IEEE Computer Society, 2019: 5840–5848
[37]	Zheng Chuanxia, Cham T, Cai Jiafei. Pluralistic image completion[C] // Proc of the 32nd IEEE Conf on Computer Vision and Pattern Recognition (CVPR). Los Alamitos, CA: IEEE Computer Society, 2019: 1438–1447
[38]	Zhao Lei, Mo Qihang, Lin Sihuan, et al. UCTGAN: Diverse image inpainting based on unsupervised cross-space translation[C] //Proc of the 33rd IEEE Conf on Computer Vision and Pattern Recognition (CVPR). Los Alamitos, CA: IEEE Computer Society, 2020: 5741–5750
[39]	Liao Liang, Xiao Jing, Wang Zheng, et al. Image inpainting guided by coherence priors of semantics and textures [C] //Proc of the 34th IEEE Conf on Computer Vision and Pattern Recognition (CVPR). Los Alamitos, CA: IEEE Computer Society, 2021: 6539−6548
[40]	Liu Hongyu, Jiang Bin, Song Yibing, et al. Rethinking image inpainting via a mutual encoder decoder with feature equalizations [C] //Proc of the 18th European Conf on Computer Vision (ECCV). Berlin: Springer, 2020: 725−741
[41]	Nazeri K, Ng E, Joseph T, et al. EdgeConnect: Generative image inpainting with adversarial edge learning [C] //Proc of the 11th IEEE Int Conf on Computer Vision (ICCV). Piscataway, NJ: IEEE, 2019: 472−480
[42]	Yu Jiahui, Lin Zhe, Yang Jimei, et al. Free-form image inpainting with gated convolution [C] //Proc of the 11th IEEE Int Conf on Computer Vision (ICCV). Piscataway, NJ: IEEE, 2019: 4471−4480
[43]	Yu Fang, Koltun V, Funkhouser T. Dilated residual networks[C] //Proc of the 31st IEEE Conf on Computer Vision and Pattern Recognition (CVPR). Los Alamitos, CA: IEEE Computer Society, 2017: 472−480
[44]	Clevert D, Unterthiner T, Hochreiter S. Fast and accurate deep network learning by exponential linear units (Elus)[J]. arXiv preprint, arXiv: 1511.07289, 2015
[45]	Zhang Han, Goodfellow I, Metaxas D, et al. Self-attention generative adversarial networks [C] //Proc of the 36th Int Conf on Machine Learning (ICML). New York: ACM, 2019: 7354−7363
[46]	Gulrajani I, Ahmed F, Arjovsky M, et al. Improved training of wasserstein GANs[C] // Proc of the 30th Advances in Neural Information Processing Systems (NIPs). La Jolla, CA: NIPS, 2017: 5767−5777
[47]	Arjovsky M, Chintala S, Bottou L. Wasserstein generative adversarial networks [C] //Proc of the 30th Advances in Neural Information Processing Systems (NIPS). La Jolla, CA: NIPS, 2017: 214−223
[48]	Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition[J]. arXiv preprint, arXiv: 1409.1556, 2014
[49]	封顺. 论中国画中的“留白”艺术[J]. 大舞台,2013(6):90−91 Feng Shun. On the art of "leaving blank" in Chinese painting[J]. Grand Stage, 2013(6): 90−91 (in Chinese)
[50]	Arjovsky M, Chintala S, Bottou L. Wasserstein generative adversarial networks[C] //Proc of the 34th Int Conf on Machine Learning (ICML). New York: ACM, 2017: 214−223
[51]	Yu Jiahui, Lin Zhe, Yang Jimeiecc, et al. Generative image inpainting with contextual attention [C] // Proc of the 31st IEEE Conf on Computer Vision and Pattern Recognition (CVPR). Los Alamitos, CA: IEEE Computer Society, 2018: 5505−5514
[52]	Liu Guilin, Reda F A, Shih K J, et al. Image inpainting for irregular holes using partial convolutions [C] //Proc of the 16th European Conf on Computer Vision (ECCV). Berlin: Springer, 2018: 85−100

[1]	Lin Liansheng, Zheng Huanqin, Su Shen, Lei Kai, Chen Xiaofeng, Tian Zhihong. An On-Chain Mechanism Against DeFi Price Manipulation Attacks[J]. Journal of Computer Research and Development, 2025, 62(2): 443-457. DOI: 10.7544/issn1000-1239.202330291
[2]	Song Shuwei, Ni Xiaoze, Chen Ting. Gas Optimization for Smart Contracts: A Survey[J]. Journal of Computer Research and Development, 2023, 60(2): 311-325. DOI: 10.7544/issn1000-1239.202220887
[3]	Ying Chenhao, Xia Fuyuan, Li Jie, Si Xueming, Luo Yuan. Incentive Mechanism Based on Truth Estimation of Private Data for Blockchain-Based Mobile Crowdsensing[J]. Journal of Computer Research and Development, 2022, 59(10): 2212-2232. DOI: 10.7544/issn1000-1239.20220493
[4]	Feng Jingyu, Yang Jinwen, Zhang Ruitong, Zhang Wenbo. A Spectrum Sharing Incentive Scheme Against Location Privacy Leakage in IoT Networks[J]. Journal of Computer Research and Development, 2020, 57(10): 2209-2220. DOI: 10.7544/issn1000-1239.2020.20200453
[5]	Hai Mo, Zhu Jianming. A Propagation Mechanism Combining an Optimal Propagation Path and Incentive in Blockchain Networks[J]. Journal of Computer Research and Development, 2019, 56(6): 1205-1218. DOI: 10.7544/issn1000-1239.2019.20180419
[6]	He Yunhua, Li Mengru, Li Hong, Sun Limin, Xiao Ke, Yang Chao. A Blockchain Based Incentive Mechanism for Crowdsensing Applications[J]. Journal of Computer Research and Development, 2019, 56(3): 544-554. DOI: 10.7544/issn1000-1239.2019.20170670
[7]	He Haiwu, Yan An, Chen Zehua. Survey of Smart Contract Technology and Application Based on Blockchain[J]. Journal of Computer Research and Development, 2018, 55(11): 2452-2466. DOI: 10.7544/issn1000-1239.2018.20170658
[8]	Xiong Jinbo, Ma Rong, Niu Ben, Guo Yunchuan, Lin Li. Privacy Protection Incentive Mechanism Based on User-Union Matching in Mobile Crowdsensing[J]. Journal of Computer Research and Development, 2018, 55(7): 1359-1370. DOI: 10.7544/issn1000-1239.2018.20180080
[9]	Wang Bo, Huang Chuanhe, Yang Wenzhong, Dan Feng, and Xu Liya. An Incentive-Cooperative Forwarding Model Based on Punishment Mechanism in Wireless Ad Hoc Networks[J]. Journal of Computer Research and Development, 2011, 48(3): 398-406.
[10]	Yue Guangxue, Li Renfa, Chen Zhi, Zhou Xu. Analysis of Free-riding Behaviors and Modeling Restrain Mechanisms for Peer-to-Peer Networks[J]. Journal of Computer Research and Development, 2011, 48(3): 382-397.

Cited By

Cited by

Periodical cited type(2)

1.	李硕，王馨爽. 多场景融合的码号数据分发架构及关键技术研究. 数据通信. 2024(06): 1-3+11 .
2.	俞惠芳，李磊. 基于椭圆曲线签密的跨链医疗数据共享方案. 通信学报. 2024(12): 57-66 .