Transformer-Based Image Restoration Method for Cultural Relics

Wang Zhenyan; Jiang Shengcheng; Song Qihong; Liu Bo; Bi Xiuli; Xiao Bin

doi:10.7544/issn1000-1239.202220623

Journal of Computer Research and Development > 2024 > 61(3): 748-761. > DOI: 10.7544/issn1000-1239.202220623 CSTR: 32373.14.issn1000-1239.202220623

Wang Zhenyan, Jiang Shengcheng, Song Qihong, Liu Bo, Bi Xiuli, Xiao Bin. Transformer-Based Image Restoration Method for Cultural Relics[J]. Journal of Computer Research and Development, 2024, 61(3): 748-761. DOI: 10.7544/issn1000-1239.202220623

Citation:

PDF (12678 KB)

Transformer-Based Image Restoration Method for Cultural Relics

Chongqing Key Laboratory of Image Cognition(Chongqing University of Posts and Telecommunications), Chongqing 400065

Funds: This work was supported by the National Natural Science Foundation of China (62172067，61976031), the National Key Research and Development Program of China (2019YFE0110800), the Natural Science Foundation of Chongqing for Distinguished Young Scholars (CSTB2022NSCQ-JQX0001), and the Science and Technology Research Program of Chongqing Municipal Education Commission(KJQN202200635).

More Information

Author Bio:
Wang Zhenyan: born in 1999. Bachelor. His main research interest includes image processing

Jiang Shengcheng: born in 2000. Bachelor. His main research interest includes image processing

Song Qihong: born in 2001. Bachelor. His main research interest includes image processing

Liu Bo: born in 1987. PhD, lecturer. Member of CCF. His main research interests include image processing and multimedia forensics

Bi Xiuli: born in 1982. PhD, associate professor, PhD supervisor. Member of CCF. Her main research interests include image processing and multimedia information security

Xiao Bin: born in 1982. PhD, professor, PhD supervisor. Member of CCF. His main research interests include image processing and pattern recognition
Received Date: July 06, 2022
Revised Date: June 29, 2023
Available Online: November 30, 2023

Graphical Abstract

Abstract

Abstract

Cultural relics are prone to partial losses of structure or texture due to improper preservation. In order to restore the original image of cultural relics, we propose a new method for restoring cultural relics by using Transformer’s global structure understanding ability to restore the coherent structure of cultural relics and using convolutional neural networks’ local texture understanding ability to restore the delicate texture of cultural relics. To achieve this goal, the restoration work is divided into two steps: the first step is to use Transformer to reconstruct the rough image and restore the coherent structure; the second step is to use a convolutional neural network to enlarge the rough image and restore the fine texture of the missing area. Considering that there is no high-quality, large-scale heritage database in China and abroad, a new heritage image database is also proposed. The experimental results show that the restoration results outperform the current image restoration algorithms in both subjective and objective evaluations in both breakage restoration experiments and large-area breakage restoration experiments that match realistic scenes. At the same time, the proposed method supports diversified output, which provides diverse references for restorers and improves the restoration efficiency.
- cultural relic database,
- cultural relic image completion,
- Transformer,
- convolutional neural networks,
- super-resolution,
- virtual restoration

FullText(HTML)

References (40)

References

[1]	Wang Hua, Li Qing, Jia Songtao. A global and local feature weighted method for ancient murals inpainting[J]. International Journal of Machine Learning and Cybernetics, 2020, 11(6): 1197−1216 doi: 10.1007/s13042-019-01032-2
[2]	Chen Hao, Zhao Lu, Wang Zhenyu, et al. DualAST: Dual style-learning networks for artistic style transfer[C]//Proc of the 34th IEEE/CVF Int Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE , 2021: 872−881
[3]	Haliassos A, Barmpoutis P, Stathaki T, et al. Classification and detection of symbols in ancient papyri[C]//Proc of the 5th Visual Computing for Cultural Heritage. Berlin: Springer, 2020 : 121−140
[4]	Opgenhaffen L. The impact of digital technology on archaeological recording strategies and ensuing open research archives[J]. Digital Applications in Archaeology and Cultural Heritage, 2022, 27: 1−15
[5]	陈永,艾亚鹏,郭红光. 改进曲率驱动模型的敦煌壁画修复算法[J]. 计算机辅助设计与图形学学报,2020,32(5):787−796 Chen Yong, Ai Yapeng, Guo Hongguang. Improved curvature driven model for Dunhuang Fresco restoration algorithm[J]. Journal of Computer Aided Design & Computer Graphics, 2020, 32(5): 787−796(in Chinese)
[6]	Criminisi A, Perez P, Toyama K. Region filling and object removal by exemplar-based image inpainting[J]. IEEE Transactions on Image Processing, 2004, 13(9): 1200−1212 doi: 10.1109/TIP.2004.833105
[7]	Barnes C, Shechtman E, Finkelstein A, et al. PatchMatch: A randomized correspondence algorithm for structural image editing[J]. ACM Transactions on Graphics, 2009, 28(3): 24−34
[8]	卢小宝,王维兰. 基于样本块的破损唐卡图像修复算法的改进[J]. 计算机应用,2010,30(4):943−946 doi: 10.3724/SP.J.1087.2010.00943 Lu Xiaobao, Wang Weilan. Improvement of damaged Thangka image repair algorithm based on sample block[J]. Journal of Computer Applications, 2010, 30(4): 943−946(in Chinese) doi: 10.3724/SP.J.1087.2010.00943
[9]	Yao Feng. Damaged region filling by improved Criminisi image inpainting algorithm for Thangka[J]. Cluster Computing, 2019, 22(6): 13683−13691
[10]	Yu Tong, Lin Chuan, Zhang Shuo, et al. Artificial intelligence for dunhuang cultural heritage protection: The project and the dataset[J]. International Journal of Computer Vision, 2022, 130(11): 2646−2673 doi: 10.1007/s11263-022-01665-x
[11]	Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need[C] //Proc of the 31st Int Conf on Neural Information Processing Systems. Cambridge, MA: MIT, 2017: 6000–6010
[12]	Bertalmio M, Sapiro G, Caselles V, et al. Image inpainting[C] //Proc of the 27th Annual Conf on Computer Graphics and Interactive Techniques. New York: ACM, 2000: 417−424
[13]	Efros A A, Freeman W T. Image quilting for texture synthesis and transfer[C] //Proc of the 28th Annual Conf on Computer Graphics and Interactive Techniques. New York: ACM, 2001: 341−346
[14]	Ballester C, Bertalmio M, Caselles V, et al. Filling-in by joint interpolation of vector fields and gray levels[J]. IEEE Transactions on Image Processing, 2001, 10(8): 1200−1211 doi: 10.1109/83.935036
[15]	Darabi S, Shechtman E, Barnes C, et al. Image melding: Combining inconsistent images using patch-based synthesis[J]. ACM Transactions on Graphics, 2012, 31(4): 1−10
[16]	Hays J, Efros A A. Scene completion using millions of photographs[J]. ACM Transactions on Graphics, 2007, 26(3): 4−33 doi: 10.1145/1276377.1276382
[17]	Wang Hua, Li Qing, Zou Qian. Inpainting of Dunhuang murals by sparsely modeling the texture similarity and structure continuity[J]. Journal on Computing and Cultural Heritage, 2019, 12(3): 1−21
[18]	Xu Shoujin, Liu Dong, Xiong Zhihong. E2I: Generative inpainting from edge to image[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2021, 31(4): 1308−1322 doi: 10.1109/TCSVT.2020.3001267
[19]	Pathak D, Krähenbühl P, Donahue J, et al. Context Encoders: Feature learning by inpainting[C] //Proc of the 29th IEEE/CVF Int Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2016: 2536−2544
[20]	Liu Guilin, Reda F A, Shih K J, et al. Image inpainting for irregular holes using partial convolutions[C] //Proc of the 15th IEEE/CVF European Conf on Computer Vision. Piscataway, NJ: IEEE, 2018: 89−105
[21]	Zeng Yuhui, Lin Zhouchen, Lu Hongbin, et al. CR-Fill: Generative image inpainting with auxiliary contextual reconstruction[C] //Proc of the 18th IEEE/CVF Int Conf on Computer Vision. Piscataway, NJ: IEEE 2021: 14144−14153
[22]	Zhang Hui, Qi Yuchao, Xue Xiaoyan, et al. Ancient stone inscription image denoising and inpainting methods based on deep neural networks[J]. Discrete Dynamics in Nature and Society, 2021, 2021: 1−12
[23]	Nazeri K, Ng E, Joseph T, et al. EdgeConnect: Structure guided image inpainting using edge prediction[C] //Proc of the 17th IEEE/CVF Int Conf on Computer Vision Workshop. Piscataway, NJ: IEEE, 2019: 3265−3274
[24]	Yeh R A, Chen C, Lim T Y, et al. Semantic image inpainting with deep generative models[C] //Proc of the 30th IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2017: 6882−6890
[25]	Yu Jiahui, Lin Zhe, Yang Jimei, et al. Generative image inpainting with contextual attention[C] //Proc of the 31st IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2018: 5505−5514
[26]	Yan Zhaoyi, Li Xiaodong, Li Mu, et al. Shift-Net: Image inpainting via deep feature rearrangement[C]//Proc of the 16th IEEE/CVF European Conf on Computer Vision. Piscataway, NJ: IEEE, 2018: 3−19
[27]	Zeng Yijun, Gong Yi, Zeng Xianglong. Controllable digital restoration of ancient paintings using convolutional neural network and nearest neighbor[J]. Pattern Recognition Letters, 2020, 133: 158−164 doi: 10.1016/j.patrec.2020.02.033
[28]	Qin Jie, Bai Honggang, Zhao Yizhou. Multi-scale attention network for image inpainting[J]. Computer Vision and Image Understanding, 2021, 204: 1−12
[29]	曹建芳,李艳飞,崔红艳. 自适应样本块局部搜索算法对古代寺观壁画的修复[J]. 计算机辅助设计与图形学学报,2019,31(11):2030−2037 Cao Jianfang, Li Yanfei, Cui Hongyan, et al. Restoration of ancient temple murals by adaptive sample block local search algorithm[J]. Journal of Computer Aided Design & Computer Graphics, 2019, 31(11): 2030−2037(in Chinese)
[30]	李清泉,王欢,邹勤. 一种基于稀疏表示模型的壁画修复算法[J]. 武汉大学学报:信息科学版,2018,43(12):1847−1853 Li Qingquan, Wang Huan, Zou Qin. A mural restoration algorithm based on sparse representation model[J]. Geomatics and Information Science of Wuhan University, 2018, 43(12): 1847−1853(in Chinese)
[31]	Chen M, Radford A, Child R, et al. Generative pretraining from pixels[C] //Proc of the 37th Int Conf on Machine Learning. New York: ACM, 2020: 1691−1703
[32]	Parmar N, Vaswani A, Uszkoreit J, et al. Image Transformer[C] //Proc of the 35th Int Conf on Machine Learning. New York: ACM, 2018: 4055−4064
[33]	Wan Zijian, Zhang Jing, Chen Dongdong, et al. High-fidelity pluralistic image completion with transformers[C] //Proc of the 18th IEEE/CVF Int Conf on Computer Vision. Piscataway, NJ: IEEE 2021: 4672−4681
[34]	Yang Yuting, Jiao Licheng, Liu Xu, et al. Transformers meet visual learning understanding: A Comprehensive Review[J]. arXiv preprint, arXiv: 2203. 12944, 2022
[35]	Radford A, Wu J, Child R, et al. Language models are unsupervised multitask learners[C/OL]. 2019[2021-04-17].https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf
[36]	Devlin J, Chang M W, Lee K, et al. BERT: Pre-training of deep bidirectional transformers for language understanding[C] //Proc of the 17th Conf of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Stroudsburg, PA: ACL, 2019: 4171−4186
[37]	Ledig C, Theis L, Huszár F, et al. Photo-realistic single image super-resolution using a generative adversarial network[C] //Proc of the 30th IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2017: 105−114
[38]	Rudin L I, Osher S, Fatemi E. Nonlinear total variation based noise removal algorithms[J]. Physica D: Nonlinear Phenomena, 1992, 60(1): 259−268
[39]	Zhang R, Isola P, Efros A A, et al. The unreasonable effectiveness of deep features as a perceptual metric[C] //Proc of the 31st IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2018: 586−595
[40]	Yu Jiahui, Lin Zhe, Yang Jimei, et al. Free-form image inpainting with gated convolution[C] //Proc of the 17th IEEE/CVF Int Conf on Computer Vision. Piscataway, NJ: IEEE, 2019: 4471−4480

[1]	Wang Chenze, Shen Xuehao, Huang Zhenli, Wang Zhengxia. Interactive Visualization Framework for Panoramic Super-Resolution Images Based on Localization Data[J]. Journal of Computer Research and Development, 2024, 61(7): 1741-1753. DOI: 10.7544/issn1000-1239.202330643
[2]	Fan Wei, Liu Yong. Social Network Information Diffusion Prediction Based on Spatial-Temporal Transformer[J]. Journal of Computer Research and Development, 2022, 59(8): 1757-1769. DOI: 10.7544/issn1000-1239.20220064
[3]	Zhou Weilin, Yang Yuan, Xu Mingwei. Network Function Virtualization Technology Research[J]. Journal of Computer Research and Development, 2018, 55(4): 675-688. DOI: 10.7544/issn1000-1239.2018.20170937
[4]	Yang Shuaifeng, Zhao Ruizhen. Image Super-Resolution Reconstruction Based on Low-Rank Matrix and Dictionary Learning[J]. Journal of Computer Research and Development, 2016, 53(4): 884-891. DOI: 10.7544/issn1000-1239.2016.20140726
[5]	Dou Nuo, Zhao Ruizhen, Cen Yigang, Hu Shaohai, Zhang Yongdong. Noisy Image Super-Resolution Reconstruction Based on Sparse Representation[J]. Journal of Computer Research and Development, 2015, 52(4): 943-951. DOI: 10.7544/issn1000-1239.2015.20140047
[6]	Yang Xin, Zhou Dake, Fei Shumin. A Self-Adapting Bilateral Total Variation Technology for Image Super-Resolution Reconstruction[J]. Journal of Computer Research and Development, 2012, 49(12): 2696-2701.
[7]	Wang Kai, Hou Zifeng. A Relaxed Co-Scheduling Method of Virtual CPUs on Xen Virtual Machines[J]. Journal of Computer Research and Development, 2012, 49(1): 118-127.
[8]	Wang Dan, Feng Dengguo, and Xu Zhen. An Approach to Data Sealing Based on Trusted Virtualization Platform[J]. Journal of Computer Research and Development, 2009, 46(8): 1325-1333.
[9]	Xiao Chuangbai, Yu Jing, Xue Yi. A Novel Fast Algorithm for MAP Super-Resolution Image Reconstruction[J]. Journal of Computer Research and Development, 2009, 46(5): 872-880.
[10]	Huang Hua, Fan Xin, Qi Chun, and Zhu Shihua. Face Image Super-Resolution Reconstruction Based on Recognition and Projection onto Convex Sets[J]. Journal of Computer Research and Development, 2005, 42(10): 1718-1725.