Survey of Automatic Removal of Moiré Pattern

Qi Wenfa; Liu Yuxin; Guo Zongming

doi:10.7544/issn1000-1239.202220797

Journal of Computer Research and Development > 2024 > 61(3): 728-747. > DOI: 10.7544/issn1000-1239.202220797 CSTR: 32373.14.issn1000-1239.202220797

Qi Wenfa, Liu Yuxin, Guo Zongming. Survey of Automatic Removal of Moiré Pattern[J]. Journal of Computer Research and Development, 2024, 61(3): 728-747. DOI: 10.7544/issn1000-1239.202220797

Citation:

Qi Wenfa, Liu Yuxin, Guo Zongming. Survey of Automatic Removal of Moiré Pattern[J]. Journal of Computer Research and Development, 2024, 61(3): 728-747. DOI: 10.7544/issn1000-1239.202220797

Citation:

Qi Wenfa, Liu Yuxin, Guo Zongming. Survey of Automatic Removal of Moiré Pattern[J]. Journal of Computer Research and Development, 2024, 61(3): 728-747. DOI: 10.7544/issn1000-1239.202220797

PDF (6628 KB)

Survey of Automatic Removal of Moiré Pattern

Wangxuan Institute of Computer Technology, Peking University, Beijing 100080

Funds: This work was supported by the National Natural Science Foundation of China (U1636206) and the Key Laboratory of Science, Technology and Standard in Press Industry (the Key Laboratory of Intelligent Press Media Technology).

More Information

Author Bio:
Qi Wenfa: born in 1978. PhD, assistant professor. Senior member of CCF. His main research interests include information hiding, image processing, machine learning

Liu Yuxin: born in 1985. Master, engineer. Member of CCF. His main research interests include information hiding, image processing, and machine learning

Guo Zongming: born in 1966. PhD, professor. Distinguished member of CCF. His main research interests include network video technology, image processing, and information security
Received Date: September 05, 2022
Revised Date: May 14, 2023
Available Online: November 30, 2023

Graphical Abstract

Abstract

Abstract

Nowadays, digital cameras and smart phones have played an increasingly important role in daily lives, and have become the main tools for people to perceive the world, record information and communicate with each other. When these devices are used to shoot electronic screens, the irregular Moiré patterns in the image are produced due to the overlap of the display devices and digital grids of the camera sensor, which seriously affects the visual quality of the captured images. Therefore, the removal of Moiré pattern is of great significance for the post-processing of captured images. In this paper, recent research about Moiré removal is reviewed in detail, and the existing methods are classified into two categories according to different application scenarios and technical implementations: prior knowledge based Moiré removal and deep learning based methods respectively. According to different training data acquisition and alignment methods, deep learning based Moiré removal techniques can be divided into convolutional neural network (CNN) based methods and generative adversarial network (GAN) based methods. Then, based on the same public dataset, we implement the mainstream deep learning based Moiré removal algorithms, compare and analyze their performance, and summarize the advantages and disadvantages of various methods. Finally, possible directions for future research are discussed.
- Moiré pattern,
- image restoration,
- multiscale,
- deep learning,
- convolutional neural network,
- generative adversarial networks

FullText(HTML)

References (75)

References

[1]	Liu Shuai, Li Chenghua, Nan Nan, et al. MMDM: Multi-frame and multi-scale for image deMoiréing[C] //Proc of IEEE/CVF Conf on Computer Vision and Pattern Recognition Workshops (CVPRW). Piscataway, NJ: IEEE, 2020: 1751−1759
[2]	Niu Wenjia, Zhang Kaihao, Luo Wenhan, et al. Blind motion deblurring super-resolution: When dynamic spatio-temporal learning meets static image understanding[J]. IEEE Transactions on Image Processing, 2021, 30(8): 7101−7111
[3]	Ronneberger O, Fischer P, Brox T. U-net: Convolutional networks for biomedical image segmentation[C] //Proc of the 18th 2015 Medical Image Computing and Computer-Assisted Intervention. Berlin: Springer, 2015: 234–241
[4]	Wang Fuping, Hu Yang, Liu Weihua, et al. Face impainting algorithm combining face sketch and gate convolution[C]//Proc of 4th Int Conf on Natural Language Processing(ICNLP). Piscataway, NJ: IEEE, 2022: 81−86
[5]	蒋斌,刘虹雨,杨超,等. 一种基于局部属性生成对抗网络的人脸修复算法[J]. 计算机研究与发展,2019,56(11):2485−2493 Jiang Bin, Liu Hongyu, Yang Chao, et al. A face inpainting algorithm with local attribute generative adversarial networks[J]. Journal of Computer Research and Development, 2019, 56(11): 2485−2493(in Chinese)
[6]	Gao Tianyu, Guo Yanqing, Zheng Xin, et al. Moiré pattern removal with multi-scale feature enhancing network[C] //Proc of IEEE Int Conf on Multimedia and Expo Workshops (ICMEW). Piscataway, NJ: IEEE, 2019: 240−245
[7]	Elad M, Aharon M. Image denoising via sparse and redundant representations over learned dictionaries[J]. IEEE Transactions on Image Processing, 2006, 15(12): 3736−3745 doi: 10.1109/TIP.2006.881969
[8]	Chen Xiaogang, Kang Singbing, Yang Jie, et al. Fast patch-based denoising using approximated patch geodesic paths[C] //Proc of IEEE Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2013: 1211–1218
[9]	Li Xu, Qiong Yan, Yang Xia, et al. Structure extraction from texture via relative total variation[J]. ACM Transactions on Graphics, 2012, 31(6): 139−148
[10]	Shou Yuwen, Lin Chinteng. Image descreening by GA-CNN-based texture classification[J]. IEEE Transactions on Circuits and Systems I: Regular Papers, 2004, 51(11): 2287−2299 doi: 10.1109/TCSI.2004.836861
[11]	Nah S, Kim T H, Lee K M, Deep multi-scale convolutional neural network for dynamic scene deblurring[C] //Proc of IEEE Conf on Computer Vision and Pattern Recognition (CVPR). Piscataway, NJ: IEEE, 2017: 3883–3891
[12]	Nishioka K, Hasegawa N, Ono K, et al. Endoscope system provided with low-pass filter for Moiré removal: US, US08/917429[P]. 2000-02-15
[13]	Schoberl M, Schnurrer W, Oberdorster A, et al. Dimensioning of optical birefringent anti-alias filters for digital cameras[C] //Proc of IEEE Int Conf on Image Process (ICIP). Piscataway, NJ: IEEE, 2010: 4305–4308
[14]	Menon D, Calvagno G. Color image demosaicking: An overview[J]. Signal Processing: Image Communication, 2011, 26(8/9): 518−533
[15]	Pekkucuksen I, Altunbasak Y. Multiscale gradients-based color filter array interpolation[J]. IEEE Transactions on Image Processing, 2013, 22(1): 157−165 doi: 10.1109/TIP.2012.2210726
[16]	邱菊. 基于插值算法的去彩色摩尔纹效应的研究[D]. 南京:南京理工大学,2009 Qiu Ju. Research on color Moiré removal based on interpolation algorithm [D]. Nanjing: Nanjing University of Science and Technology, 2009 (in Chinese)
[17]	邱香香. 基于CCD静态图像的摩尔纹去除算法研究[D]. 南京:南京理工大学,2012 Qiu Xiangxiang. Research on Moiré removal algorithm based on CCD static image[D]. Nanjing: Nanjing University of Science and Technology, 2012 (in Chinese)
[18]	Hibbard R H. Apparatus and method for adaptively interpolating a full color image utilizing luminance gradients: US, US08/085520[P]. 1995-01-17
[19]	Hamilton J F. Adaptive color plane interpolation in single sensor color electronic camera: US, US08/407423[P]. 1997-05-13
[20]	Kimmel R. Demosaicing: Image reconstruction from color CCD samples[J]. IEEE Transactions on Image Processing, 1999, 9(8): 1221−1228
[21]	刘芳蕾. 基于图像分解的纹理图像摩尔纹消除方法[D]. 天津:天津大学,2017 Liu Fanglei. Moiré pattern removal from Textured images via image decomposition[D]. Tianjin : Tianjin University, 2017 (in Chinese)
[22]	张雪. 基于图像分解的屏摄图像摩尔纹消除方法[D]. 天津:天津大学,2018 Zhang Xue. Moiré pattern removal from scree-shot images via image decomposition[D]. Tianjin : Tianjin University, 2018 (in Chinese)
[23]	Liu Fanglei, Yang Jingyu, Yue Huanjing. Moiré pattern removal from texture images via low-rank and sparse matrix decomposition[C/OL] //Proc of IEEE Int Conf on Visual Communications and Image Processing (VCIP). Piscataway, NJ: IEEE, 2015[2016-08-25].https:// ieeexplore. ieee. org/stamp/stamp. jsp?tp=&arnumber=7457907
[24]	Yang Jingyu, Zhang Xue, Cai Changrui, et al. DeMoiréing for screen-shot images with multi-channel layer decomposition[C/OL] //Proc of IEEE Visual Communications and Image Processing (VCIP). Piscataway, NJ: IEEE, 2017[2018-03-01].https://ieeexplore.ieee.org/stamp/stamp.jsp?tp =& arnumber=8305057
[25]	Fang Faming, Wang Tingting, Wu Shuyan, et al. Removing Moiré patterns from single images[J]. Information Sciences, 2020, 514(8): 56−71
[26]	Sasada R, Yamada M, Hara S, et al. Stationary grid pattern removal using 2D technique for Moiré-free radiographic image display[C] //Proc of Conf on Medical Imaging: Visualization, Image-Guided Procedures, and Display. Bellingham, WA: SPIE, 2003: 688–697
[27]	Sidorov D N, Kokaram A C, Suppression of Moiré patterns via spectral analysis[C] //Proc of Visual Communications and Image Processing. Bellingham, WA: SPIE, 2002: 895–906
[28]	Wei Zhouping, Wang Jian, Nichol H, et al. A median-Gaussian filtering framework for Moiré pattern noise removal from X-ray microscopy image[J]. Micron, 2012, 43(2/3): 170−176
[29]	Hazavei S M, Shahdoosti H R. A new method for removing the Moiré pattern from images[J]. arXiv preprint, arXiv: 1701. 09037, 2017
[30]	Garcia D C, Queiroz R L. Face-spoofing 2D-detection based on Moiré-pattern analysis[J]. IEEE Transactions on Information Forensics and Security, 2015, 10(4): 778−786
[31]	Sidorov D N, Kokaram A C. Removing Moiré from degraded video archives[C/OL] //Proc of the 11th European Signal Processing Conf. Piscataway, NJ: IEEE , 2002[2015-03-30].https://ieeexplore.ieee.org/ stamp/stamp.jsp?tp=&arnumber=7071920
[32]	吕旭,胡柏青,李开龙,等. 非线性目标跟踪滤波算法的比较研究[J]. 火力与指挥控制,2021,46(4):24−30 Lü Xu, Hu Baiqing, Li Kailong, et al. Comparison of nonlinear target tracking filtering algorithm[J]. Fire Control & Command Control, 2021, 46(4): 24−30(in Chinese)
[33]	Sun Bin, Li Shutao, Sun Jun. Scanned image descreening with image redundancy and adaptive filtering[J]. IEEE Transactions on Image Processing, 2014, 23(8): 3698−3710
[34]	Dong Chao, Loy Chenchang, He Kaiming, et al. Learning a deep convolutional network for image super-resolution[C] //Proc of the 13th European Conf on Computer Vision. Berlin: Springer, 2014: 184–199
[35]	Kim J, Lee J K, Lee K M, Convolutional networks for biomedical using very deep convolutional networks[C] //Proc of IEEE Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2016: 1646–1654
[36]	Gharbi M, Chaurasia G, Paris S, et al, Deep joint demosaicking and denoising[J], ACM Transaction of Graphics, 2016, 35(6): 1−12
[37]	Zhang Kai, Zuo Wangmeng, Chen Yujin, et al. Beyond a Gaussian denoiser: Residual learning of deep CNN for image denoising[J]. IEEE Transactions on Image Processing, 2017, 26(7): 3142−3155 doi: 10.1109/TIP.2017.2662206
[38]	Zhang Kai, Zuo Wangmeng, Chen Yujin, et al. Learning deep CNN denoiser prior for image restoration[C] //Proc of IEEE Conf on Computer Vision and Pattern Recognition (CVPR). Piscataway, NJ: IEEE, 2017: 2808–2817
[39]	Abraham E. Moiré pattern detection using wavelet decomposition and convolutional neural network[C] //Proc of IEEE Symp Series on Computational Intelligence (SSCI). Piscataway, NJ: IEEE, 2018: 1275−1279
[40]	Sun Yujing, Yu Yizhou, Wang Wenping. Moiré photo restoration using multiresolution convolutional neural networks[J]. IEEE Transactions on Image Processing, 2018, 27(8): 4160−4172 doi: 10.1109/TIP.2018.2834737
[41]	Liu Bolin, Shu Xiao, Wu Xiaolin. DeMoiréing of camera-captured screen images using deep convolutional neural network[J]. arXiv preprint, arXiv: 1804. 03809, 2018
[42]	Gao Tianyu, Guo Yanqing, Zheng Xin, et al. Moiré pattern removal with multi-scale feature enhancing network[C] //Proc of IEEE Int Conf on Multimedia and Expo Workshops. Piscataway, NJ: IEEE, 2019: 240−245
[43]	He Bin, Wang Ce, Shi Boxin, et al. Mop Moiré patterns using mopnet[C] //Proc of IEEE/CVF Int Conf on Computer Vision. Piscataway, NJ: IEEE, 2019: 2424−2432
[44]	Yang Shanghui, Lei Yajing, Xiong Shuangyu, et al. High resolution deMoiré network[C] //Proc of IEEE Int Conf on Image Processing (ICIP). Piscataway, NJ: IEEE, 2020: 888-892
[45]	Cheng Xi, Fu Zhenyong, Yang Jian. Multi-scale dynamic feature encoding network for image deMoiréing[C] //Proc of IEEE/CVF Int Conf on Computer Vision Workshop (ICCVW). Piscataway, NJ: IEEE, 2019: 3486-3493
[46]	Cheng Xi, Fu Zhenyong, Yang Jian. Improved multi-scale dynamic feature encoding network for image deMoiréing[J]. Pattern Recognition, 2021, 116(1): 1−12
[47]	Zheng Bolun, Yuan Shanxi, Slabaugh G, et al. Image deMoiréing with learnable bandpass filters[C] //Proc of IEEE/CVF Conf on Computer Vision and Pattern Recognition (CVPR). Piscataway, NJ: IEEE, 2020: 3636–3645
[48]	Park H, Vien A G, Koh Y J, et al. Unpaired image deMoiréing based on cyclic Moiré learning[C] //Proc of Asia–Pacific Signal Information Processing Association Annual Summit and Conf. Piscataway, NJ: IEEE, 2021: 146–150
[49]	Yue Huanjin, Cheng Yijia, Liu Fanglong, et al. Unsupervised Moiré pattern removal for recaptured screen images[J]. Neurocomputing, 2021, 456: 352−363 doi: 10.1016/j.neucom.2021.05.099
[50]	Park H, Vien A G, Kim H, et al. Unpaired screen-shot image deMoiréing with cyclic Moiré learning[J]. IEEE Access,2022,10:16254-16268
[51]	Yang Jingyu, Liu Fanglei, Yue Huanjing, et al. Textured image deMoiréing via signal decomposition and guided filtering[J]. IEEE Transactions on Image Processing, 2017, 26(7): 3528−3541
[52]	Ok J, Youn S, Seo G, et al. Paper check image quality enhancement with Moiré reduction[J]. Multimedia Tools and Applications, 2017, 76(20): 21423−21450 doi: 10.1007/s11042-016-4080-0
[53]	Ronneberger O, Fischer P, Brox T. U-net: Convolutional networks for biomedical image segmentation[J]. arXiv preprint, arXiv: 1505. 04597v1, 2015
[54]	Gao Huang, Zhuang Liu, Laurens V D M, et al. Densely connected convolutional networks[J]. arXiv preprint, arXiv: 1608. 06993v5, 2018
[55]	Hu Jie, Shen Li, Sun Gang. Squeeze-and-excitation networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020, 42(8): 2011−2023 doi: 10.1109/TPAMI.2019.2913372
[56]	Huang Xun, Belongie S. Arbitrary style transfer in real-time with adaptive instance normalization[C] //Proc of IEEE Int Conf on Computer Vision (ICCV). Piscataway, NJ: IEEE, 2017: 1501–1510
[57]	Vien A G, Park H, Lee C. Dual-domain deep convolutional neural networks for image deMoiréing[C] //Proc of IEEE/CVF Conf on Computer Vision and Pattern Recognition Workshops (CVPRW). Piscataway, NJ: IEEE, 2020: 1934−1942
[58]	Vien A G, Park H, Lee C. Moiré artifacts removal in screen-shot images via multiple domain learning[C] //Proc of Asia-Pacific Signal and Information Processing Association Annual Summit and Conf (APSIPA ASC). Piscataway, NJ: IEEE, 2020: 1268−1273
[59]	Wang Hailing, Tian Qiaoyu, Li Liang, et al. Image deMoiréing with a dual domain distilling network[C/OL] //Proc of IEEE Int Conf on Multimedia and Expo (ICME). Piscataway, NJ: IEEE, 2021[2021-06-09]. https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=9428091
[60]	Zheng Bolun, Yuan Shanxi, Yan Chengganng, et al. Learning frequency domain priors for image deMoiréing[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021, 44(11): 7705−7717
[61]	Liu Lin, Liu Jianzhuang, Yuan Shanxin, et al. Wavelet-based dual-branch network for image deMoiréing[J]. arXiv preprint, arXiv: 2007. 07173v2, 2020
[62]	Gao Huang, Zhuang Liu, Laurens M, et al. Densely connected convolutional networks[C] //Proc of IEEE Conf on Computer Vision and Pattern Recognition (CVPR). Piscataway, NJ: IEEE, 2017, 2261−2269
[63]	Yu F, Koltun V. Multi-scale context aggregation by dilated convolutions[J]. arXiv preprint, arXiv: 1511. 07122, 2016
[64]	Zheng Yuanjie, Yu Jingyi, Kang Singbing, et al. Single-image vignetting correction using radial gradient symmetry[C/OL] //Proc of IEEE Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2008[2008-08-05].https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=& arnumber =4587413
[65]	Dai Tao, Cai Jianrui, Zhang Yongbing, et al. Second-order attention network for single image super-resolution[C] //Proc of IEEE/CVF Conf on Computer Vision and Pattern Recognition (CVPR). Piscataway, NJ: IEEE, 2019: 11065–11074
[66]	Mao Xudong, Li Qing, Xie Haoran, et al. Least squares generative adversarial networks[C] //Proc of IEEE Int Conf on Computer Vision (ICCV). Piscataway, NJ: IEEE, 2017: 2794–2802
[67]	Yue Huanjing, Mao Yan, Liang Lipu, et al. Recaptured screen image deMoiréing[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2021, 31(1): 49−60 doi: 10.1109/TCSVT.2020.2969984
[68]	Yuan Shanxin, Timofte R, Gregory S, et al. Aim 2019 challenge on image demoreing: Dataset and study[C] //Proc of IEEE Int Conf on Computer Vision (ICCV) Workshops. Piscataway, NJ: IEEE, 2019: 3526–3533
[69]	Yuan Shanxin, Timofte R, Leonardis A, et al. NTIRE 2020 challenge on image demoireing: Methods and results[C] //Proc of IEEE/CVF Conf on Computer Vision and Pattern Recognition Workshops (CVPRW). Piscataway, NJ: IEEE, 2020: 1882-1893
[70]	Lowe D G. Distinctive image features from scale-invariant keypoints[J]. International Journal of Computer Vision, 2004, 60(2): 91–110
[71]	Chin T J, Yu J, Suter D. Accelerated hypothesis generation for multi-structure robust fitting[C] //Proc of the 11th European Conf on Computer Vision. Berlin: Springer, 2010: 533–546
[72]	高天宇. 面向移动终端的摩尔纹去除系统的设计与实现[D]. 大连:大连理工大学,2021 Gao Tianyu. Design and implementation of Moiré removal system for mobile terminals[D]. Dalian: Dalian University of Technology, 2021 (in Chinese)
[73]	Kaur R, Sharma D V. Pre-processing for improving the recognition of mobile based Gurmukhi text recognition system[C/OL] //Proc of the 12th Int Conf on Computing Communication and Networking Technologies (ICCCNT). Piscataway, NJ: IEEE, 2021[2021-11-03]. http:// ieeexplore.ieee.org/stamp/ stamp.jsp?tp=&arnumber=9579867
[74]	Fang Han, Zhang Weiming, Zhou Huang, et al. Screen-shooting resilient watermarking[J]. IEEE Transactions on Information Forensics and Security, 2019, 14(6): 1403−1418 doi: 10.1109/TIFS.2018.2878541
[75]	Howard A, Sandler M, Chen Bo, et al. Searching for mobilenetv3 [C] //Proc of IEEE/CVF Int Conf on Computer Vision. Piscataway, NJ: IEEE, 2019: 1314−1324

[1]	Li Nan, Ding Yidong, Jiang Haoyu, Niu Jiafei, Yi Ping. Jailbreak Attack for Large Language Models: A Survey[J]. Journal of Computer Research and Development, 2024, 61(5): 1156-1181. DOI: 10.7544/issn1000-1239.202330962
[2]	Chen Xuanting, Ye Junjie, Zu Can, Xu Nuo, Gui Tao, Zhang Qi. Robustness of GPT Large Language Models on Natural Language Processing Tasks[J]. Journal of Computer Research and Development, 2024, 61(5): 1128-1142. DOI: 10.7544/issn1000-1239.202330801
[3]	Shu Wentao, Li Ruixiao, Sun Tianxiang, Huang Xuanjing, Qiu Xipeng. Large Language Models: Principles, Implementation, and Progress[J]. Journal of Computer Research and Development, 2024, 61(2): 351-361. DOI: 10.7544/issn1000-1239.202330303
[4]	Yang Yi, Li Ying, Chen Kai. Vulnerability Detection Methods Based on Natural Language Processing[J]. Journal of Computer Research and Development, 2022, 59(12): 2649-2666. DOI: 10.7544/issn1000-1239.20210627
[5]	Pan Xuan, Xu Sihan, Cai Xiangrui, Wen Yanlong, Yuan Xiaojie. Survey on Deep Learning Based Natural Language Interface to Database[J]. Journal of Computer Research and Development, 2021, 58(9): 1925-1950. DOI: 10.7544/issn1000-1239.2021.20200209
[6]	Zheng Haibin, Chen Jinyin, Zhang Yan, Zhang Xuhong, Ge Chunpeng, Liu Zhe, Ouyang Yike, Ji Shouling. Survey of Adversarial Attack, Defense and Robustness Analysis for Natural Language Processing[J]. Journal of Computer Research and Development, 2021, 58(8): 1727-1750. DOI: 10.7544/issn1000-1239.2021.20210304
[7]	Pan Xudong, Zhang Mi, Yan Yifan, Lu Yifan, Yang Min. Evaluating Privacy Risks of Deep Learning Based General-Purpose Language Models[J]. Journal of Computer Research and Development, 2021, 58(5): 1092-1105. DOI: 10.7544/issn1000-1239.2021.20200908
[8]	Bao Yang, Yang Zhibin, Yang Yongqiang, Xie Jian, Zhou Yong, Yue Tao, Huang Zhiqiu, Guo Peng. An Automated Approach to Generate SysML Models from Restricted Natural Language Requirements in Chinese[J]. Journal of Computer Research and Development, 2021, 58(4): 706-730. DOI: 10.7544/issn1000-1239.2021.20200757
[9]	Yu Kai, Jia Lei, Chen Yuqiang, and Xu Wei. Deep Learning: Yesterday, Today, and Tomorrow[J]. Journal of Computer Research and Development, 2013, 50(9): 1799-1804.
[10]	Che Haiyan, Feng Tie, Zhang Jiachen, Chen Wei, and Li Dali. Automatic Knowledge Extraction from Chinese Natural Language Documents[J]. Journal of Computer Research and Development, 2013, 50(4): 834-842.