Abstract: Cultural relics are prone to partial losses of structure or texture due to improper preservation. In order to restore the original image of cultural relics, we propose a new method for restoring cultural relics by using Transformer’s global structure understanding ability to restore the coherent structure of cultural relics and using convolutional neural networks’ local texture understanding ability to restore the delicate texture of cultural relics. To achieve this goal, the restoration work is divided into two steps: the first step is to use Transformer to reconstruct the rough image and restore the coherent structure; the second step is to use a convolutional neural network to enlarge the rough image and restore the fine texture of the missing area. Considering that there is no high-quality, large-scale heritage database in China and abroad, a new heritage image database is also proposed. The experimental results show that the restoration results outperform the current image restoration algorithms in both subjective and objective evaluations in both breakage restoration experiments and large-area breakage restoration experiments that match realistic scenes. At the same time, the proposed method supports diversified output, which provides diverse references for restorers and improves the restoration efficiency.