Abstract: Existing deep interactive image segmentation algorithms calculate distance maps or Gaussian maps for the click-annotations, and then concatenate them with the image as the input of network. The influence range of each click is the same, but the purpose of each interaction is different. The main role of the early interaction is selection, and the later prefers fine-tuning. To this end, a deep interactive image segmentation algorithm fused with multi-scale label information is proposed. First, by setting different Gaussian radius, two groups of Gaussian maps with different scales are calculated for each click. Secondly, by fusing with the small scale Gaussian maps, and some down-sampling modules in the basic segmentation network are removed, hence richer detailed features of targets are extracted. At the same time, in order to maintain the integrity of the target segmentation results, a non-local feature attention module is proposed and this module fuses large scale Gaussian maps. Finally, according to the probability information provided by the Gaussian map, a probability click loss is proposed to enhance the segmentation performance of the target near the click point. Experimental results show that the proposed algorithm can not only maintain the integrity of the segmentation, but also obtain the segmentation results of the target details, which greatly reduces the user’s interaction burden.