Abstract: As a basic and crucial research issue in the field of machine vision, image semantic segmentation aims to classify every pixel in a color image and predict its corresponding semantic label. Most of the existing semantic segmentation methods are all supervised learning models that are excessively dependent on the given per-pixel annotations. Although existing segmentation methods based on weakly supervision and semi-supervision learning can be integrated into unlabeled samples, semantic category mis-classification often occurs due to the lack of effective utilization of spatial semantic information, and it is difficult to directly apply to other cross-domain unlabeled data sets. In order to solve those problems, this paper proposes a semantic segmentation method based on categories-aware domain adaptation for cross-domain unlabeled data sets. Firstly, the proposed method adopts the optimized upsampling method and proposed a new loss function based on focal loss, which is an effective solution to the problem that it is very difficult to segment the categories with small data volume in the existing methods. Secondly, a categories-aware domain adaptation method is proposed to improve the mIoU of semantic segmentation of unlabeled images of target domain by 6%, compared with the state-of-the-art methods. The proposed method is verified on five data sets, and the experimental results fully demonstrate the effectiveness and generalization of the proposed method.