Abstract: Automatic image annotation is an important but highly challenging problem in content-based image retrieval. A new procedure for providing images with semantic keywords is introduced. To over the semantic gap, classified images are used to train a special multi-class classifier based on support vector machine (SVM), which maps the visual image feature into the model space to achieve the concept indexing. The model-vectors that construct the model space are the combination of the multi-class classifier's outputs, and applied to each individual image. Soft labels are then given to the unannotated images during the propagation procedure in the model space, and as keyword, each label is associated with a membership confidence estimated by a biased kernel regression algorithm. Thus conceptualized annotations of images could be provided to users. The empirical study on the COREL image database shows that the proposed model-vectors outperform visual features 14.0% in F-measure for annotation comparatively.