Abstract: With the advance of Web technology, image sharing has become much easier than ever before. Automatic image annotation, which can predict relevant labels for images, is becoming more and more important. Traditional image annotation methods usually require a large number of complete, accurate labeled data to obtain good annotation performance. However, since obtaining weak labeled training data is often much easier and costs less efforts than obtaining a large amount of fully labeled training data, image labels are often incomplete and inaccurate in real world environment. In addition, different labels usually have large frequency differences. To effectively harness these weakly labeled images, in this paper, an automatic image annotation approach based on semantic neighborhood learning (SNLWL) is proposed. The missing labels are replenished by minimizing the reweighted error functions on training data. Then, the semantic neighborhood is obtained by a progressive neighborhood construction approach. We incorporate label completeness, global similarity, conceptual similarity, and partly correlation into the stage. In addition, an effective label inference strategy is proposed by minimizing the neighborhood reconstruction error to handle the noise in the labels. Extensive experimental results on different benchmark datasets show that the proposed approach makes a marked improvement as compared with other methods.