ISSN 1000-1239 CN 11-1777/TP

• 论文 •

跨媒体相关性推理与检索研究

1. 1(武汉科技大学计算机科学与技术学院 武汉 430081) 2(浙江大学计算机科学与技术学院 杭州 310027) (hong790919@163.com)
• 出版日期: 2008-05-15

Cross Media Correlation Reasoning and Retrieval

Zhang Hong1,2, Wu Fei2, and Zhuang Yueting2

1. 1(College of Computer Science & Technology, Wuhan University of Science & Technology, Wuhan 430081) 2(College of Computer Science and Technology, Zhejiang University, Hangzhou 310027)
• Online: 2008-05-15

Abstract: A cross media retrieval approach is proposed to solve the problem of cross media correlation measuring between different modalities, such as image and audio data. First both intra and crossmedia correlations among multimodality datasets are explored. Intramedia correlation measures the similarity between multimedia data of the same modality, and crossmedia correlation measures how similar in semantic level two multimedia objects of different modalities are. Cross media correlation is very difficult to measure because of the heterogeneity in lowlevel features. For example, images are represented with visual feature vectors and audio clips are represented with heterogeneous auditory feature vectors. Intramedia correlation is calculated based on geodesic distance, and crossmedia correlation is estimated according to link information among WebPages. Then both kinds of correlations are formalized in a crossmedia correlation graph. Based on this graph crossmedia retrieval is enabled by the weight of the shortest path. A unique relevance feedback technique is developed to update the knowledge of multimodal correlations by learning from user behaviors, and to enhance the retrieval performance in a progressive manner. This approach breakthroughs the limitation of modality during retrieval process, and is applicable for querybyexample and crossretrieval multimedia applications. Experiment results on imageaudio dataset are encouraging, and show that the performance of the approach is effective.