Cross Media Correlation Reasoning and Retrieval

A cross media retrieval approach is proposed to solve the problem of cross media correlation measuring between different modalities, such as image and audio data. First both intra and crossmedia correlations among multimodality datasets are explored. Intramedia correlation measures the similarity between multimedia data of the same modality, and crossmedia correlation measures how similar in semantic level two multimedia objects of different modalities are. Cross media correlation is very difficult to measure because of the heterogeneity in lowlevel features. For example, images are represented with visual feature vectors and audio clips are represented with heterogeneous auditory feature vectors. Intramedia correlation is calculated based on geodesic distance, and crossmedia correlation is estimated according to link information among WebPages. Then both kinds of correlations are formalized in a crossmedia correlation graph. Based on this graph crossmedia retrieval is enabled by the weight of the shortest path. A unique relevance feedback technique is developed to update the knowledge of multimodal correlations by learning from user behaviors, and to enhance the retrieval performance in a progressive manner. This approach breakthroughs the limitation of modality during retrieval process, and is applicable for querybyexample and crossretrieval multimedia applications. Experiment results on imageaudio dataset are encouraging, and show that the performance of the approach is effective.