Abstract: In image retrieval and annotation systems, multiple instance learning (MIL) has been studied actively. Since each image contains several regions and each region can be regarded as an instance, the image retrieval is then transformed into a MIL problem. The key assumption of MIL is that: a bag is positive if at least one of its instances is a positive example; otherwise, the bag is negative. In the setting of MIL, each image is viewed as a bag of semantic regions. Most of the state-of-the-art methods solve the MIL problem in a supervised way. However, two unsupervised frameworks for clustering multi-instance objects based on expectation maximization (EM) approach and iterative heuristic optimization are proposed respectively. Under each framework, three new algorithms are introduced to find users’ interests on specific Web images without any manual labeled data. The EM approach takes instances as members of concepts. Each concept is modeled by a statistical process. Then a cluster of MI objects is considered as a multinomial distribution over the components of the mixture model of instances. The other framework is based on the idea of iterative heuristic optimization. It selects an instance from each MI object in every iteration process to determine the clustering model of MI objects. Hence it transforms the multi-instance object clustering problem into a normal clustering problem. Furthermore, all the algorithms are evaluated on both the MUSK benchmark data sets and a real-world Web image dataset downloaded from Yahoo. And comparative studies show the effectiveness of the proposed algorithms.