Abstract: It is always difficult to ascertain the number of clusters for image segmentation, while in this paper, the problem is solved well by using a method based on the measurement of difference of mutual information. The difference of mutual information describes the increase of mutual information in both original and segmented image when the number of segments is increasing. Being a measure of ascertaining the number of segments, it is considered getting the balance between the number of segments and the mutual information of segmented images. According to it, a rule of determining the number of segments is put forward. For the segmentation algorithm, Gauss-Markov random field model is often used, which takes advantage of both image intensity and spatial information imposed by Gibbs priori probability. The model can be used to effectively segment the noised images. However it is always difficult to confirm the Gibbs penalty factor β. As usual, it requires a tedious trial-and-error process. So to solve this problem, a class-adaptive penalty factor β is defined. It is automatically estimated from the posteriori probability and is anisotropic for each class. Furthermore, the model iteratively gets their parameters estimation in the EM-MAP algorithm. Finally, by the application of this algorithm in medical image segmentation, it is proved effective.