Abstract: As one of the most important parts of human-computer interaction, the research and implementation of sign language recognition (SLR) has significant academic value as well as broad application prospect. In this paper, a framework of tied-mixture hidden Markov models (TMHMM) is proposed for vision-based SLR. TMHMM can efficiently speed up the vision-based SLR without significant loss of recognition accuracy compared with the continuous hidden Markov models (CHMM) due to its excellent properties, i.e., the modeling resolution of TMHMM approximates that of CHMM and the computation cost of TMHMM is greatly reduced by tying similar Gaussian mixture components. For the sign feature extraction, an effective hierarchical feature characterization scheme is proposed, which is more suitable to realize medium or larger vocabulary SLR, through employing principal component analysis to characterize the finger area distribution feature more elaborately. Further, by integrating the techniques of robust hands detection, background subtraction and pupils detection to extract the feature information precisely with the aid of simple colored gloves against the unconstrained background, a medium vocabulary vision-based recognition system for Chinese sign language (CSL) is implemented. Experimental results show that the proposed methods can work well for the medium vocabulary CSL recognition in the environment without special constraints.