Abstract: Many researches have shown that Gabor feature is robust to illumination and expression variations and has been successfully used in face recognition. However, Gabor-based methods suffer from the high dimensional problem, also known as the curse of dimensionality. A novel method for face description and recognition based on Gabor filters is proposed in this paper. The normalized face image is first decomposed by convolving with multi-scale and multi-orientation Gabor filters, and then separated into several blocks. Through statistical techniques, including mean and variance of Gabor coefficients inside each block, the block feature vector (BFV) can be obtained. The face feature vector (FFV) of the whole image is then constructed by conjugating the BFVs in row column order. Thus FFV can be used to describe the face, and the high dimensional problem is effectively solved. In the classification stage, a robust face recognition system which performs multi-class classification is built based on several two-class classifiers using voting mechanism. For each two-class classifier, a feature extraction module adaptively selects the most important features. Therefore, only the most distinguishable features in FFV are picked out, called best subset feature vector (BSFV). The results compared with the published results on Yale face database verify the validity of the proposed method.