Effective Property and Best Combination of Classifier Linear Combination

Several classifiers are usually combined to promote the precision of classification in machine learning. The effectiveness of the combination is proved by the weak learning theory. The linear combination of classifiers, called weighted voting, is one of the most common combination methods. The widely-used AdaBoost and Bagging adopt weighted voting methods. The effectiveness of classifier combination and the problem of best combination both have to be solved. The coefficient selection condition for the effectiveness of classifier combination and the coefficient formula of best combination problem are given when there are many classifiers and every classifier is not relevant to other classifiers. The error of combined combination classifier is analyzed. It is concluded that the classification error rate drops exponentially with the increase of classifiers even simple voting method is adopted when the classification error rate of every classifier has unified boundary. Based on this conclusion, according to AdaBoost, some new integrated learning algorithms are proposed. One of them is to directly and rapidly promote the classification precision of the combined classifier. The reasonableness and scienctific nature of this algorithm are analyzed. It is the extension of traditional classifier trading and selecting method to minimize the classification error rate. It is proved that the combination in AdaBoost is efficient and sometimes is the best combination. A classifier combination theory and conclusion on multi-classification problem are given, which are similar to that on two-class classification problem, including effective condition, best combination, error estimation, etc. Moreover, AdaBoost is extended to some extent.