FedDW: Distilling Weights through Consistency Optimization in Heterogeneous Federated Learning

Federated learning is a classic distributed machine learning paradigm that allows collaborative model training without centralized data. This method has significant advantages in ensuring data privacy. However, due to the significant data heterogeneity between clients and the continuous expansion of the federation scale, it faces many challenges in training efficiency and model performance. Previous studies have shown that in an independent and identically distributed environment, the parameter structure of the model usually satisfies a specific consistency relationship, and these relationships are often preserved in the intermediate results of the neural network training process. If the above consistency relationship can be identified and regularized under non-IID data, it will help to align the parameter distribution to the IID case, thereby alleviating the impact of data heterogeneity. Based on the above ideas, this paper first introduces the concept of deep learning encrypted data, and proposes a consistency optimization paradigm based on it. Then, it explores the consistency relationship between soft labels and classification layer weight matrices, and constructs a federated learning framework based on this. We conduct experiments on four public datasets and multiple neural network models (including ResNet and ViT). The results show that in a highly heterogeneous data setting, the average accuracy of this method is improved by about 3% compared with the 10 most mainstream federated learning methods. In addition, this paper also theoretically proves that the method in this paper has higher training efficiency, and its additional back-propagation computational overhead is negligible.