Personalized Federated Learning Method Based on Attention-Enhanced Meta-Learning Network

Gao Yujia; Wang Pengfei; Liu Liang; Ma Huadong

doi:10.7544/issn1000-1239.202220922

Journal of Computer Research and Development > 2024 > 61(1): 196-208. > DOI: 10.7544/issn1000-1239.202220922 CSTR: 32373.14.issn1000-1239.202220922

Gao Yujia, Wang Pengfei, Liu Liang, Ma Huadong. Personalized Federated Learning Method Based on Attention-Enhanced Meta-Learning Network[J]. Journal of Computer Research and Development, 2024, 61(1): 196-208. DOI: 10.7544/issn1000-1239.202220922

Citation:

PDF (2152 KB)

Personalized Federated Learning Method Based on Attention-Enhanced Meta-Learning Network

Gao Yujia^{1, 2,},
Wang Pengfei¹,
Liu Liang^{2, 3, ,},
Ma Huadong^{1, 2}

1.
School of Computer Science (National Pilot Software Engineering School), Beijing University of Posts and Telecommunications, Beijing 100876
2.
Beijing Key Laboratory of Intelligent Telecommunications Software and Multimedia (Beijing University of Posts and Telecommunications), Beijing 100876
3.
School of Artificial Intelligence, Beijing University of Posts and Telecommunications, Beijing 100876

Funds: This work was supported by the National Natural Science Foundation of China (61932013, 62061146002, 62225204) and the National 111 Project (B18008).

More Information

Author Bio:
Gao Yujia: born in 1995. PhD. Her main research interests include federated learning, deep learning, and intelligent sensing

Wang Pengfei: born in 1989. PhD, associate professor. His main research interests include data mining, machine learning, and recommendation system

Liu Liang: born in 1982. PhD, professor. Senior member of CCF. His main research interests include Internet of things and intelligence sensing

Ma Huadong: born in 1964. PhD, professor, fellow of CCF, fellow of IEEE. His main research interests include Internet of things, sensor networks, multimedia system and networking
Received Date: November 02, 2022
Revised Date: March 21, 2023
Available Online: January 07, 2024

Graphical Abstract

Abstract

Abstract

Federated learning is a distributed machine learning framework, which enables clients to conduct model training without transmitting their data to the servers. It is used to solve the dilemma of data silos and data privacy. It can work well on clients with similar data characteristics and distribution. However, in many scenarios, the differences of data distribution cause difficulties in global model training. Therefore, personalized federated learning is proposed as a new federated learning paradigm. It aims to guarantee the effectiveness of client personalized models through the collaboration between clients and the servers. Intuitively, providing a tighter collaboration for clients with similar data characteristics and distribution can facilitate the construction of personalized models. However, due to the invisibility of client data, it is a challenge to extract client features at a fine-grained level and define the collaborative relationships between clients. In this paper, we design an attention-enhanced meta-learning network (AMN) to address this issue. AMN can utilize model parameters as features and train the meta-learning network to provide an additional meta-model for each client to automatically analyze client feature similarity. According to two-layers framework of AMN, a trade-off between clients’ personality and commonality can be reasonably achieved, and a hybrid model with useful information from all clients is provided. Considering the need to maintain both the meta-model and the client’s local model during the training process, we design an alternative training strategy to perform the training in an end-to-end manner. To demonstrate the effectiveness of our method, we conduct extensive experiments on two benchmark datasets and eight baseline methods. Compared with the existing best-performing personalized federated learning methods, our method improves the accuracy rate by 3.39% and 2.45% on average in two datasets.
- federated learning,
- attention mechanism,
- deep learning,
- meta-learning,
- distributed machine learning

FullText(HTML)

References (33)

References

[1]	McMahan B, Moore E, Ramage D, et al. Communication-efficient learning of deep networks from decentralized data[C]// Proc of the 20th Int Conf on Artificial Intelligence and Statistics. New York: PMLR, 2017: 1273−1282
[2]	Birje M N, Hanji S S. Internet of things based distributed healthcare systems: A review[J]. Journal of Data, Information and Management, 2020, 2(3): 149−165 doi: 10.1007/s42488-020-00027-x
[3]	Lee Mingchang, Lin Jiachun. DALC: Distributed automatic LSTM customization for fine-grained traffic speed prediction[C]// Proc of the 34th Int Conf on Advanced Information Networking and Applications. Berlin: Springer, 2020: 164−175
[4]	Kairouz P, McMahan H B, Avent B, et al. Advances and open problems in federated learning[J]. Foundations and Trends in Machine Learning, 2021, 14(1/2): 1−210
[5]	Wu Qiong, He Kaiwen, Chen Xu. Personalized federated learning for intelligent IoT applications: A cloud-edge based framework[J]. IEEE Open Journal of the Computer Society, 2020, 1: 35−44 doi: 10.1109/OJCS.2020.2993259
[6]	Chen Hongyou, Chao Weilun. On bridging generic and personalized federated learning[J]. arXiv preprint, arXiv: 2107.00778, 2021
[7]	Lu Xiaofeng, Liao Yuying, Liu Chao, et al. Heterogeneous model fusion federated learning mechanism based on model mapping[J]. IEEE Internet of Things Journal, 2021, 9(8): 6058−6068
[8]	Feng Jie, Rong Can, Sun Funing, et al. PMF: A privacy-preserving human mobility prediction framework via federated learning[J]. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 2020, 4(1): 1−21
[9]	Arivazhagan M G, Aggarwal V, Singh A K, et al. Federated learning with personalization layers[J]. arXiv preprint, arXiv: 1912. 00818, 2019
[10]	Hanzely F, Richtárik P. Federated learning of a mixture of global and local models[J]. arXiv preprint, arXiv: 2002. 05516, 2020
[11]	Shen Tao, Zhang Jie, Jia Xinkang, et al. Federated mutual learning[J]. arXiv preprint, arXiv: 2006. 16765, 2020
[12]	钟正仪,包卫东,王吉,等. 一种面向云边端系统的分层异构联邦学习方法[J]. 计算机研究与发展,2022,59(11):2408−2422 doi: 10.7544/issn1000-1239.20220458 Zhong Zhengyi, Bao Weidong, Wang Ji, et al. A hierarchically heterogeneous federated learning method for cloud-edge-end system[J]. Journal of Computer Research and Development, 2022, 59(11): 2408−2422 (in Chinese) doi: 10.7544/issn1000-1239.20220458
[13]	Smith V, Chiang C K, Sanjabi M, et al. Federated multi-task learning[C]// Proc of the 30th Advances in Neural Information Processing Systems. San Francisco, CA: Morgan Kaufmann, 2017: 4424−4434
[14]	Huang Yutao, Chu Lingyang, Zhou Zirui, et al. Personalized cross-silo federated learning on Non-IID data[C]// Proc of the 11th Symp on Educational Advances in Artificial Intelligence. Palo Alto, CA: AAAI, 2021: 7865−7873
[15]	Jamal M A, Qi Guojun. Task agnostic meta-learning for few-shot learning[C]//Proc of the 2019 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2019: 11719-11727
[16]	Bello I, Zoph B, Vasudevan V, et al. Neural optimizer search with reinforcement learning[C]//Proc of the 34th Int Conf on Machine Learning. New York: PMLR, 2017: 459−468
[17]	Sun Qianru, Liu Yaoyao, Chua T, et al. Meta-transfer learning for few-shot learning[C]// Proc of the IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2019: 403−412
[18]	Santoro A, Bartunov S, Botvinick M, et al. Meta-learning with memory-augmented neural networks[C]// Proc of the 33rd Int Conf on Machine Learning. New York: PMLR, 2016: 1842-1850
[19]	Houthooft R, Chen Yuhua, Isola P, et al. Evolved policy gradients[C]// Proc of the 31st Advances in Neural Information Processing Systems. San Francisco, CA: Morgan Kaufmann, 2018: 5405−5414
[20]	Lin Jianxin, Wang Yijun, Chen Zhibo, et al. Learning to transfer: Unsupervised domain translation via meta-learning[C]// Proc of the 34th AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2020: 11507−11514
[21]	Jiang Yihan, Konečný J, Rush K, et al. Improving federated learning personalization via model agnostic meta learning[J]. arXiv preprint, arXiv: 1909. 12488, 2019
[22]	Fallah A, Mokhtari A, Ozdaglar A. Personalized federated learning: A meta-learning approach[J]. arXiv preprint, arXiv: 2002. 07948, 2020
[23]	Khodak M, Balcan M F F, Talwalkar A S. Adaptive gradient-based meta-learning methods[C]// Proc of the 32nd Advances in Neural Information Processing Systems. San Francisco, CA: Morgan Kaufmann, 2019: 5915−5926
[24]	Krishna K, Murty M N. Genetic K-means algorithm[J]. IEEE Transactions on Systems, Man, and Cybernetics: Part B (Cybernetics), 1999, 29(3): 433−439
[25]	Carreira-Perpinan M A. Gaussian mean-shift is an EM algorithm[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2007, 29(5): 767−776 doi: 10.1109/TPAMI.2007.1057
[26]	Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need[C]// Proc of the 30th Advances in Neural Information Processing Systems. San Francisco, CA: Morgan Kaufmann, 2017: 5998−6008
[27]	LeCun Y. The MNIST database of handwritten digits[EB/OL]. 1998[2021-06-08]. http://yann. lecun. com/exdb/mnist/
[28]	Xiao Han, Rasul K, Vollgraf R. Fashion-MNIST: A novel image dataset for benchmarking machine learning algorithms[J]. arXiv preprint, arXiv: 1708. 07747, 2017
[29]	无锡市滨湖区人民政府. GB 3095—2012 环境空气质量标准[S]. 无锡:滨湖生态环境局,2012 Wuxi Binhu District People’s Government. GB 3095—2012 Environmental Air Quality Standard [S]. Wuxi: Binhu Ecological Environment Bureau, 2012 (in Chinese)
[30]	Krizhevsky A, Sutskever I, Hinton G E. ImageNet classification with deep convolutional neural networks[J]. Communications of the ACM, 2017, 60(6): 84−90 doi: 10.1145/3065386
[31]	Chung J, Gulcehre C, Cho K H, et al. Empirical evaluation of gated recurrent neural networks on sequence modeling[J]. arXiv preprint, arXiv: 1412. 3555, 2014
[32]	Li Tian, Sahu A K, Zaheer M, et al. Federated optimization in heterogeneous networks[J]. Proceedings of Machine Learning and Systems, 2020, 2: 429−450
[33]	Chen Yiqiang, Qin Xin, Wang Jindong, et al. FedHealth: A federated transfer learning framework for wearable healthcare[J]. IEEE Intelligent Systems, 2020, 35(4): 83−93 doi: 10.1109/MIS.2020.2988604

[1]	Zhang Hengshan, Gao Yukun, Chen Yanping, Wang Zhongmin. Clustering Ensemble Algorithm with Cluster Connection Based on Wisdom of Crowds[J]. Journal of Computer Research and Development, 2018, 55(12): 2611-2619. DOI: 10.7544/issn1000-1239.2018.20180575
[2]	Zhou Jun, Li Huawei, Wang Tiancheng, Li Xiaowei. A Lightweight Fine-Grained Fault-Tolerant Scheme for 3D Networks-on-Chip[J]. Journal of Computer Research and Development, 2016, 53(2): 341-353. DOI: 10.7544/issn1000-1239.2016.20148436
[3]	Yuan Xinpan, Long Jun, Zhang Zuping, Luo Yueyi, Zhang Hao, and Gui Weihua. Connected Bit Minwise Hashing[J]. Journal of Computer Research and Development, 2013, 50(4): 883-890.
[4]	Liu Xiaozhu, Peng Zhiyong. On-Line Dynamic Index Hybrid Update Scheme Based on Self-Learning of Allocated Space[J]. Journal of Computer Research and Development, 2012, 49(10): 2118-2130.
[5]	Qi Shubo, Li Jinwen, Yue Daheng, Zhao Tianlei, and Zhang Minxuan. Adaptive Buffer Management for Leakage Power Optimization in NoC Routers[J]. Journal of Computer Research and Development, 2011, 48(12): 2400-2409.
[6]	Li Zhi, Zha Xuanyue, Liu Fengyu, and Zhang Hong. Indexing Based Multi-Level Clustering Routing Algorithm in Public Transportation Delay Tolerant Networks[J]. Journal of Computer Research and Development, 2011, 48(3): 407-414.
[7]	Tang Mingdong, Zhang Guoqing, Yang Jing. Graph Embedding-Based Scalable Routing in Large Networks[J]. Journal of Computer Research and Development, 2010, 47(7): 1225-1233.
[8]	Jiang Xuefeng, Heng Xingchen, Qin Zheng, Shao Liping. Efficient Extension Join Algorithm for Querying XML Data Based on Index Techniques[J]. Journal of Computer Research and Development, 2008, 45(6).
[9]	Yu Yaxin, Wang Guoren, Zhang Haining, and Li Jianxin. An Index for Supporting XML Structural Join Efficiently and Effectively—CATI[J]. Journal of Computer Research and Development, 2007, 44(1): 111-118.
[10]	Liao Wei, Xiong Wei, Jing Ning, Chen Hongsheng, and Zhong Zhinong. Hybrid Indexing of Moving Objects with Frequent Updates[J]. Journal of Computer Research and Development, 2006, 43(5): 888-893.