Personalized Federated Learning Method Based on Attention-Enhanced Meta-Learning Network

Gao Yujia; Wang Pengfei; Liu Liang; Ma Huadong

doi:10.7544/issn1000-1239.202220922

Journal of Computer Research and Development > 2024 > 61(1): 196-208. > DOI: 10.7544/issn1000-1239.202220922

Gao Yujia, Wang Pengfei, Liu Liang, Ma Huadong. Personalized Federated Learning Method Based on Attention-Enhanced Meta-Learning Network[J]. Journal of Computer Research and Development, 2024, 61(1): 196-208. DOI: 10.7544/issn1000-1239.202220922

Citation:

PDF (2152 KB)

Personalized Federated Learning Method Based on Attention-Enhanced Meta-Learning Network

Gao Yujia^{1, 2,},
Wang Pengfei¹,
Liu Liang^{2, 3, ,},
Ma Huadong^{1, 2}

1.
School of Computer Science (National Pilot Software Engineering School), Beijing University of Posts and Telecommunications, Beijing 100876
2.
Beijing Key Laboratory of Intelligent Telecommunications Software and Multimedia (Beijing University of Posts and Telecommunications), Beijing 100876
3.
School of Artificial Intelligence, Beijing University of Posts and Telecommunications, Beijing 100876

Funds: This work was supported by the National Natural Science Foundation of China (61932013, 62061146002, 62225204) and the National 111 Project (B18008).

More Information

Author Bio:
Gao Yujia: born in 1995. PhD. Her main research interests include federated learning, deep learning, and intelligent sensing

Wang Pengfei: born in 1989. PhD, associate professor. His main research interests include data mining, machine learning, and recommendation system

Liu Liang: born in 1982. PhD, professor. Senior member of CCF. His main research interests include Internet of things and intelligence sensing

Ma Huadong: born in 1964. PhD, professor, fellow of CCF, fellow of IEEE. His main research interests include Internet of things, sensor networks, multimedia system and networking
Received Date: November 02, 2022
Revised Date: March 21, 2023
Available Online: January 07, 2024

Graphical Abstract

Abstract

Abstract

Federated learning is a distributed machine learning framework, which enables clients to conduct model training without transmitting their data to the servers. It is used to solve the dilemma of data silos and data privacy. It can work well on clients with similar data characteristics and distribution. However, in many scenarios, the differences of data distribution cause difficulties in global model training. Therefore, personalized federated learning is proposed as a new federated learning paradigm. It aims to guarantee the effectiveness of client personalized models through the collaboration between clients and the servers. Intuitively, providing a tighter collaboration for clients with similar data characteristics and distribution can facilitate the construction of personalized models. However, due to the invisibility of client data, it is a challenge to extract client features at a fine-grained level and define the collaborative relationships between clients. In this paper, we design an attention-enhanced meta-learning network (AMN) to address this issue. AMN can utilize model parameters as features and train the meta-learning network to provide an additional meta-model for each client to automatically analyze client feature similarity. According to two-layers framework of AMN, a trade-off between clients’ personality and commonality can be reasonably achieved, and a hybrid model with useful information from all clients is provided. Considering the need to maintain both the meta-model and the client’s local model during the training process, we design an alternative training strategy to perform the training in an end-to-end manner. To demonstrate the effectiveness of our method, we conduct extensive experiments on two benchmark datasets and eight baseline methods. Compared with the existing best-performing personalized federated learning methods, our method improves the accuracy rate by 3.39% and 2.45% on average in two datasets.
- federated learning,
- attention mechanism,
- deep learning,
- meta-learning,
- distributed machine learning

FullText(HTML)

References (33)

References

[1]	McMahan B, Moore E, Ramage D, et al. Communication-efficient learning of deep networks from decentralized data[C]// Proc of the 20th Int Conf on Artificial Intelligence and Statistics. New York: PMLR, 2017: 1273−1282
[2]	Birje M N, Hanji S S. Internet of things based distributed healthcare systems: A review[J]. Journal of Data, Information and Management, 2020, 2(3): 149−165 doi: 10.1007/s42488-020-00027-x
[3]	Lee Mingchang, Lin Jiachun. DALC: Distributed automatic LSTM customization for fine-grained traffic speed prediction[C]// Proc of the 34th Int Conf on Advanced Information Networking and Applications. Berlin: Springer, 2020: 164−175
[4]	Kairouz P, McMahan H B, Avent B, et al. Advances and open problems in federated learning[J]. Foundations and Trends in Machine Learning, 2021, 14(1/2): 1−210
[5]	Wu Qiong, He Kaiwen, Chen Xu. Personalized federated learning for intelligent IoT applications: A cloud-edge based framework[J]. IEEE Open Journal of the Computer Society, 2020, 1: 35−44 doi: 10.1109/OJCS.2020.2993259
[6]	Chen Hongyou, Chao Weilun. On bridging generic and personalized federated learning[J]. arXiv preprint, arXiv: 2107.00778, 2021
[7]	Lu Xiaofeng, Liao Yuying, Liu Chao, et al. Heterogeneous model fusion federated learning mechanism based on model mapping[J]. IEEE Internet of Things Journal, 2021, 9(8): 6058−6068
[8]	Feng Jie, Rong Can, Sun Funing, et al. PMF: A privacy-preserving human mobility prediction framework via federated learning[J]. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 2020, 4(1): 1−21
[9]	Arivazhagan M G, Aggarwal V, Singh A K, et al. Federated learning with personalization layers[J]. arXiv preprint, arXiv: 1912. 00818, 2019
[10]	Hanzely F, Richtárik P. Federated learning of a mixture of global and local models[J]. arXiv preprint, arXiv: 2002. 05516, 2020
[11]	Shen Tao, Zhang Jie, Jia Xinkang, et al. Federated mutual learning[J]. arXiv preprint, arXiv: 2006. 16765, 2020
[12]	钟正仪,包卫东,王吉,等. 一种面向云边端系统的分层异构联邦学习方法[J]. 计算机研究与发展,2022,59(11):2408−2422 doi: 10.7544/issn1000-1239.20220458 Zhong Zhengyi, Bao Weidong, Wang Ji, et al. A hierarchically heterogeneous federated learning method for cloud-edge-end system[J]. Journal of Computer Research and Development, 2022, 59(11): 2408−2422 (in Chinese) doi: 10.7544/issn1000-1239.20220458
[13]	Smith V, Chiang C K, Sanjabi M, et al. Federated multi-task learning[C]// Proc of the 30th Advances in Neural Information Processing Systems. San Francisco, CA: Morgan Kaufmann, 2017: 4424−4434
[14]	Huang Yutao, Chu Lingyang, Zhou Zirui, et al. Personalized cross-silo federated learning on Non-IID data[C]// Proc of the 11th Symp on Educational Advances in Artificial Intelligence. Palo Alto, CA: AAAI, 2021: 7865−7873
[15]	Jamal M A, Qi Guojun. Task agnostic meta-learning for few-shot learning[C]//Proc of the 2019 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2019: 11719-11727
[16]	Bello I, Zoph B, Vasudevan V, et al. Neural optimizer search with reinforcement learning[C]//Proc of the 34th Int Conf on Machine Learning. New York: PMLR, 2017: 459−468
[17]	Sun Qianru, Liu Yaoyao, Chua T, et al. Meta-transfer learning for few-shot learning[C]// Proc of the IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2019: 403−412
[18]	Santoro A, Bartunov S, Botvinick M, et al. Meta-learning with memory-augmented neural networks[C]// Proc of the 33rd Int Conf on Machine Learning. New York: PMLR, 2016: 1842-1850
[19]	Houthooft R, Chen Yuhua, Isola P, et al. Evolved policy gradients[C]// Proc of the 31st Advances in Neural Information Processing Systems. San Francisco, CA: Morgan Kaufmann, 2018: 5405−5414
[20]	Lin Jianxin, Wang Yijun, Chen Zhibo, et al. Learning to transfer: Unsupervised domain translation via meta-learning[C]// Proc of the 34th AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2020: 11507−11514
[21]	Jiang Yihan, Konečný J, Rush K, et al. Improving federated learning personalization via model agnostic meta learning[J]. arXiv preprint, arXiv: 1909. 12488, 2019
[22]	Fallah A, Mokhtari A, Ozdaglar A. Personalized federated learning: A meta-learning approach[J]. arXiv preprint, arXiv: 2002. 07948, 2020
[23]	Khodak M, Balcan M F F, Talwalkar A S. Adaptive gradient-based meta-learning methods[C]// Proc of the 32nd Advances in Neural Information Processing Systems. San Francisco, CA: Morgan Kaufmann, 2019: 5915−5926
[24]	Krishna K, Murty M N. Genetic K-means algorithm[J]. IEEE Transactions on Systems, Man, and Cybernetics: Part B (Cybernetics), 1999, 29(3): 433−439
[25]	Carreira-Perpinan M A. Gaussian mean-shift is an EM algorithm[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2007, 29(5): 767−776 doi: 10.1109/TPAMI.2007.1057
[26]	Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need[C]// Proc of the 30th Advances in Neural Information Processing Systems. San Francisco, CA: Morgan Kaufmann, 2017: 5998−6008
[27]	LeCun Y. The MNIST database of handwritten digits[EB/OL]. 1998[2021-06-08]. http://yann. lecun. com/exdb/mnist/
[28]	Xiao Han, Rasul K, Vollgraf R. Fashion-MNIST: A novel image dataset for benchmarking machine learning algorithms[J]. arXiv preprint, arXiv: 1708. 07747, 2017
[29]	无锡市滨湖区人民政府. GB 3095—2012 环境空气质量标准[S]. 无锡:滨湖生态环境局,2012 Wuxi Binhu District People’s Government. GB 3095—2012 Environmental Air Quality Standard [S]. Wuxi: Binhu Ecological Environment Bureau, 2012 (in Chinese)
[30]	Krizhevsky A, Sutskever I, Hinton G E. ImageNet classification with deep convolutional neural networks[J]. Communications of the ACM, 2017, 60(6): 84−90 doi: 10.1145/3065386
[31]	Chung J, Gulcehre C, Cho K H, et al. Empirical evaluation of gated recurrent neural networks on sequence modeling[J]. arXiv preprint, arXiv: 1412. 3555, 2014
[32]	Li Tian, Sahu A K, Zaheer M, et al. Federated optimization in heterogeneous networks[J]. Proceedings of Machine Learning and Systems, 2020, 2: 429−450
[33]	Chen Yiqiang, Qin Xin, Wang Jindong, et al. FedHealth: A federated transfer learning framework for wearable healthcare[J]. IEEE Intelligent Systems, 2020, 35(4): 83−93 doi: 10.1109/MIS.2020.2988604

[1]	Cao Yiran, Zhu Youwen, He Xingyu, Zhang Yue. Utility-Optimized Local Differential Privacy Set-Valued Data Frequency Estimation Mechanism[J]. Journal of Computer Research and Development, 2022, 59(10): 2261-2274. DOI: 10.7544/issn1000-1239.20220504
[2]	Hong Jinxin, Wu Yingjie, Cai Jianping, Sun Lan. Differentially Private High-Dimensional Binary Data Publication via Attribute Segmentation[J]. Journal of Computer Research and Development, 2022, 59(1): 182-196. DOI: 10.7544/issn1000-1239.20200701
[3]	Wu Wanqing, Zhao Yongxin, Wang Qiao, Di Chaofan. A Safe Storage and Release Method of Trajectory Data Satisfying Differential Privacy[J]. Journal of Computer Research and Development, 2021, 58(11): 2430-2443. DOI: 10.7544/issn1000-1239.2021.20210589
[4]	Zhang Yuxuan, Wei Jianghong, Li Ji, Liu Wenfen, Hu Xuexian. Graph Degree Histogram Publication Method with Node-Differential Privacy[J]. Journal of Computer Research and Development, 2019, 56(3): 508-520. DOI: 10.7544/issn1000-1239.2019.20170886
[5]	Zhu Weijun, You Qingguang, Yang Weidong, Zhou Qinglei. Trajectory Privacy Preserving Based on Statistical Differential Privacy[J]. Journal of Computer Research and Development, 2017, 54(12): 2825-2832. DOI: 10.7544/issn1000-1239.2017.20160647
[6]	Wu Yingjie, Zhang Liqun, Kang Jian, Wang Yilei. An Algorithm for Differential Privacy Streaming Data Adaptive Publication[J]. Journal of Computer Research and Development, 2017, 54(12): 2805-2817. DOI: 10.7544/issn1000-1239.2017.20160555
[7]	Wang Liang, Wang Weiping, Meng Dan. Privacy Preserving Data Publishing via Weighted Bayesian Networks[J]. Journal of Computer Research and Development, 2016, 53(10): 2343-2353. DOI: 10.7544/issn1000-1239.2016.20160465
[8]	Lu Guoqing, Zhang Xiaojian, Ding Liping, Li Yanfeng, Liao Xin. Frequent Sequential Pattern Mining under Differential Privacy[J]. Journal of Computer Research and Development, 2015, 52(12): 2789-2801. DOI: 10.7544/issn1000-1239.2015.20140516
[9]	Ouyang Jia, Yin Jian, Liu Shaopeng, Liu Yubao. An Effective Differential Privacy Transaction Data Publication Strategy[J]. Journal of Computer Research and Development, 2014, 51(10): 2195-2205. DOI: 10.7544/issn1000-1239.2014.20130824
[10]	Ni Weiwei, Chen Geng, Chong Zhihong, Wu Yingjie. Privacy-Preserving Data Publication for Clustering[J]. Journal of Computer Research and Development, 2012, 49(5): 1095-1104.

Cited By

Cited by

Periodical cited type(5)

1.	张涵，于航，周继威，白云开，赵路坦. 面向隐私计算的可信执行环境综述. 计算机应用. 2025(02): 467-481 .
2.	付裕，林璟锵，冯登国. 虚拟化与密码技术应用:现状与未来. 密码学报(中英文). 2024(01): 3-21 .
3.	徐传康，李忠月，刘天宇，种统洪，杨发雪. 基于可信执行环境的汽车域控系统安全研究. 汽车实用技术. 2024(15): 18-25+73 .
4.	徐文嘉，岑孟杰，陈亮. 隐私保护下单细胞RNA测序数据细胞分类研究. 医学信息学杂志. 2024(10): 86-89 .
5.	孙钰，熊高剑，刘潇，李燕. 基于可信执行环境的安全推理研究进展. 信息网络安全. 2024(12): 1799-1818 .