Federated Learning Method Under User-Level Local Differential Privacy

Zhang Xiaojian; Zhang Leilei; Zhang Zhizheng

doi:10.7544/issn1000-1239.202330167

Journal of Computer Research and Development > 2025 > 62(2): 472-487. > DOI: 10.7544/issn1000-1239.202330167 CSTR: 32373.14.issn1000-1239.202330167

Zhang Xiaojian, Zhang Leilei, Zhang Zhizheng. Federated Learning Method Under User-Level Local Differential Privacy[J]. Journal of Computer Research and Development, 2025, 62(2): 472-487. DOI: 10.7544/issn1000-1239.202330167

Citation:

PDF (2394 KB)

Federated Learning Method Under User-Level Local Differential Privacy

School of Computer & Information Engineering, Henan University of Economics and Law, Zhengzhou 450016

Funds: This work was supported by the National Natural Science Foundation of China (62072156, 61502146, 91646203, 91746115) and the Basic Research Special Projects of Key Research Projects in Higher Education Institutes in Henan Province (25ZX012).

More Information

Author Bio:
Zhang Xiaojian: born in 1980. PhD, professor, master supervisor. His main research interests include differential privacy, data mining, and graph data management

Zhang Leilei: born in 1997. Master candidate. His main research interests include differential privacy and federated learning

Zhang Zhizheng: born in 1996. Master candidate. His main research interests include differential privacy and federated analytics
Received Date: March 16, 2023
Revised Date: January 07, 2024
Accepted Date: March 05, 2024
Available Online: March 06, 2024

Graphical Abstract

Abstract

Abstract

Federated learning with user-level local differential privacy (ULDP) has attracted considerable research attention in recent years. The trade-off among federated data types, the mechanism of clipping local updates, the allocation of privacy budget, and user dropout directly constrain the accuracy of the global learning model. All existing federated learning methods are vulnerable to handling these problems. To remedy the deficiency caused by the current methods, we employ ULDP to propose an efficient algorithm, called ULDP-FED, to achieve global federated optimization. ULDP-FED can simultaneously handle IID and non-IID federated data types. Compared with those methods with fixed clipping thresholds, ULDP-FED uses a threshold dynamic decay strategy to balance the noise error caused by the Gauss mechanism and the bias caused by update clipping. To allocate the privacy budget of each user carefully, in each round, ULDP-FED relies on the similarity to replace the current local update with the historical noise updates. If the historical updates are obtainted, the user only sends the index of the historical update to the server, which can reduce the communication cost. ULDP-FED is compared with existing methods over MNIST and CIFAR 10 datasets. The experimental results show that our algorithm outperforms its competitors, and achieves the accurate results of federated learning.
- differential privacy,
- user-level differential privacy (ULDP),
- federated learning,
- federated optimization,
- local gradient update

FullText(HTML)

References (24)

References

[1]	Kairouz P, McMahan H B, Avent B, et al. Advances and open problems in federated learning[J]. Foundations and Trends® in Machine Learning, 2021, 14(1/2): 1−210
[2]	Fang Minghong, Cao Xiaoyu, Jia Jinyuan, et al. Local model poisoning attacks to Byzantine-robust federated learning[C]//Proc of the 29th USENIX Security Symp (S&P 2020). Berkeley, CA: USENIX Association, 2020: 1623−1640
[3]	Cao Di, Chang Shan, Lin Zhijian, et al. Understanding distributed poisoning attack in federated learning[C]//Proc of the 25th Int Conf on Parallel and Distributed Systems (ICPADS 2019). Piscataway, NJ: IEEE, 2019: 233−239
[4]	So J, Güler B, Avestimehr A S. Byzantine-resilient secure federated learning[J]. IEEE Journal on Selected Areas in Communications, 2020, 39(7): 2168−2181
[5]	Zhu Ligeng, Liu Zhijian, Han Song. Deep leakage from gradients[C]//Proc of the 33rd Neural Information Processing Systems (NIPS 2019). Cambridge, MA: MIT, 2019: 14774–14784
[6]	Zhao Bao, Mopuri K R, Bilen H. iDLG: Improved deep leakage from gradients[J]. arXiv preprint, arXiv: 2001.02610, 2020
[7]	Nasr M, Shokri R, Houmansadr A. Comprehensive privacy analysis of deep learning: Passive and active white-box inference attacks against centralized and federated learning[C]//Proc of the 40th Symp on Security and Privacy (SP 2019). Piscataway, NJ: IEEE, 2019: 739−753
[8]	Liu Ruixuan, Cao Yang, Chen Hong, et al. Flame: Differentially private federated learning in the shuffle model[C]//Proc of the 35th Association for the Advance of Artificial Intelligence (AAAI 2021). Palo Alto, CA: AAAI, 2021: 8688−8696
[9]	Sun Lichao, Qian Jianwei, Chen Xun, et al. Ldp-FL: Practical private aggregation in federated learning with local differential privacy[C]//Proc of the 30th Int Joint Conf on Artificial Intelligence (IJCAI 2021). San Francisco, CA: Morgan Kaufmann, 2021: 1571−1578
[10]	McMahan H B, Ramage D, Talwar K, et al. Learning differentially private recurrent language models[J]. arXiv preprint, arXiv: 1710.06963, 2017
[11]	Geyer R C, Klein T, Nabi M. Differentially private federated learning: A client level perspective[J]. arXiv preprint, arXiv: 1712.07557, 2017
[12]	Wei Kang, Li Jun, Ding Ming, et al. User-level privacy-preserving federated learning: Analysis and performance optimization[J]. IEEE Transactions on Mobile Computing, 2021, 21(9): 3388−3401
[13]	Wei Kang, Li Jun, Ding Ming, et al. Federated learning with differential privacy: Algorithms and performance analysis[J]. IEEE Transactions on Information Forensics and Security, 2020, 15: 3454−3469 Wei Kang,Li Jun,Ding Ming,et al. Federated learning with differential privacy:Algorithms and performance analysis[J]. IEEE Transactions on Information Forensics and Security,2020,15:3454−3469
[14]	Andrew G, Thakkar O, McMahan B, et al. Differentially private learning with adaptive clipping[C]//Proc of the 35th Neural Information Processing Systems (NIPS 2021). Cambridge, MA: MIT, 2021: 17455−17466
[15]	Wang Lun, Jia Ruoxi, Song Dawn. D2P-Fed: Differentially private federated learning with efficient communication[J]. arXiv preprint, arXiv: 2006.13039, 2020
[16]	Shi Yi, Wei Kang, Li Shen, et al. Towards the flatter landscape and better generalization in federated learning under client-level differential privacy[J]. arXiv preprint, arXiv: 2305.00873, 2023
[17]	Shi Yi, Liu Yingqi, Wei Kang, et al. Make landscape flatter in differentially private federated learning[J]. arXiv preprint, arXiv: 2303.11242, 2023
[18]	Cheng Anda, Wang Peisong, Jian Cheng et al. Differentially private federated learning with local regularization and sparsification[C]//Proc of the 36th Computer Vision and Pattern Recognition (CVPR 2022). Los Alamitos, CA: IEEE Computer Society, 2022: 10112−10121
[19]	Abadi M, Chu Andy, Goodfellow I, et al. Deep learning with differential privacy[C]//Proc of the 2016 ACM SIGSAC Computer and Communications Security（CCS 2016）. New York: ACM, 2016: 308−318 Abadi M,Chu Andy,Goodfellow I,et al. Deep learning with differential privacy[C]//Proc of the 2016 ACM SIGSAC Computer and Communications Security(CCS 2016). New York:ACM,2016:308−318
[20]	Frank M. Privacy integrated queries: An extensible platform for privacy-preserving data analysis[C]//Proc of the 53rd ACM SIGMOD Int Conf on Management of Data (SIGMOD 2009). New York: ACM, 2009: 19–30
[21]	Dwork C, Roth A. The algorithmic foundations of differential privacy[J]. Foundations and Trends in Theoretical Computer Science, 2014, 9(3/4): 211−407
[22]	Wang Luping, Wang Wei, Li Bo. CMFL: Mitigating communication overhead for federated learning[C]//Proc of the 39th Int Conf on Distributed Computing Systems (ICDCS 2019). Piscataway, NJ: IEEE, 2019: 954−964
[23]	Zhang Xinwei, Chen Xiangyi, Hong Mingyi, et al. Understanding clipping for federated learning: Convergence and client-level differential privacy[C]//Proc of the 19th Int Conf on Machine Learning (ICML 2022). New York: ACM, 2022: 26048−26067
[24]	McMahan B, Moore E, Ramage D, et al. Communication-efficient learning of deep networks from decentralized data[C]//Proc of the 20th Artificial Intelligence and Statistics (AISTATS 2017). Cambridge, MA: MIT, 2017: 1273−1284

[1]	Xie Wenbing, Guan Ruixue, Zhang Yiming, Li Jiamei, Wang Jun. Efficient Optimization of Erasure Coding for Storage Library[J]. Journal of Computer Research and Development. DOI: 10.7544/issn1000-1239.202440091
[2]	Yan Zhiyuan, Xie Biwei, Bao Yungang. HVMS: A Hybrid Vectorization-Optimized Mechanism of SpMV[J]. Journal of Computer Research and Development, 2024, 61(12): 2969-2984. DOI: 10.7544/issn1000-1239.202330204
[3]	Wang Chuang, Ding Yan, Huang Chenlin, Song Liantao. Bitsliced Optimization of SM4 Algorithm with the SIMD Instruction Set[J]. Journal of Computer Research and Development, 2024, 61(8): 2097-2109. DOI: 10.7544/issn1000-1239.202220531
[4]	Shen Jie, Long Biao, Jiang Hao, Huang Chun. Implementation and Optimization of Vector Trigonometric Functions on Phytium Processors[J]. Journal of Computer Research and Development, 2020, 57(12): 2610-2620. DOI: 10.7544/issn1000-1239.2020.20190721
[5]	Yan Hongfei, Zhang Xudong, Shan Dongdong, Mao Xianling, Zhao Xin. SIMD-Based Inverted Index Compression Algorithms[J]. Journal of Computer Research and Development, 2015, 52(5): 995-1004. DOI: 10.7544/issn1000-1239.2015.20131548
[6]	Zhao Long, Han Wenbao, and Yang Hongzhi. Research on ECC Attacking Algorithm Based on SIMD Instructions[J]. Journal of Computer Research and Development, 2012, 49(7): 1553-1559.
[7]	He Yi, Ren Ju, Wen Mei, Yang Qianming, Wu Nan, Zhang Chunyuan, and Guo Min. Research on FPGA-Based Paging-Simulation Model for SIMD Architecture[J]. Journal of Computer Research and Development, 2011, 48(1): 9-18.
[8]	Huang Shuangqu, Xiang Bo, Bao Dan, Chen Yun, and Zeng Xiaoyang. VLSI Implementation of Multi-Standard LDPC Decoder Based on SIMD Architecture[J]. Journal of Computer Research and Development, 2010, 47(7): 1313-1320.
[9]	Li Zhaopeng, Chen Yiyun, Ge Lin, and Hua Baojian. A Formal Certifying Framework for Assembly Programs[J]. Journal of Computer Research and Development, 2008, 45(5): 825-833.
[10]	Lin Jiao, Chen Wenguang, Li Qiang, Zheng Weimin, Zhang Yimin. A New Data Clustering Algorithm for Parallel Whole-Genome Shotgun Sequence Assembly[J]. Journal of Computer Research and Development, 2006, 43(8): 1323-1329.