A Machine Unlearning Method via Feature Constraint and Adaptive Loss Balance

Yin Yuyu; Wu Guangqiang; Li Youhuizi; Wang Xinyu; Gao Honghao

doi:10.7544/issn1000-1239.202440476

Journal of Computer Research and Development > 2024 > 61(10): 2649-2661. > DOI: 10.7544/issn1000-1239.202440476 CSTR: 32373.14.issn1000-1239.202440476

Yin Yuyu, Wu Guangqiang, Li Youhuizi, Wang Xinyu, Gao Honghao. A Machine Unlearning Method via Feature Constraint and Adaptive Loss Balance[J]. Journal of Computer Research and Development, 2024, 61(10): 2649-2661. DOI: 10.7544/issn1000-1239.202440476

Citation:

PDF (2006 KB)

A Machine Unlearning Method via Feature Constraint and Adaptive Loss Balance

Yin Yuyu^{1, 2,},
Wu Guangqiang¹,
Li Youhuizi^{1, 2, ,},
Wang Xinyu¹,
Gao Honghao^{3, 4}

1.
School of Computer Science, Hangzhou Dianzi University, Hangzhou 310018
2.
Key Laboratory of Complex Systems Modeling and Simulation (Hangzhou Dianzi University), Ministry of Education, Hangzhou 310018
3.
School of Computer Engineering and Science, Shanghai University, Shanghai 200444
4.
Department of Computer Engineering, Gachon University, Seongnam, Republic of Korea 461701

Funds: This work was supported by the National Natural Science Foundation of China (62272140) , the Natural Science Foundation of Zhejiang Province (LY22F020018), and the “Pioneer”and“Leading Goose” Research and Development Program of Zhejiang Province (2024C01166).

More Information

Author Bio:
Yin Yuyu: born in 1980. PhD, professor. Member of CCF. His main research interests include service computing, edge computing, and business process management

Wu Guangqiang: born in 1999. Master. His main research interests include machine unlearning, deep learning, and incremental learning

Li Youhuizi: born in 1989. PhD, associate professor. Member of CCF. Her main research interests include edge computing, privacy security, and mobile edge computing

Wang Xinyu: born in 2000. Master. His main research interests include edge computing, deep learning, and natural language processing

Gao Honghao: born in 1985. PhD, associate professor. Member of CCF. His main research interests include software formal verification, service collaborative computing, wireless networks and industrial Internet of things, and intelligent medical image processing
Received Date: June 02, 2024
Revised Date: July 18, 2024
Available Online: September 13, 2024

Graphical Abstract

Abstract

Abstract

With the accelerated advancement of digitization, data elements have become the core driving force for the operation of modern society. However, at the same time, data security issues have become increasingly prominent, with frequent occurrences of data breaches and privacy violations, causing serious losses to individuals, organizations, and even countries. Against this backdrop, the security of data elements has become the focus of attention from all sectors of society, and the issue of data privacy protection in deep learning models has also attracted widespread attention. Among them, machine unlearning, as a key technology for protecting user’s privacy, aims to enable models to remove the influence of specific data while maintaining generalization performance for remaining data, providing an effective solution for protecting the security of data elements in deep learning models. Existing machine unlearning methods are mainly divided into two categories: exact unlearning and approximate unlearning. However, exact unlearning methods need to intervene in the original training process of the models, while approximate unlearning methods find it difficult to strike a balance between unlearning performance and model generalization ability. To address these issues, we propose an approximate unlearning framework based on feature constraints and adaptive loss balancing. We adopt a “forgetting-recovering” machine unlearning framework. First, for the “forgetting” process, in order to mimic the feature outputs of retrained models for the forgetting samples, we use a randomly initialized model that has not been trained on the forgetting samples to guide the feature outputs of the unlearning model, constraining forgetting at the feature level to avoid easily obtaining forgetting data information from the model. Then, a small amount of data is used for fine-tuning to “recover” the generalization performance of models on the remaining data. At the same time, we regard the above machine unlearning framework as a multi-task optimization problem and introduce adaptive loss balance to automatically balance the “forgetting” and “recovering” tasks, preventing the model from “over-forgetting” or “over-recovering”, so that the “forgetting” and “recovering” tasks can be trained relatively balanced and steadily. Extensive experiments on 3 image classification datasets show that our method can effectively forget the forgetting data and achieve optimal performance in multiple metrics.
- data element security,
- machine unlearning,
- feature constraints,
- multi-task optimization,
- adaptive loss balance

FullText(HTML)

References (34)

References

[1]	Pulido-Gaytan L B, Tchernykh A, Cortés-Mendoza J M, et al. A survey on privacy-preserving machine learning with fully homomorphic encryption[C]//Proc of Latin American High Performance Computing Conf. Cham: Springer International Publishing, 2020: 115−129
[2]	Liu Yi, Xu Lei, Yuan Xingliang, et al. The right to be forgotten in federated learning: An efficient realization with rapid retraining[C]//Proc of IEEE Conf on Computer Communications (IEEE INFOCOM 2022). Piscataway, NJ: IEEE, 2022: 1749−1758
[3]	Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition[C/OL]//Proc of the Int Conf on Learning Representations. 2015 [2024-05-28]. https://arxiv.org/abs/1409.1556
[4]	Goodfellow I, Pouget-Abadie J, Mirza M, et al. Generative adversarial nets[J]. Advances in Neural Information Processing Systems, 2014, 27: 2672−2680
[5]	Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need[C]//Advances in Neural Information Processing Systems, Cambridge, MA: MIT Press, 2017: 5998−6008
[6]	Fredrikson M, Jha S, Ristenpart T. Model inversion attacks that exploit confidence information and basic countermeasures[C]//Proc of the 22nd ACM SIGSAC Conf on Computer and Communications Security. New York: ACM, 2015: 1322−1333
[7]	Voigt P, Von dem Bussche A. The EU General Data Protection Regulation (GDPR)[EB/OL]. https://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:32016R0679
[8]	Bourtoule L, Chandrasekaran V, Choquette-Choo C A, et al. Machine unlearning[C]//Proc of 2021 IEEE Symp on Security and Privacy (SP). Piscataway, NJ: IEEE, 2021: 141−159
[9]	Yan Haonan, Li Xiaoguang, Guo Ziyao, et al. ARCANE: An efficient architecture for exact machine unlearning[C]//Proc of the 31st Int Joint Conf on Artificial Intelligence. Menlo Park, CA: AAAI, 2022: 4006−4013
[10]	Dukler Y, Bowman B, Achille A, et al. SAFE: Machine unlearning with shard graphs[C]//Proc of the IEEE/CVF Int Conf on Computer Vision. Piscataway, NJ: IEEE, 2023: 17108−17118
[11]	Golatkar A, Achille A, Soatto S. Eternal sunshine of the spotless net: Selective forgetting in deep networks[C]//Proc of the IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2020: 9304−9312
[12]	Huang H, Ma X, Erfani S M, et al. Unlearnable examples: Making personal data unexploitable[C/OL]//Proc of the Int Conf on Learning Representations. 2021 [2024-01-17]. https://arxiv.org/abs/2101.04898
[13]	Tarun A K, Chundawat V S, Mandal M, et al. Fast yet effective machine unlearning[J]. IEEE Transactions on Neural Networks and Learning Systems, 2023. Doi: 10.1109/TNNLS.2023.3266233, 2023
[14]	Kim J, Woo S S. Efficient two-stage model retraining for machine unlearning[C]//Proc of the IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2022: 4361−4369
[15]	Chundawat V S, Tarun A K, Mandal M, et al. Can bad teaching induce forgetting? unlearning in deep networks using an incompetent teacher[C]//Proc of the AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2023, 37(6): 7210−7217
[16]	Chen Min, Gao Weizhuo, Liu Gaoyang, et al. Boundary unlearning: Rapid forgetting of deep networks via shifting the decision boundary[C]//Proc of the IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2023: 7766−7775
[17]	Vandenhende S, Georgoulis S, Van Gansbeke W, et al. Multi-task learning for dense prediction tasks: A survey[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021, 44(7): 3614−3633
[18]	Zhao Shiji, Yu Jie, Sun Zhenlong, et al. Enhanced accuracy and robustness via multi-teacher adversarial distillation[C]//Proc of European Conf on Computer Vision. Cham: Springer, 2022: 585−602
[19]	Kirkpatrick J, Pascanu R, Rabinowitz N, et al. Overcoming catastrophic forgetting in neural networks[J]. Proceedings of the National Academy of Sciences, 2017, 114(13): 3521−3526 doi: 10.1073/pnas.1611835114
[20]	Hinton G, Vinyals O, Dean J. Distilling the knowledge in a neural network[J]. Computer Science, 2015, 14(7): 38−39
[21]	Zhang Xulong, Wang Jianzong, Cheng Ning, et al. Machine unlearning methodology based on stochastic teacher network[C]//Proc of Int Conf on Advanced Data Mining and Applications. Cham: Springer, 2023: 250−261
[22]	郭虎升,张洋,王文剑. 面向不同类型概念漂移的两阶段自适应集成学习方法[J]. 计算机研究与发展,2024,61(7):1799−1811 doi: 10.7544/issn1000-1239.202330452 Guo Husheng, Zhang Yang, Wang Wenjian. Two-stage adaptive ensemble learning method for different types of concept drift[J]. Journal of Computer Research and Development, 2024, 61(7): 1799−1811(in Chinese) doi: 10.7544/issn1000-1239.202330452
[23]	Foster J, Schoepf S, Brintrup A. Fast machine unlearning without retraining through selective synaptic dampening[C]//Proc of the AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2024, 38(11): 12043−12051
[24]	Aich A. Elastic weight consolidation (EWC): Nuts and bolts[J]. arXiv preprint, arXiv: 2105.04093, 2021
[25]	Cotogni M, Bonato J, Sabetta L, et al. DUCK: Distance-based unlearning via centroid kinematics[J]. arXiv preprint, arXiv: 2312.02052, 2023
[26]	Paszke A, Gross S, Massa F, et al. Pytorch: An imperative style, high-performance deep learning library[C]//Advances in Neural Information Processing Systems Cambridge, MA: MIT, 2019: 8024−8035
[27]	Krizhevsky A, Hinton G. Learning multiple layers of features from tiny images[R]. Toronto: University of Toronto, 2009
[28]	LeCun Y, Bottou L, Bengio Y, et al. Gradient-based learning applied to document recognition[J]. Proceedings of the IEEE, 1998, 86(11): 2278−2324 doi: 10.1109/5.726791
[29]	Springenberg J T, Dosovitskiy A, Brox T, et al. Striving for simplicity: The all convolutional net[C/OL]//Proc of the Int Conf on Learning Representations. 2015 [2024-02-02]. https://arxiv.org/abs/1412.6806
[30]	He Kaiming, Zhang Xiangyu, Ren Shaoqing, et al. Deep residual learning for image recognition[C]//Proc of the IEEE Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2016: 770−778
[31]	Shokri R, Stronati M, Song C, et al. Membership inference attacks against machine learning models[C]//Proc of 2017 IEEE Symp on Security and Privacy (SP). Piscataway, NJ: IEEE, 2017: 3−18
[32]	李文斌,熊亚锟,范祉辰,等. 持续学习的研究进展与趋势[J]. 计算机研究与发展,2024,61(6):1476−1496 doi: 10.7544/issn1000-1239.202220820 Li Wenbin, Xiong Yakun, Fan Zhichen, et al. Advances and trends of continual learning[J]. Journal of Computer Research and Development, 2024, 61(6): 1476−1496 (in Chinese) doi: 10.7544/issn1000-1239.202220820
[33]	Goodfellow I J, Shlens J, Szegedy C. Explaining and harnessing adversarial examples[C/OL]//Proc of the Int Conf on Learning Representations. 2015 [2024-02-02]. https://arxiv.org/abs/ 1412.6572
[34]	Selvaraju R R, Cogswell M, Das A, et al. Grad-CAM: Visual explanations from deep networks via gradient-based localization[C]//Proc of the IEEE Int Conf on Computer Vision. Piscataway, NJ: IEEE, 2017: 618−626

[1]	Jia Xibin, Li Chen, Wang Luo, Zhang Muchen, Liu Xiaojian, Zhang Yangyang, Wen Jiakai. A Multimodal Cross-Domain Sentiment Analysis Algorithm Based on Feature Disentanglement Meta-Optimization[J]. Journal of Computer Research and Development. DOI: 10.7544/issn1000-1239.202440624
[2]	Wang Jina, Chen Junhua, Gao Jianhua. ECC Multi-Label Code Smell Detection Method Based on Ranking Loss[J]. Journal of Computer Research and Development, 2021, 58(1): 178-188. DOI: 10.7544/issn1000-1239.2021.20190836
[3]	Xing Xinying, Ji Junzhong, Yao Yao. Brain Networks Classification Based on an Adaptive Multi-Task Convolutional Neural Networks[J]. Journal of Computer Research and Development, 2020, 57(7): 1449-1459. DOI: 10.7544/issn1000-1239.2020.20190186
[4]	Dong Xueshi, Dong Wenyong, Wang Yufeng. Hybrid Algorithms for Multi-Objective Balanced Traveling Salesman Problem[J]. Journal of Computer Research and Development, 2017, 54(8): 1751-1762. DOI: 10.7544/issn1000-1239.2017.20170347
[5]	Zhu Yelei, Wang Yujun, Luo Qiang, and Tao Qing. A Soft-Thresholding Coordinate Descent Algorithm for Solving Truncated Hinge Loss[J]. Journal of Computer Research and Development, 2013, 50(11): 2295-2303.
[6]	Yang Liu, Yu Jian, Jing Liping. An Adaptive Large Margin Nearest Neighbor Classification Algorithm[J]. Journal of Computer Research and Development, 2013, 50(11): 2269-2277.
[7]	Kong Kang, Tao Qing, Wang Qunshan, Chu Dejun. A Sub-Gadient Based Solver for L1-Rgularization+Hinge-Loss Problem[J]. Journal of Computer Research and Development, 2012, 49(7): 1494-1499.
[8]	Gong Maoguo, Cheng Gang, Jiao Licheng, and Liu Chao. Nondominated Individual Selection Strategy Based on Adaptive Partition for Evolutionary Multi-Objective Optimization[J]. Journal of Computer Research and Development, 2011, 48(4): 545-557.
[9]	Wu Jiawei, Li Xiongfei, Sun Tao, and Li Wei. A Density-Based Clustering Algorithm Concerning Neighborhood Balance[J]. Journal of Computer Research and Development, 2010, 47(6): 1044-1052.
[10]	Chai Yunpeng, Gu Lei, and Li Sanli. Cluster-Based Edge Streaming Server with Adaptive Load Balance in Mobile Grid[J]. Journal of Computer Research and Development, 2007, 44(12): 2136-2142.