Concept Drift Convergence Method Based on Adaptive Deep Ensemble Networks

Guo Husheng; Sun Ni; Wang Jiahao; Wang Wenjian

doi:10.7544/issn1000-1239.202220835

Journal of Computer Research and Development > 2024 > 61(1): 172-183. > DOI: 10.7544/issn1000-1239.202220835

Guo Husheng, Sun Ni, Wang Jiahao, Wang Wenjian. Concept Drift Convergence Method Based on Adaptive Deep Ensemble Networks[J]. Journal of Computer Research and Development, 2024, 61(1): 172-183. DOI: 10.7544/issn1000-1239.202220835

Citation:

PDF (2126 KB)

Concept Drift Convergence Method Based on Adaptive Deep Ensemble Networks

Guo Husheng^{1, 2,},
Sun Ni¹,
Wang Jiahao¹,
Wang Wenjian^{1, 2, ,}

1.
School of Computer and Information Technology , Shanxi University , Taiyuan 030006
2.
Key Laboratory of Computational Intelligence and Chinese Information Processing ( Shanxi University ), Ministry of Education , Taiyuan 030006

Funds: This work was supported by the National Natural Science Foundation of China （62276157，U21A20513，62076154）and the Key Research and Development Program of Shanxi Province (202202020101003).

More Information

Author Bio:
Guo Husheng: born in 1986. PhD, professor, PhD supervisor. Senior member of CCF. His main research interests include data mining, machine learning, and computational intelligence

Sun Ni: born in 1998. Master. Her main research interests include stream data mining and online machine learning

Wang Jiahao: born in 2000. Bachelor. His main research interest includes data mining

Wang Wenjian: born in 1968. PhD, professor, PhD supervisor. Distinguished member of CCF. Her main research interests include machine learning, data mining, and computational intelligence
Received Date: September 27, 2022
Revised Date: April 05, 2023
Available Online: November 29, 2023

Graphical Abstract

Abstract

Abstract

Concept drift is an important and challenging problem in streaming data mining field. However, most existing methods can only deal with linear or simple nonlinear mappings. In spite of the ability of fitting nonlinear functions, neural network models have difficulty in adjusting dynamically according to the changing data streaming because only one sample or one batch of samples is available at a time for model training in the context of streaming data mining task. In order to solve above problem, the thought of gradient boosting algorithm is introduced to solve the problem of streaming data mining task with concept drift and a concept drift convergence method based on adaptive deep ensemble networks (CD_ADEN) is proposed. The proposed model combines several shallow neural networks as base leaner, and subsequent base learner corrects the output of precedent base learner to make the final output achieve high real-time generalization performance. Besides, because of the high convergence speed of shallow neural network, the proposed model will quickly recover from accuracy decrease caused by concept drift. The experimental results on multiple datasets show that the average real-time accuracy of the proposed CD_ADEN method is significantly improved compared with the comparative methods, the average real-time accuracy is improved by 1%−5%, and the average ordinal value ranks first in the comparison of several algorithms. It shows that the proposed model can correct the error of the pre-order output, and the learning model can quickly recover from the accuracy drop caused by concept drift, which improves the real-time generalization performance of the online learning model.
- streaming data,
- concept drift,
- gradient boosting,
- deep learning,
- quick adaptation

FullText(HTML)

References (41)

References

[1]	Lu Jie, Liu Anjin, Dong Fan, et al. Learning under concept drift: A review[J]. IEEE Transactions on Knowledge and Data Engineering, 2018, 31(12): 2346−2363
[2]	文益民,唐诗淇,冯超,等. 基于在线迁移学习的重现概念漂移数据流分类[J]. 计算机研究与发展,2016,53(8):1781−1791 Wen Yimin, Tang Shiqi, Feng Chao, et al. Recurrent concept drift data stream classification based on online transfer learning[J]. Journal of Computer Research and Development, 2016, 53(8): 1781−1791(in Chinese)
[3]	翟婷婷,高阳,朱俊武. 面向流数据分类的在线学习综述[J]. 软件学报,2020,31(4):912−931 Zhai Tingting, Gao Yang, Zhu Junwu. A review of online learning for streaming data classification[J]. Journal of Software, 2020, 31(4): 912−931(in Chinese)
[4]	Lu Yang, Cheung Y M, Tang Yangyang. Adaptive chunk-based dynamic weighted majority for imbalanced data streams with concept drift[J]. IEEE Transactions on Neural Networks and Learning Systems, 2019, 31(8): 2764−2778
[5]	Sun Yu, Tang Ke, Zhu Zexuan, et al. Concept drift adaptation by exploiting historical knowledge[J]. IEEE Transactions on Neural Networks and Learning Systems, 2018, 29(10): 4822−4832 doi: 10.1109/TNNLS.2017.2775225
[6]	Bifet A, Gavalda R. Learning from time-changing data with adaptive windowing [C] // Proc of the 7th SIAM Int Conf on Data Mining. Philadelphia, PA: SIAM, 2007: 443−448
[7]	Brzeziński D, Stefanowski J. Accuracy updated ensemble for data streams with concept drift [C] // Proc of the 6th Int Conf on Hybrid Artificial Intelligence Systems. Berlin: Springer, 2011: 155−163
[8]	Mohamad S, Sayed-Mouchaweh M, Bouchachia A. Active learning for classifying data streams with unknown number of classes[J]. Neural Networks, 2018, 98: 1−15 doi: 10.1016/j.neunet.2017.10.004
[9]	Bu Li, Alippi C, Zhao Dongbin. A pdf-free change detection test based on density difference estimation[J]. IEEE Transactions on Neural Networks and Learning Systems, 2018, 29(2): 324−334 doi: 10.1109/TNNLS.2016.2619909
[10]	Ross G J, Adams N M, Tasoulis D K, et al. Exponentially weighted moving average charts for detecting concept drift[J]. Pattern Recognition Letters, 2012, 33(2): 191−198 doi: 10.1016/j.patrec.2011.08.019
[11]	蔡桓,陆克中,伍启荣,等. 面向概念漂移数据流的自适应分类算法[J]. 计算机研究与发展,2022,59(3):633−646 Cai Huan, Lu Kezhong, Wu Qirong, et al. Adaptive classification algorithm for concept drift data streams[J]. Journal of Computer Research and Development, 2022, 59(3): 633−646(in Chinese)
[12]	Bifet A, Holmes G, Pfahringer B, et al. New ensemble methods for evolving data streams [C] // Proc of the 15th ACM SIGKDD Int Conf on Knowledge Discovery and Data Mining. New York: ACM, 2009: 139−148
[13]	Brzezinski D, Stefanowski J. Reacting to different types of concept drift: The accuracy updated ensemble algorithm[J]. IEEE Transactions on Neural Networks and Learning Systems, 2014, 25(1): 81−94 doi: 10.1109/TNNLS.2013.2251352
[14]	Junior J R. Graph embedded rules for explainable predictions in data streams[J]. Neural Networks, 2020, 129: 174−192 doi: 10.1016/j.neunet.2020.05.035
[15]	郭虎升,张爱娟,王文剑. 基于在线性能测试的概念漂移检测方法[J]. 软件学报,2020,31(4):932−947 Guo Husheng, Zhang Aijuan, Wang Wenjian. Concept drift detection method based on online performance testing[J]. Journal of Software, 2020, 31(4): 932−947(in Chinese)
[16]	Shen Yanning, Chen Tianyi, Giannakis G B. Random feature-based online multi-kernel learning in environments with unknown dynamics[J]. Journal of Machine Learning Research, 2019, 20(1): 773−808
[17]	杜航原,王文剑,白亮. 一种基于优化模型的演化数据流聚类方法[J]. 中国科学:信息科学,2017,47(11):1464−1482 doi: 10.1360/N112017-00107 Du Hangyuan, Wang Wenjian, Bai Liang. An evolutionary data stream clustering method based on optimization model[J]. SCIENTIA SINICA Informationis, 2017, 47(11): 1464−1482 (in Chinese) doi: 10.1360/N112017-00107
[18]	Sun Hao, Zheng Xiangtao, Lu Xiaoqing. A supervised segmentation network for hyperspectral image classification [J]. IEEE Transactions on Image Processing, 2021, 30: 2810−2825 Sun Hao,Zheng Xiangtao,Lu Xiaoqing. A supervised segmentation network for hyperspectral image classification [J]. IEEE Transactions on Image Processing,2021,30:2810−2825
[19]	Tian Yicong, Dehghan Y, Shah M. On detection, data association and segmentation for multi-target tracking[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019, 41(9): 2146−2160 doi: 10.1109/TPAMI.2018.2849374
[20]	Vaswani A, Shazeer N, Parmar N, et al. Attention is all you need [C] // Proc of the 31st Annual Conf on Neural Information Processing Systems. Cambridge, MA: MIT, 2017: 5998−6008
[21]	Rubio J, Angelov P, Pacheco J. Uniformly stable backpropagation algorithm to train a feedforward neural network[J]. IEEE Transactions on Neural Networks, 2011, 22(3): 356−366 doi: 10.1109/TNN.2010.2098481
[22]	Chen Wenzheng, Mirdehghan P, Fidler S, et al. Auto-tuning structured light by optical stochastic gradient descent [C] // Proc of the 2020 IEEE/CVF Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2020: 5969−5979
[23]	Lu Ning, Lu Jie, Zhang Guangquan, et al. A concept drift-tolerant case-base editing technique[J]. Artificial Intelligence, 2016, 230: 108−133 doi: 10.1016/j.artint.2015.09.009
[24]	Zhu Qun, Hu Xuegang, Zhang Yuhong, et al. A double-window based classification algorithm for concept drifting data streams [C] // Proc of the 2010 IEEE Int Conf on Granular Computing. Piscataway, NJ: IEEE, 2010: 639−644
[25]	Yang Ying, Wu Xindong, Zhu Xingquan. Combining proactive and reactive predictions for data streams [C] // Proc of the 11th ACM SIGKDD Int Conf on Knowledge Discovery and Data Mining. New York: ACM, 2015: 710−715
[26]	Guo Husheng, Li Hai, Ren Qiaoyan, et al. Concept drift type identification based on multi-sliding windows[J]. Information Sciences, 2022, 585: 1−25 doi: 10.1016/j.ins.2021.11.023
[27]	郭虎升,任巧燕,王文剑. 基于时序窗口的概念漂移类别检测[J]. 计算机研究与发展,2022,59(1):127−143 doi: 10.7544/issn1000-1239.20200562 Guo Husheng, Ren Qiaoyan, Wang Wenjian. Category detection of concept drift based on time series window[J]. Journal of Computer Research and Development, 2022, 59(1): 127−143(in Chinese) doi: 10.7544/issn1000-1239.20200562
[28]	Street W N, Kim Y S. A streaming ensemble algorithm (SEA) for large-scale classification [C] // Proc of the 7th ACM SIGKDD Int Conf on Knowledge Discovery and Data Mining. New York: ACM, 2001: 377−382
[29]	Kolter J Z, Maloof M A. Dynamic weighted majority: An ensemble method for drifting concepts[J]. Journal of Machine Learning Research, 2007, 8: 2755−2790
[30]	赵鹏,周志华. 基于决策树模型重用的分布变化流数据学习[J]. 中国科学:信息科学,2021,51(1):1−12 doi: 10.1360/SSI-2020-0170 Zhao Peng, Zhou Zhihua. Data learning of distribution changes based on decision tree model reuse[J]. SCIENTIA SINICA Informationis, 2021, 51(1): 1−12 (in Chinese) doi: 10.1360/SSI-2020-0170
[31]	Elwell R, Polikar R. Incremental learning of concept drift in non-stationary environments[J]. IEEE Transactions on Neural Networks, 2011, 22(10): 1517−1531 doi: 10.1109/TNN.2011.2160459
[32]	Mirza B, Lin Zhipin. Meta-cognitive online sequential extreme learning machine for imbalanced and concept-drifting data classification[J]. Neural Networks, 2016, 80: 79−94 doi: 10.1016/j.neunet.2016.04.008
[33]	Hammami Z, Mouelhi W, Said L B. Online self-adaptive framework for tailoring a neural-agent learning model addressing dynamic real-time scheduling problems[J]. Journal of Manufacturing Systems, 2017, 45: 97−108 doi: 10.1016/j.jmsy.2017.08.003
[34]	Ashfahani A, Pratama M. Autonomous deep learning: Continual learning approach for dynamic environments [C] // Proc of the 2019 SIAM Int Conf on Data Mining. Philadelphia, PA: SIAM, 2019: 666−674
[35]	Yu Hualong, Webb G I. Adaptive online extreme learning machine by regulating forgetting factor by concept drift map[J]. Neurocomputing, 2019, 343: 141−153 doi: 10.1016/j.neucom.2018.11.098
[36]	Sahoo D, Pham Q, Lu Jing, et al. Online deep learning: Learning deep neural networks on the fly [C] // Proc of the 27th Int Joint Conf on Artificial Intelligence. San Francisco, CA: Morgan Kaufmann, 2018: 2660−2666
[37]	Guo Husheng, Zhang Shuai, Wang Wenjian. Selective ensemble-based online adaptive deep neural networks for streaming data with concept drift[J]. Neural Networks, 2021, 142: 437−456 doi: 10.1016/j.neunet.2021.06.027
[38]	He Kaiming, Zhang Xiangyu, Ren Shaoqing, et al. Deep residual learning for image recognition [C] // Proc of the 2016 IEEE Conf on Computer Vision and Pattern Recognition. Piscataway, NJ: IEEE, 2016: 770−778
[39]	Srivastava R K, Greff K, Schmidhuber J. Training very deep networks [C] // Proc of the 2015 on Neural Information Processing Systems. Cambridge, MA: MIT, 2015: 2377−2385
[40]	Brodersen K H, Ong C S, Stephan K E, et al. The balanced accuracy and its posterior distribution [C] // Proc of the 20th Int Conf on Pattern Recognition. Piscataway, NJ: IEEE, 2010: 3121−3124
[41]	Demsar J. Statistical comparisons of classifiers over multiple datasets[J]. Journal of Machine Learning Research, 2006, 7(1): 1−3

[1]	Guo Husheng, Zhang Yang, Wang Wenjian. Two-Stage Adaptive Ensemble Learning Method for Different Types of Concept Drift[J]. Journal of Computer Research and Development, 2024, 61(7): 1799-1811. DOI: 10.7544/issn1000-1239.202330452
[2]	Guo Husheng, Zhang Yutong, Wang Wenjian. Elastic Gradient Ensemble for Concept Drift Adaptation[J]. Journal of Computer Research and Development. DOI: 10.7544/issn1000-1239.202440407
[3]	Guo Husheng, Cong Lu, Gao Shuhua, Wang Wenjian. Adaptive Classification Method for Concept Drift Based on Online Ensemble[J]. Journal of Computer Research and Development, 2023, 60(7): 1592-1602. DOI: 10.7544/issn1000-1239.202220245
[4]	Cai Derun, Li Hongyan. A Metric Learning Based Unsupervised Domain Adaptation Method with Its Application on Mortality Prediction[J]. Journal of Computer Research and Development, 2022, 59(3): 674-682. DOI: 10.7544/issn1000-1239.20200693
[5]	Cai Huan, Lu Kezhong, Wu Qirong, Wu Dingming. Adaptive Classification Algorithm for Concept Drift Data Stream[J]. Journal of Computer Research and Development, 2022, 59(3): 633-646. DOI: 10.7544/issn1000-1239.20201017
[6]	Yu Xian, Li Zhenyu, Sun Sheng, Zhang Guangxing, Diao Zulong, Xie Gaogang. Adaptive Virtual Machine Consolidation Method Based on Deep Reinforcement Learning[J]. Journal of Computer Research and Development, 2021, 58(12): 2783-2797. DOI: 10.7544/issn1000-1239.2021.20200366
[7]	Bai Chenjia, Liu Peng, Zhao Wei, Tang Xianglong. Active Sampling for Deep Q-Learning Based on TD-error Adaptive Correction[J]. Journal of Computer Research and Development, 2019, 56(2): 262-280. DOI: 10.7544/issn1000-1239.2019.20170812
[8]	Zhang Yuanpeng, Deng Zhaohong, Chung Fu-lai, Hang Wenlong, Wang Shitong. Fast Self-Adaptive Clustering Algorithm Based on Exemplar Score Strategy[J]. Journal of Computer Research and Development, 2018, 55(1): 163-178. DOI: 10.7544/issn1000-1239.2018.20160937
[9]	Ma Anxiang, Zhang Bin, Gao Kening, Qi Peng, and Zhang Yin. Deep Web Data Extraction Based on Result Pattern[J]. Journal of Computer Research and Development, 2009, 46(2): 280-288.
[10]	Dandan, Li Zusong, Wang Jian, Zhang Longbing, Hu Weiwu, Liu Zhiyong. Adaptive Stack Cache with Fast Address Generation[J]. Journal of Computer Research and Development, 2007, 44(1): 169-176.