Citation: | Jiao Pengfei, Chen Shuxin, Guo Xuan, He Dongxiao, Liu Dong. Survey on Graph Neural Ordinary Differential Equations[J]. Journal of Computer Research and Development, 2024, 61(8): 2045-2066. DOI: 10.7544/issn1000-1239.202440192 |
Graph neural networks (GNNs) are powerful tools for handling graph-structured data, capable of capturing complex relationships and features among nodes. However, the discrete architecture of GNNs leads to numerous challenges in representing graph structures, modeling graph evolution, adapting to irregular data, and managing computational costs. In response to these challenges, neural ordinary differential equations (ODEs) have been introduced as a novel method to address the challenges faced by GNNs, as they can simulate the continuous evolution of system states, providing continuous deep encoding and inference capabilities. However, neural ODEs are designed for Euclidean structured data and cannot directly capture the characteristics of graphs. Therefore, researchers have proposed graph neural ODEs, a new type of architectures that combines neural ODEs with GNNs, which can better adapt to graph-structured data and fully utilize its characteristics. In recent years, research related to graph neural ODEs has delved into various directions of graph machine learning, sparking a new research trend. In this context, we systematically review the relevant research of graph neural ODEs in a timely manner. Firstly, we review the key advantages of GNNs and the challenges they face, and elucidate the theoretical basis and practical significance of introducing neural ODEs and combining them with GNNs. Subsequently, we elaborate on the background and basic concepts of graph neural ODEs, proposing a novel taxonomy, and comprehensively describe some important current methods on the taxonomy. Then, we introduce commonly used verification methods in related research, including downstream tasks and datasets. Furthermore, we delve into the applications of graph neural ODEs in multiple practical fields. Finally, we summarize and prospect the challenges and future development trends of graph neural ODEs.
[1] |
徐冰冰,岑科廷,黄俊杰,等. 图卷积神经网络综述[J]. 计算机学报,2020,43(5):755−780 doi: 10.11897/SP.J.1016.2020.00755
Xu Bingbing, Cen Keting, Huang Junjie, et al. A survey on graph convolutional network[J]. Chinese Journal of Computers, 2020, 43(5): 755−780 (in Chinese) doi: 10.11897/SP.J.1016.2020.00755
|
[2] |
马帅,刘建伟,左信. 图神经网络综述[J]. 计算机研究与发展,2022,59(1):47−80 doi: 10.7544/issn1000-1239.20201055
Ma Shuai, Liu Jianwei, Zuo Xin. Survey on graph neural network[J]. Journal of Computer Research and Development, 2022, 59(1): 47−80 (in Chinese) doi: 10.7544/issn1000-1239.20201055
|
[3] |
Wu Zonghan, Pan Shirui, Chen Fengwen, et al. A comprehensive survey on graph neural networks[J]. IEEE Transactions on Neural Networks and Learning Systems, 2020, 32(1): 4−24
|
[4] |
Liu Juncheng, Hooi B, Kawaguchi K, et al. Scalable and effective implicit graph neural networks on large graphs[C/OL]//Proc of the 12th Int Conf on Learning Representations. 2024[2024-05-16].https://openreview.net/forum?id=QcMdPYBwTu
|
[5] |
Xu Keyulu, Hu Weihua, Leskovec J, et al. How powerful are graph neural networks?[C/OL]//Proc of the 7th Int Conf on Learning Representations. 2019[2024-05-16].https://openreview.net/forum?id=ryGs6iA5Km
|
[6] |
Gasteiger J, Bojchevski A, Günnemann S. Predict then propagate: Graph neural networks meet personalized pagerank[C/OL]//Proc of the 7th Int Conf on Learning Representations. 2019[2024-05-16].https://openreview.net/forum?id=H1gL-2A9Ym
|
[7] |
Hajiramezanali E, Hasanzadeh A, Narayanan K, et al. Variational graph recurrent neural networks[C]//Proc of the 33rd Int Conf on Neural Information Processing Systems. New York: Curran Associates, Inc, 2019, 32: 10701−10711
|
[8] |
Xu Da, Ruan Chuanwei, Korpeoglu E, et al. Inductive representation learning on temporal graphs[C/OL]//Proc of the 8th Int Conf on Learning Representations. 2020[2024-05-16].https://openreview.net/forum?id=rJeW1yHYwH
|
[9] |
Wu Yongji, Lian Defu, Xu Yiheng, et al. Graph convolutional networks with Markov random field reasoning for social spammer detection[C]//Proc of the 34th AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2020, 34(1): 1054−1061
|
[10] |
Zhang Si, Tong Hanghang, Xu Jiejun, et al. Graph convolutional networks: A comprehensive review[J]. Computational Social Networks, 2019, 6(1): 1−23 doi: 10.1186/s40649-019-0061-6
|
[11] |
Ying Rex, He Ruining, Chen Kaifeng, et al. Graph convolutional neural networks for web-scale recommender systems[C]//Proc of the 24th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. New York: ACM, 2018: 974−983
|
[12] |
王磊,熊于宁,李云鹏,等. 一种基于增强图卷积神经网络的协同推荐模型[J]. 计算机研究与发展,2021,58(9):1987−1996 doi: 10.7544/issn1000-1239.2021.20200617
Wang Lei, Xiong Yuning, Li Yunpeng, el al. A collaborative recommendation model based on enhanced graph convolutional neural network[J]. Journal of Computer Research and Development, 2021, 58(9): 1987−1996 (in Chinese) doi: 10.7544/issn1000-1239.2021.20200617
|
[13] |
Wang Shoujin, Hu Liang, Wang Yan, et al. Graph learning based recommender systems: A review[C]//Proc of the 30th Int Joint Conf on Artificial Intelligence. San Francisco, CA: IJCAI, 2021: 4644−4652
|
[14] |
Fout A, Byrd J, Shariat B, et al. Protein interface prediction using graph convolutional networks[C]//Proc of the 31st Int Conf on Neural Information Processing Systems. New York: Curran Associates, Inc, 2017, 30: 6533−6542
|
[15] |
Borgwardt K M, Ong C S, Schönauer S, et al. Protein function prediction via graph kernels[J]. Bioinformatics, 2005, 21(suppl_1): i47−i56
|
[16] |
Guo Shengnan, Lin Youfang, Feng Ning, et al. Attention based spatial-temporal graph convolutional networks for traffic flow forecasting[C]//Proc of the 33rd AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2019: 922−929
|
[17] |
Yu Bing, Yin Haoteng, Zhu Zhanxing. Spatio-temporal graph convolutional networks: A deep learning framework for traffic forecasting[C]//Proc of the 28th Int Joint Conf on Artificial Intelligence. San Francisco, CA: IJCAI, 2018: 3634−3640
|
[18] |
杜圣东,李天瑞,杨燕,等. 一种基于序列到序列时空注意力学习的交通流预测模型[J]. 计算机研究与发展,2020,57(8):1715−1728 doi: 10.7544/issn1000-1239.2020.20200169
Du Shengdong, Li Tianrui, Yang Yan, et al. A sequence-to-sequence spatial-temporal attention learning model for urban traffic flow prediction[J]. Journal of Computer Research and Development, 2020, 57(8): 1715−1728 (in Chinese) doi: 10.7544/issn1000-1239.2020.20200169
|
[19] |
Bodnar C, Di Giovanni F, Chamberlain B, et al. Neural sheaf diffusion: A topological perspective on heterophily and oversmoothing in GNNS[C]//Proc of the 36th Int Conf on Neural Information Processing Systems. New York: Curran Associates, Inc, 2022, 35: 18527−18541
|
[20] |
Rusch T K, Bronstein M M, Mishra S. A survey on oversmoothing in graph neural networks[J]. arXiv preprint, arXiv: 2303.10993, 2023
|
[21] |
Rong Yu, Huang Wenbing, Xu Tingyang, et al. DropEdge: Towards deep graph convolutional networks on node classification[C/OL]//Proc of the 8th Int Conf on Learning Representations. 2020 [2024-05-16].https://openreview.net/forum?id=Hkx1qkrKPr
|
[22] |
Zhao L, Akoglu L. PairNorm: Tackling oversmoothing in GNN[C]//Proc of the 8th Int Conf on Learning Representations. 2020[2024-05-16].https://openreview.net/forum?id=rkecl1rtwB
|
[23] |
Xu Keyulu, Li Chengtao, Tian Yonglong, et al. Representation learning on graphs with jumping knowledge networks[C]//Proc of the 35th Int Conf on Machine Learning. New York: PMLR, 2018: 5453−5462
|
[24] |
Barzel B, Liu Yangyu, Barabási A L. Constructing minimal models for complex system dynamics[J]. Nature Communication, 2015, 6(1): 7186 doi: 10.1038/ncomms8186
|
[25] |
Alvarez-Rodriguez U, Battiston F, de Arruda G F, et al. Evolutionary dynamics of higher-order interactions in social networks[J]. Nature Human Behaviour, 2021, 5(5): 586−595 doi: 10.1038/s41562-020-01024-1
|
[26] |
Zhang Yanfu, Gao Shangqian, Pei Jian, et al. Improving social network embedding via new second-order continuous graph neural networks[C]//Proc of the 28th ACM SIGKDD Conf on Knowledge Discovery and Data Mining. New York: ACM, 2022: 2515−2523
|
[27] |
Hou Jinlin, Guo Xuan, Liu Jiye, et al. Structure-enhanced graph neural ODE network for temporal link prediction[C]//Proc of the 33rd Int Conf on Artificial Neural Networks. Berlin: Springer, 2023: 563−575
|
[28] |
Fey M, Lenssen J E, Weichert F, et al. Gnnautoscale: Scalable and expressive graph neural networks via historical embeddings[C]//Proc of the 38th Int Confon Machine Learning. New York: PMLR, 2021: 3294−3304
|
[29] |
Chen R T Q, Rubanova Y, Bettencourt J, et al. Neural ordinary differential equations[C]//Proc of the 32nd Int Conf on Neural Information Processing Systems. New York: Curran Associates, Inc, 2018, 31: 6572−6583
|
[30] |
Xhonneux L P, Qu Meng, Tang Jian. Continuous graph neural networks[C]//Proc of the 37th Int Conf on Machine Learning. New York: PMLR, 2020: 10432−10441
|
[31] |
Zang Chengxi, Wang Fei. Neural dynamics on complex networks[C]//Proc of the 26th ACM SIGKDD Conf on Knowledge Discovery and Data Mining. New York: ACM, 2020: 892−902
|
[32] |
Wang Zihui, Yang Peizhen, Fan Xiaoliang, et al. ConTIG: Continuous representation learning on temporal interaction graphs[J]. Neural Networks, 2024, 172: 106151 doi: 10.1016/j.neunet.2024.106151
|
[33] |
Veličković P, Cucurull G, Casanova A, et al. Graph attention networks[C/OL]//Proc of the 6th Int Conf on Learning Representations. 2018[2024-05-16].https://openreview.net/forum?id=rJXMpikCZ
|
[34] |
Rossi E, Chamberlain B, Frasca F, et al. Temporal graph networks for deep learning on dynamic graphs[C]//Proc of the ICML 2020 Workshop on Graph Representation Learning and Beyond. New York: PMLR, 2020: 1−9
|
[35] |
Ma Yao, Guo Ziyi, Ren Zhaocun, et al. Streaming graph neural networks[C]//Proc of the 43rd Int ACM SIGIR Conf on Research and Development in Information Retrieval. New York: ACM, 2020: 719−728
|
[36] |
Zhou Hongkuan, Zheng Da, Nisa I, et al. TGL: A general framework for temporal gnn training on billion-scale graphs[C]//Proc of the 48th Int Conf on Very Large Databases. New York: VLDB Endowment, 2022, 15(8): 1572−1580
|
[37] |
Pareja A, Domeniconi G, Chen Jie, et al. EvolveGCN: Evolving graph convolutional networks for dynamic graphs[C]//Proc of the 34th AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2020: 5363−5370
|
[38] |
Taheri A, Gimpel K, Berger-Wolf T. Learning to represent the evolution of dynamic graphs with recurrent models[C]//Proc of Companion Proc of the 2019 World Wide Web Conf. New York: ACM, 2019: 301−307
|
[39] |
Wang Yanbang, Li Pan, Bai Chongyang, et al. Generic representation learning for dynamic social interaction[C]//Proc of the 26th ACM SIGKDD Conf on Knowledge Discovery and Data Mining Workshop. New York: ACM, 2020: 1−9
|
[40] |
You Jiaxuan, Du Tianyu, Leskovec J. ROLAND: Graph learning framework for dynamic graphs[C]//Proc of the 28th ACM SIGKDD Conf on Knowledge Discovery and Data Mining. New York: ACM, 2022: 2358−2366
|
[41] |
Pontryagin L S. Mathematical Theory of Optimal Processes[M]. London: Routledge, 2018
|
[42] |
Rubanova Y, Chen R T Q, Duvenaud D K. Latent ordinary differential equations for irregularly-sampled time series[C]//Proc of the 33rd Int Conf on Neural Information Processing Systems. New York: Curran Associates, Inc, 2019, 32: 5320−5330
|
[43] |
Dupont E, Doucet A, Teh Y W. Augmented neural ODEs[C]//Proc of the 33rd Int Conf on Neural Information Processing Systems. New York: Curran Associates, Inc, 2019, 32: 3140−3150
|
[44] |
Biswas B N, Chatterjee S, Mukherjee S P, et al. A discussion on Euler method: A review[J]. Electronic Journal of Mathematical Analysis and Applications, 2013, 1(2): 2090−2792
|
[45] |
Dormand J R, Prince P J. A family of embedded Runge-Kutta formulae[J]. Journal of Computational and Applied Mathematics, 1980, 6(1): 19−26 doi: 10.1016/0771-050X(80)90013-3
|
[46] |
Fang Zheng, Long Qingqing, Song Guojie, et al. Spatial-temporal graph ode networks for traffic flow forecasting[C]//Proc of the 27th ACM SIGKDD Conf on Knowledge Discovery and Data Mining. New York: ACM, 2021: 364−373
|
[47] |
Jin Ming, Zheng Yu, Li Yuanfang, et al. Multivariate time series forecasting with dynamic graph neural ODES[J]. IEEE Transactions on Knowledge and Data Engineering, 2023, 35(9): 9168−80 doi: 10.1109/TKDE.2022.3221989
|
[48] |
Huang Zijie, Sun Yizhou, Wang Wei. Coupled graph ODE for learning interacting system dynamics[C]//Proc of the 27th ACM SIGKDD Conf on Knowledge Discovery and Data Mining. New York: ACM, 2021: 705−715
|
[49] |
Choi J, Jeon J, Park N. LT-OCF: Learnable-time ode-based collaborative filtering[C]//Proc of the 30th ACM Int Conf on Information and Knowledge Management. New York, ACM, 2021: 251−260
|
[50] |
Jiang Song, Huang Zijie, Luo Xiao, et al. CF-GODE: Continuous-time causal inference for multi-agent dynamical systems[C]//Proc of the 29th ACM SIGKDD Conf on Knowledge Discovery and Data Mining. New York: ACM, 2023: 997−1009
|
[51] |
Rusch T K, Chamberlain B, Rowbottom J, et al. Graph-coupled oscillator networks[C]//Proc of the 39th Int Conf on Machine Learning. New York: PMLR, 2022: 18888−18909
|
[52] |
Huang Zijie, Zhao Wanjie, Gao Jingdong, et al. TANGO: Time-reversal latent GraphODE for multi-agent dynamical systems[C/OL]//Proc of NeurIPS 2023 Workshop on the Symbiosis of Deep Learning and Differential Equations. 2023[2024-05-16]. https://openreview.net/forum?id=8AYetbupoZ
|
[53] |
Liu Yang, Cheng Jiashun, Zhao Haihong, et al. Physics-inspired neural graph ODE for long-term dynamical simulation[J]. arXiv preprint, arXiv: 2308.13212, 2023
|
[54] |
Schober M, Särkkä S, Hennig P. A probabilistic model for the numerical solution of initial value problems[J]. Statistics and Computing, 2019, 29(1): 99−122 doi: 10.1007/s11222-017-9798-7
|
[55] |
Poli M, Massaroli S, Park J, et al. Graph neural ordinary differential equations [J]. arXiv preprint, arXiv: 1911.07532, 2019
|
[56] |
Huang Zijie, Sun Yizhou, Wang Wei. Learning continuous system dynamics from irregularly-sampled partial observations[C]//Proc of the 34th Int Conf on Neural Information Processing Systems. New York: Curran Associates, Inc, 2020, 33: 16177−16187
|
[57] |
Huang Zijie, Sun Yizhou, Wang Wei. Generalizing graph ODE for learning complex system dynamics across environments[C]//Proc of the 29th ACM SIGKDD Conf on Knowledge Discovery and Data Mining. New York: ACM, 2023: 798−809
|
[58] |
Hwang J, Choi J, Choi H, et al. Climate modeling with neural diffusion equations[C]//Proc of 2021 IEEE Int Conf on Data Mining. Piscataway, NJ: IEEE, 2021: 230−239
|
[59] |
Chen Yibi, Qin Yunchuan, Li Kenli, et al. Adaptive spatial-temporal graph convolution networks for collaborative local-global learning in traffic prediction[J]. IEEE Transactions on Vehicular Technology, 2023, 72(10): 12653−12663 doi: 10.1109/TVT.2023.3276752
|
[60] |
Guo Jiayan, Zhang Peiyan, Li Chaozhuo, et al. Evolutionary preference learning via graph nested GRU ODE for session-based recommendation[C]//Proc of the 31st ACM Int Conf on Information and Knowledge Management. New York: ACM, 2022: 624−634
|
[61] |
Luo Xiao, Yuan Jingyang, Huang Zijie, et al. HOPE: High-order graph ODE for modeling interacting dynamics[C]//Proc of the 40th Int Conf on Machine Learning. New York: PMLR, 2023: 23124−23139
|
[62] |
Jin Ming, Li Yuanfang, Pan Shirui. Neural temporal walks: Motif-aware representation learning on continuous-time dynamic graphs[C]//Proc of the 36th Int Conf on Neural Information Processing Systems. New York: Curran Associates, Inc, 2022, 35: 19874−19886
|
[63] |
Qin Yifang, Ju Wei, Wu Hongjun, et al. Learning graph node for continuous-time sequential recommendation [J]. IEEE Transactions on Knowledge and Data Engineering, 2024, 36(7): 1−14
|
[64] |
Gao Zhihan, Wang Hao, Wang Yuyang, et al. Probabilistic continuous-time whole-graph forecasting[C]//Proc of the 8th SIGKDD Int Workshop on Mining and Learning from Time Series. New York: ACM, 2022: 1−14
|
[65] |
Kipf T N, Welling M. Semi-supervised classification with graph convolutional networks[C/OL]//Proc of the 5th Int Conf on Learning Representations. 2017[2024-05-16]. https://openreview.net/forum?id=SJU4ayYgl
|
[66] |
Skarding J, Gabrys B, Musial K. Foundations and modeling of dynamic networks using dynamic graph neural networks: A survey[J]. IEEE Access, 2021, 9: 79143−79168 doi: 10.1109/ACCESS.2021.3082932
|
[67] |
Sankar A, Wu Yanhong, Gou Liang, et al. DySAT: Deep neural representation learning on dynamic graphs via self-attention networks[C]//Proc of the 13th Int Conf on Web Search and Data Mining. New York: ACM, 2020: 519−527
|
[68] |
Min Shengjie, Gao Zhan, Peng Jing, et al. STGSN—a spatial temporal graph neural network framework for time-evolving social networks[J]. Knowledge-Based Systems, 2021, 214: 106746 doi: 10.1016/j.knosys.2021.106746
|
[69] |
Gao Jianfei, Ribeiro B. On the equivalence between temporal and static equivariant graph representations[C]//Proc of the 39th Int Conf on Machine Learning. New York: PMLR, 2022: 7052−7076
|
[70] |
Yao Liuyi, Chu Zhixuan, Li Sheng, et al. A survey on causal inference[J]. ACM Transactions on Knowledge Discovery from Data, 2021, 15(5): 1−46
|
[1] | Liu Jialang, Guo Yanming, Lao Mingrui, Yu Tianyuan, Wu Yulun, Feng Yunhao, Wu Jiazhuang. Survey of Backdoor Attack and Defense Algorithms Based on Federated Learning[J]. Journal of Computer Research and Development, 2024, 61(10): 2607-2626. DOI: 10.7544/issn1000-1239.202440487 |
[2] | Zhu Suxia, Wang Jinyin, Sun Guanglu. Perceptual Similarity-Based Multi-Objective Optimization for Stealthy Image Backdoor Attack[J]. Journal of Computer Research and Development, 2024, 61(5): 1182-1192. DOI: 10.7544/issn1000-1239.202330521 |
[3] | Zhang Runlian, Pan Zhaoxuan, Li Jinlin, Wu Xiaonian, Wei Yongzhuang. A Side Channel Attack Based on Multi-Source Data Aggregation Neural Network[J]. Journal of Computer Research and Development, 2024, 61(1): 261-270. DOI: 10.7544/issn1000-1239.202220172 |
[4] | Zheng Mingyu, Lin Zheng, Liu Zhengxiao, Fu Peng, Wang Weiping. Survey of Textual Backdoor Attack and Defense[J]. Journal of Computer Research and Development, 2024, 61(1): 221-242. DOI: 10.7544/issn1000-1239.202220340 |
[5] | Chen Jingxue, Gao Kehan, Zhou Erqiang, Qin Zhen. Robust Source Anonymous Federated Learning Shuffle Protocol in IoT[J]. Journal of Computer Research and Development, 2023, 60(10): 2218-2233. DOI: 10.7544/issn1000-1239.202330393 |
[6] | Zhang Zhenyu, Jiang Yuan. Label Noise Robust Learning Algorithm in Environments Evolving Features[J]. Journal of Computer Research and Development, 2023, 60(8): 1740-1753. DOI: 10.7544/issn1000-1239.202330238 |
[7] | Chen Dawei, Fu Anmin, Zhou Chunyi, Chen Zhenzhu. Federated Learning Backdoor Attack Scheme Based on Generative Adversarial Network[J]. Journal of Computer Research and Development, 2021, 58(11): 2364-2373. DOI: 10.7544/issn1000-1239.2021.20210659 |
[8] | Zheng Haibin, Chen Jinyin, Zhang Yan, Zhang Xuhong, Ge Chunpeng, Liu Zhe, Ouyang Yike, Ji Shouling. Survey of Adversarial Attack, Defense and Robustness Analysis for Natural Language Processing[J]. Journal of Computer Research and Development, 2021, 58(8): 1727-1750. DOI: 10.7544/issn1000-1239.2021.20210304 |
[9] | Jin Jun and Zhang Daoqiang. Semi-Supervised Robust On-Line Clustering Algorithm[J]. Journal of Computer Research and Development, 2008, 45(3): 496-502. |
[10] | Hu Yusuo and Chen Zonghai. A Novel Robust Estimation Algorithm Based on Linear EIV Model[J]. Journal of Computer Research and Development, 2006, 43(3): 483-488. |