Survey on Graph Neural Ordinary Differential Equations

Jiao Pengfei; Chen Shuxin; Guo Xuan; He Dongxiao; Liu Dong

doi:10.7544/issn1000-1239.202440192

Journal of Computer Research and Development > 2024 > 61(8): 2045-2066. > DOI: 10.7544/issn1000-1239.202440192 CSTR: 32373.14.issn1000-1239.202440192

Jiao Pengfei, Chen Shuxin, Guo Xuan, He Dongxiao, Liu Dong. Survey on Graph Neural Ordinary Differential Equations[J]. Journal of Computer Research and Development, 2024, 61(8): 2045-2066. DOI: 10.7544/issn1000-1239.202440192

Citation:

PDF (2092 KB)

Survey on Graph Neural Ordinary Differential Equations

1.
School of Cyberspace, Hangzhou Dianzi University, Hangzhou 310018
2.
ZhuoYue Honors College, Hangzhou Dianzi University, Hangzhou 310018
3.
College of Intelligence and Computing, Tianjin University, Tianjin 300350
4.
College of Computer and Information Engineering, Henan Normal University, Xinxiang, Henan 453007

Funds: This work was supported by the National Natural Science Foundation of China (62372146, 62072160, 62276187) and the Fundamental Research Funds for the Provincial Universities of Zhejiang (GK229909299001-008).

More Information

Author Bio:
Jiao Pengfei: born in 1990. PhD, professor. His main research interest includes complex network analysis and its applications

Chen Shuxin: born in 2004. Undergraduate. Her main research interest includes graph neural network

Guo Xuan: born in 1996. PhD candidate. His main research interest includes graph machine learning

He Dongxiao: born in 1984. PhD, professor. Her main research interests include analysis of complex networks, graph data mining, and graph neural network

Liu Dong: born in 1976. PhD, professor. His main research interests include educational data mining and complex network analysis
Received Date: March 15, 2024
Revised Date: May 14, 2024
Available Online: July 04, 2024

Graphical Abstract

Abstract

Abstract

Graph neural networks (GNNs) are powerful tools for handling graph-structured data, capable of capturing complex relationships and features among nodes. However, the discrete architecture of GNNs leads to numerous challenges in representing graph structures, modeling graph evolution, adapting to irregular data, and managing computational costs. In response to these challenges, neural ordinary differential equations (ODEs) have been introduced as a novel method to address the challenges faced by GNNs, as they can simulate the continuous evolution of system states, providing continuous deep encoding and inference capabilities. However, neural ODEs are designed for Euclidean structured data and cannot directly capture the characteristics of graphs. Therefore, researchers have proposed graph neural ODEs, a new type of architectures that combines neural ODEs with GNNs, which can better adapt to graph-structured data and fully utilize its characteristics. In recent years, research related to graph neural ODEs has delved into various directions of graph machine learning, sparking a new research trend. In this context, we systematically review the relevant research of graph neural ODEs in a timely manner. Firstly, we review the key advantages of GNNs and the challenges they face, and elucidate the theoretical basis and practical significance of introducing neural ODEs and combining them with GNNs. Subsequently, we elaborate on the background and basic concepts of graph neural ODEs, proposing a novel taxonomy, and comprehensively describe some important current methods on the taxonomy. Then, we introduce commonly used verification methods in related research, including downstream tasks and datasets. Furthermore, we delve into the applications of graph neural ODEs in multiple practical fields. Finally, we summarize and prospect the challenges and future development trends of graph neural ODEs.
- graph machine learning,
- graph neural networks,
- neural ordinary differential equations,
- taxonomy,
- survey

FullText(HTML)

References (70)

References

[1]	徐冰冰,岑科廷,黄俊杰,等. 图卷积神经网络综述[J]. 计算机学报,2020,43(5):755−780 doi: 10.11897/SP.J.1016.2020.00755 Xu Bingbing, Cen Keting, Huang Junjie, et al. A survey on graph convolutional network[J]. Chinese Journal of Computers, 2020, 43(5): 755−780 (in Chinese) doi: 10.11897/SP.J.1016.2020.00755
[2]	马帅,刘建伟,左信. 图神经网络综述[J]. 计算机研究与发展,2022,59(1):47−80 doi: 10.7544/issn1000-1239.20201055 Ma Shuai, Liu Jianwei, Zuo Xin. Survey on graph neural network[J]. Journal of Computer Research and Development, 2022, 59(1): 47−80 (in Chinese) doi: 10.7544/issn1000-1239.20201055
[3]	Wu Zonghan, Pan Shirui, Chen Fengwen, et al. A comprehensive survey on graph neural networks[J]. IEEE Transactions on Neural Networks and Learning Systems, 2020, 32(1): 4−24
[4]	Liu Juncheng, Hooi B, Kawaguchi K, et al. Scalable and effective implicit graph neural networks on large graphs[C/OL]//Proc of the 12th Int Conf on Learning Representations. 2024[2024-05-16].https://openreview.net/forum?id=QcMdPYBwTu
[5]	Xu Keyulu, Hu Weihua, Leskovec J, et al. How powerful are graph neural networks?[C/OL]//Proc of the 7th Int Conf on Learning Representations. 2019[2024-05-16].https://openreview.net/forum?id=ryGs6iA5Km
[6]	Gasteiger J, Bojchevski A, Günnemann S. Predict then propagate: Graph neural networks meet personalized pagerank[C/OL]//Proc of the 7th Int Conf on Learning Representations. 2019[2024-05-16].https://openreview.net/forum?id=H1gL-2A9Ym
[7]	Hajiramezanali E, Hasanzadeh A, Narayanan K, et al. Variational graph recurrent neural networks[C]//Proc of the 33rd Int Conf on Neural Information Processing Systems. New York: Curran Associates, Inc, 2019, 32: 10701−10711
[8]	Xu Da, Ruan Chuanwei, Korpeoglu E, et al. Inductive representation learning on temporal graphs[C/OL]//Proc of the 8th Int Conf on Learning Representations. 2020[2024-05-16].https://openreview.net/forum?id=rJeW1yHYwH
[9]	Wu Yongji, Lian Defu, Xu Yiheng, et al. Graph convolutional networks with Markov random field reasoning for social spammer detection[C]//Proc of the 34th AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2020, 34(1): 1054−1061
[10]	Zhang Si, Tong Hanghang, Xu Jiejun, et al. Graph convolutional networks: A comprehensive review[J]. Computational Social Networks, 2019, 6(1): 1−23 doi: 10.1186/s40649-019-0061-6
[11]	Ying Rex, He Ruining, Chen Kaifeng, et al. Graph convolutional neural networks for web-scale recommender systems[C]//Proc of the 24th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. New York: ACM, 2018: 974−983
[12]	王磊,熊于宁,李云鹏,等. 一种基于增强图卷积神经网络的协同推荐模型[J]. 计算机研究与发展,2021,58(9):1987−1996 doi: 10.7544/issn1000-1239.2021.20200617 Wang Lei, Xiong Yuning, Li Yunpeng, el al. A collaborative recommendation model based on enhanced graph convolutional neural network[J]. Journal of Computer Research and Development, 2021, 58(9): 1987−1996 (in Chinese) doi: 10.7544/issn1000-1239.2021.20200617
[13]	Wang Shoujin, Hu Liang, Wang Yan, et al. Graph learning based recommender systems: A review[C]//Proc of the 30th Int Joint Conf on Artificial Intelligence. San Francisco, CA: IJCAI, 2021: 4644−4652
[14]	Fout A, Byrd J, Shariat B, et al. Protein interface prediction using graph convolutional networks[C]//Proc of the 31st Int Conf on Neural Information Processing Systems. New York: Curran Associates, Inc, 2017, 30: 6533−6542
[15]	Borgwardt K M, Ong C S, Schönauer S, et al. Protein function prediction via graph kernels[J]. Bioinformatics, 2005, 21(suppl_1): i47−i56
[16]	Guo Shengnan, Lin Youfang, Feng Ning, et al. Attention based spatial-temporal graph convolutional networks for traffic flow forecasting[C]//Proc of the 33rd AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2019: 922−929
[17]	Yu Bing, Yin Haoteng, Zhu Zhanxing. Spatio-temporal graph convolutional networks: A deep learning framework for traffic forecasting[C]//Proc of the 28th Int Joint Conf on Artificial Intelligence. San Francisco, CA: IJCAI, 2018: 3634−3640
[18]	杜圣东,李天瑞,杨燕,等. 一种基于序列到序列时空注意力学习的交通流预测模型[J]. 计算机研究与发展,2020,57(8):1715−1728 doi: 10.7544/issn1000-1239.2020.20200169 Du Shengdong, Li Tianrui, Yang Yan, et al. A sequence-to-sequence spatial-temporal attention learning model for urban traffic flow prediction[J]. Journal of Computer Research and Development, 2020, 57(8): 1715−1728 (in Chinese) doi: 10.7544/issn1000-1239.2020.20200169
[19]	Bodnar C, Di Giovanni F, Chamberlain B, et al. Neural sheaf diffusion: A topological perspective on heterophily and oversmoothing in GNNS[C]//Proc of the 36th Int Conf on Neural Information Processing Systems. New York: Curran Associates, Inc, 2022, 35: 18527−18541
[20]	Rusch T K, Bronstein M M, Mishra S. A survey on oversmoothing in graph neural networks[J]. arXiv preprint, arXiv: 2303.10993, 2023
[21]	Rong Yu, Huang Wenbing, Xu Tingyang, et al. DropEdge: Towards deep graph convolutional networks on node classification[C/OL]//Proc of the 8th Int Conf on Learning Representations. 2020 [2024-05-16].https://openreview.net/forum?id=Hkx1qkrKPr
[22]	Zhao L, Akoglu L. PairNorm: Tackling oversmoothing in GNN[C]//Proc of the 8th Int Conf on Learning Representations. 2020[2024-05-16].https://openreview.net/forum?id=rkecl1rtwB
[23]	Xu Keyulu, Li Chengtao, Tian Yonglong, et al. Representation learning on graphs with jumping knowledge networks[C]//Proc of the 35th Int Conf on Machine Learning. New York: PMLR, 2018: 5453−5462
[24]	Barzel B, Liu Yangyu, Barabási A L. Constructing minimal models for complex system dynamics[J]. Nature Communication, 2015, 6(1): 7186 doi: 10.1038/ncomms8186
[25]	Alvarez-Rodriguez U, Battiston F, de Arruda G F, et al. Evolutionary dynamics of higher-order interactions in social networks[J]. Nature Human Behaviour, 2021, 5(5): 586−595 doi: 10.1038/s41562-020-01024-1
[26]	Zhang Yanfu, Gao Shangqian, Pei Jian, et al. Improving social network embedding via new second-order continuous graph neural networks[C]//Proc of the 28th ACM SIGKDD Conf on Knowledge Discovery and Data Mining. New York: ACM, 2022: 2515−2523
[27]	Hou Jinlin, Guo Xuan, Liu Jiye, et al. Structure-enhanced graph neural ODE network for temporal link prediction[C]//Proc of the 33rd Int Conf on Artificial Neural Networks. Berlin: Springer, 2023: 563−575
[28]	Fey M, Lenssen J E, Weichert F, et al. Gnnautoscale: Scalable and expressive graph neural networks via historical embeddings[C]//Proc of the 38th Int Confon Machine Learning. New York: PMLR, 2021: 3294−3304
[29]	Chen R T Q, Rubanova Y, Bettencourt J, et al. Neural ordinary differential equations[C]//Proc of the 32nd Int Conf on Neural Information Processing Systems. New York: Curran Associates, Inc, 2018, 31: 6572−6583
[30]	Xhonneux L P, Qu Meng, Tang Jian. Continuous graph neural networks[C]//Proc of the 37th Int Conf on Machine Learning. New York: PMLR, 2020: 10432−10441
[31]	Zang Chengxi, Wang Fei. Neural dynamics on complex networks[C]//Proc of the 26th ACM SIGKDD Conf on Knowledge Discovery and Data Mining. New York: ACM, 2020: 892−902
[32]	Wang Zihui, Yang Peizhen, Fan Xiaoliang, et al. ConTIG: Continuous representation learning on temporal interaction graphs[J]. Neural Networks, 2024, 172: 106151 doi: 10.1016/j.neunet.2024.106151
[33]	Veličković P, Cucurull G, Casanova A, et al. Graph attention networks[C/OL]//Proc of the 6th Int Conf on Learning Representations. 2018[2024-05-16].https://openreview.net/forum?id=rJXMpikCZ
[34]	Rossi E, Chamberlain B, Frasca F, et al. Temporal graph networks for deep learning on dynamic graphs[C]//Proc of the ICML 2020 Workshop on Graph Representation Learning and Beyond. New York: PMLR, 2020: 1−9
[35]	Ma Yao, Guo Ziyi, Ren Zhaocun, et al. Streaming graph neural networks[C]//Proc of the 43rd Int ACM SIGIR Conf on Research and Development in Information Retrieval. New York: ACM, 2020: 719−728
[36]	Zhou Hongkuan, Zheng Da, Nisa I, et al. TGL: A general framework for temporal gnn training on billion-scale graphs[C]//Proc of the 48th Int Conf on Very Large Databases. New York: VLDB Endowment, 2022, 15(8): 1572−1580
[37]	Pareja A, Domeniconi G, Chen Jie, et al. EvolveGCN: Evolving graph convolutional networks for dynamic graphs[C]//Proc of the 34th AAAI Conf on Artificial Intelligence. Palo Alto, CA: AAAI, 2020: 5363−5370
[38]	Taheri A, Gimpel K, Berger-Wolf T. Learning to represent the evolution of dynamic graphs with recurrent models[C]//Proc of Companion Proc of the 2019 World Wide Web Conf. New York: ACM, 2019: 301−307
[39]	Wang Yanbang, Li Pan, Bai Chongyang, et al. Generic representation learning for dynamic social interaction[C]//Proc of the 26th ACM SIGKDD Conf on Knowledge Discovery and Data Mining Workshop. New York: ACM, 2020: 1−9
[40]	You Jiaxuan, Du Tianyu, Leskovec J. ROLAND: Graph learning framework for dynamic graphs[C]//Proc of the 28th ACM SIGKDD Conf on Knowledge Discovery and Data Mining. New York: ACM, 2022: 2358−2366
[41]	Pontryagin L S. Mathematical Theory of Optimal Processes[M]. London: Routledge, 2018
[42]	Rubanova Y, Chen R T Q, Duvenaud D K. Latent ordinary differential equations for irregularly-sampled time series[C]//Proc of the 33rd Int Conf on Neural Information Processing Systems. New York: Curran Associates, Inc, 2019, 32: 5320−5330
[43]	Dupont E, Doucet A, Teh Y W. Augmented neural ODEs[C]//Proc of the 33rd Int Conf on Neural Information Processing Systems. New York: Curran Associates, Inc, 2019, 32: 3140−3150
[44]	Biswas B N, Chatterjee S, Mukherjee S P, et al. A discussion on Euler method: A review[J]. Electronic Journal of Mathematical Analysis and Applications, 2013, 1(2): 2090−2792
[45]	Dormand J R, Prince P J. A family of embedded Runge-Kutta formulae[J]. Journal of Computational and Applied Mathematics, 1980, 6(1): 19−26 doi: 10.1016/0771-050X(80)90013-3
[46]	Fang Zheng, Long Qingqing, Song Guojie, et al. Spatial-temporal graph ode networks for traffic flow forecasting[C]//Proc of the 27th ACM SIGKDD Conf on Knowledge Discovery and Data Mining. New York: ACM, 2021: 364−373
[47]	Jin Ming, Zheng Yu, Li Yuanfang, et al. Multivariate time series forecasting with dynamic graph neural ODES[J]. IEEE Transactions on Knowledge and Data Engineering, 2023, 35(9): 9168−80 doi: 10.1109/TKDE.2022.3221989
[48]	Huang Zijie, Sun Yizhou, Wang Wei. Coupled graph ODE for learning interacting system dynamics[C]//Proc of the 27th ACM SIGKDD Conf on Knowledge Discovery and Data Mining. New York: ACM, 2021: 705−715
[49]	Choi J, Jeon J, Park N. LT-OCF: Learnable-time ode-based collaborative filtering[C]//Proc of the 30th ACM Int Conf on Information and Knowledge Management. New York, ACM, 2021: 251−260
[50]	Jiang Song, Huang Zijie, Luo Xiao, et al. CF-GODE: Continuous-time causal inference for multi-agent dynamical systems[C]//Proc of the 29th ACM SIGKDD Conf on Knowledge Discovery and Data Mining. New York: ACM, 2023: 997−1009
[51]	Rusch T K, Chamberlain B, Rowbottom J, et al. Graph-coupled oscillator networks[C]//Proc of the 39th Int Conf on Machine Learning. New York: PMLR, 2022: 18888−18909
[52]	Huang Zijie, Zhao Wanjie, Gao Jingdong, et al. TANGO: Time-reversal latent GraphODE for multi-agent dynamical systems[C/OL]//Proc of NeurIPS 2023 Workshop on the Symbiosis of Deep Learning and Differential Equations. 2023[2024-05-16]. https://openreview.net/forum?id=8AYetbupoZ
[53]	Liu Yang, Cheng Jiashun, Zhao Haihong, et al. Physics-inspired neural graph ODE for long-term dynamical simulation[J]. arXiv preprint, arXiv: 2308.13212, 2023
[54]	Schober M, Särkkä S, Hennig P. A probabilistic model for the numerical solution of initial value problems[J]. Statistics and Computing, 2019, 29(1): 99−122 doi: 10.1007/s11222-017-9798-7
[55]	Poli M, Massaroli S, Park J, et al. Graph neural ordinary differential equations [J]. arXiv preprint, arXiv: 1911.07532, 2019
[56]	Huang Zijie, Sun Yizhou, Wang Wei. Learning continuous system dynamics from irregularly-sampled partial observations[C]//Proc of the 34th Int Conf on Neural Information Processing Systems. New York: Curran Associates, Inc, 2020, 33: 16177−16187
[57]	Huang Zijie, Sun Yizhou, Wang Wei. Generalizing graph ODE for learning complex system dynamics across environments[C]//Proc of the 29th ACM SIGKDD Conf on Knowledge Discovery and Data Mining. New York: ACM, 2023: 798−809
[58]	Hwang J, Choi J, Choi H, et al. Climate modeling with neural diffusion equations[C]//Proc of 2021 IEEE Int Conf on Data Mining. Piscataway, NJ: IEEE, 2021: 230−239
[59]	Chen Yibi, Qin Yunchuan, Li Kenli, et al. Adaptive spatial-temporal graph convolution networks for collaborative local-global learning in traffic prediction[J]. IEEE Transactions on Vehicular Technology, 2023, 72(10): 12653−12663 doi: 10.1109/TVT.2023.3276752
[60]	Guo Jiayan, Zhang Peiyan, Li Chaozhuo, et al. Evolutionary preference learning via graph nested GRU ODE for session-based recommendation[C]//Proc of the 31st ACM Int Conf on Information and Knowledge Management. New York: ACM, 2022: 624−634
[61]	Luo Xiao, Yuan Jingyang, Huang Zijie, et al. HOPE: High-order graph ODE for modeling interacting dynamics[C]//Proc of the 40th Int Conf on Machine Learning. New York: PMLR, 2023: 23124−23139
[62]	Jin Ming, Li Yuanfang, Pan Shirui. Neural temporal walks: Motif-aware representation learning on continuous-time dynamic graphs[C]//Proc of the 36th Int Conf on Neural Information Processing Systems. New York: Curran Associates, Inc, 2022, 35: 19874−19886
[63]	Qin Yifang, Ju Wei, Wu Hongjun, et al. Learning graph node for continuous-time sequential recommendation [J]. IEEE Transactions on Knowledge and Data Engineering, 2024, 36(7): 1−14
[64]	Gao Zhihan, Wang Hao, Wang Yuyang, et al. Probabilistic continuous-time whole-graph forecasting[C]//Proc of the 8th SIGKDD Int Workshop on Mining and Learning from Time Series. New York: ACM, 2022: 1−14
[65]	Kipf T N, Welling M. Semi-supervised classification with graph convolutional networks[C/OL]//Proc of the 5th Int Conf on Learning Representations. 2017[2024-05-16]. https://openreview.net/forum?id=SJU4ayYgl
[66]	Skarding J, Gabrys B, Musial K. Foundations and modeling of dynamic networks using dynamic graph neural networks: A survey[J]. IEEE Access, 2021, 9: 79143−79168 doi: 10.1109/ACCESS.2021.3082932
[67]	Sankar A, Wu Yanhong, Gou Liang, et al. DySAT: Deep neural representation learning on dynamic graphs via self-attention networks[C]//Proc of the 13th Int Conf on Web Search and Data Mining. New York: ACM, 2020: 519−527
[68]	Min Shengjie, Gao Zhan, Peng Jing, et al. STGSN—a spatial temporal graph neural network framework for time-evolving social networks[J]. Knowledge-Based Systems, 2021, 214: 106746 doi: 10.1016/j.knosys.2021.106746
[69]	Gao Jianfei, Ribeiro B. On the equivalence between temporal and static equivariant graph representations[C]//Proc of the 39th Int Conf on Machine Learning. New York: PMLR, 2022: 7052−7076
[70]	Yao Liuyi, Chu Zhixuan, Li Sheng, et al. A survey on causal inference[J]. ACM Transactions on Knowledge Discovery from Data, 2021, 15(5): 1−46

[1]	Yang Yi, Li Ying, Chen Kai. Vulnerability Detection Methods Based on Natural Language Processing[J]. Journal of Computer Research and Development, 2022, 59(12): 2649-2666. DOI: 10.7544/issn1000-1239.20210627
[2]	Xu Lixin, Wu Huayao. Collective Intelligence Based Software Engineering[J]. Journal of Computer Research and Development, 2020, 57(3): 487-512. DOI: 10.7544/issn1000-1239.2020.20190626
[3]	Qiu Yu, Wang Chi, Qi Kaiyue, Shen Yao, Li Chao, Zhang Chengmi, Guo Minyi. A Survey of Smart Health: System Design from the Cloud to the Edge[J]. Journal of Computer Research and Development, 2020, 57(1): 53-73. DOI: 10.7544/issn1000-1239.2020.20190002
[4]	Wang Jice, Li Yilian, Jia Yan, Zhou Wei, Wang Yucheng, Wang He, Zhang Yuqing. Survey of Smart Home Security[J]. Journal of Computer Research and Development, 2018, 55(10): 2111-2124. DOI: 10.7544/issn1000-1239.2018.20180585
[5]	Zhang Yuqing, Zhou Wei, Peng Anni. Survey of Internet of Things Security[J]. Journal of Computer Research and Development, 2017, 54(10): 2130-2143. DOI: 10.7544/issn1000-1239.2017.20170470
[6]	Wei Shanshan, Xie Wei, He Zhiqiang. Digital Video Stabilization Techniques: A Survey[J]. Journal of Computer Research and Development, 2017, 54(9): 2044-2058. DOI: 10.7544/issn1000-1239.2017.20160078
[7]	Zhang Yuqing, Fang Zhejun, Wang Kai, Wang Zhiqiang, Yue Hongzhou, Liu Qixu, He Yuan, Li Xiaoqi, Yang Gang. Survey of Android Vulnerability Detection[J]. Journal of Computer Research and Development, 2015, 52(10): 2167-2177. DOI: 10.7544/issn1000-1239.2015.20150572
[8]	Zhang Yuqing, Wang Kai, Yang Huan, Fang Zhejun, Wang Zhiqiang, and Cao Chen. Survey of Android OS Security[J]. Journal of Computer Research and Development, 2014, 51(7): 1385-1396.
[9]	Li Renfa, Xie Yong, Li Rui, and Li Lang. Survey of Cyber-Physical Systems[J]. Journal of Computer Research and Development, 2012, 49(6): 1149-1161.
[10]	Fang Binxing, Cui Xiang, Wang Wei. Survey of Botnets[J]. Journal of Computer Research and Development, 2011, 48(8): 1315-1331.