Abstract: Training neural networks by using the linear multi-step method is studied, which is a classical numerical method for differential dynamics. It is pointed out that the iteration formula of BP algorithm is equivalent to the Euler method of differential dynamic systems under certain conditions, and the asymptotic solutions of the two formulas are consistent, and thus training a neural network can be converted to computing numerical solution of differential dynamic systems. Two algorithms to train neural networks are presented, namely the Milne method and the Hamming method. Finally three experiment examples are implemented to illustrate the fitness of training neural networks by the numerical method for differential dynamics.