Abstract: Deep neural networks (DNNs) and their learning algorithms are well known in the academic community and industry as the most successful methods for big data analysis. Compared with traditional methods, deep learning methods use data-driven and can extract features (knowledge) automatically from data. Deep learning methods have significant advantages in analyzing unstructured, unknown and varied model and cross field big data. At present, the most widely used deep neural networks in big data analysis are feedforward neural networks (FNNs). They work well in extracting the correlation from static data and suiting for data application scenarios based on classification. But limited by its intrinsic structure, the ability of feedforward neural networks to extract time sequence features is weak. Infinite deep neural networks, i.e. recurrent neural networks (RNNs) are dynamical systems essentially. Their essential character is that the states of the networks change with time and couple the time parameter. Hence they are very suit for extracting time sequence features. It means that infinite deep neural networks can perform the prediction of big data. If extending recurrent structure of recurrent neural networks in the time dimension, the depth of networks can be infinite with time running, so they are called infinite deep neural networks. In this paper, we focus on the topology and some learning algorithms of infinite deep neural networks, and introduce some successful applications in speech recognition and image understanding.