Information Geometric Analysis of Pruning Algorithm

Pruning algorithm is an important method to set up and optimize the structure of neural network model. The research on pruning nowadays mostly focuses on the algorithm description while less effort is spent on its immanent mechanism. Research on its mechanism can provide theoretical basis for pruning strategy. The immanent mechanism of pruning is analyzed based on information geometry and a set of theoretical explanation of pruning is given. The pruning process is depicted as a series of information projections from the current model manifold to its submanifolds utilizing the hierarchical structure of neural manifold parameter space. A new pruning algorithm is presented based on the theoretical analysis and its validity and the efficiency is verified by experiments.