Abstract: Side channel attack based on deep learning needs to model and train each key byte of the cryptographic algorithm, which costs a lot of data acquisition and model training. To solve this problem, a side channel attack method based on multi-source data aggregation neural network is proposed. In order to screen the leaked data of key byte with good generalization quality for data aggregation, taking AES-128 algorithm as an example, firstly 16 single key byte models are trained based on the leaked data of 16 key bytes, and models are used to recover 16 key bytes respectively. Secondly, a scoring mechanism is designed to evaluate the generalization effect of each single key byte model, and models with the best recovery effect for each key byte are selected according to score sorting. Finally, a multi-source data aggregation model is constructed based on the key byte leaked data sets corresponding to the selected models to realize key recovery. The tested results show that the multi-source data aggregation model has good generalization effect, effectively improves the accuracy and efficiency of key recovery, reduces the number of traces used to recover the key, and the model also has good attack effect in the case of less traces.